BGP session having trouble all of a sudden on 7204 router

We have a 7204 router that has two BGP sessions to our upstream provider. The first one provides us a single default route, and the second one provides us a small list of Internet2 related routes. This setup has been working just fine for many years, but recently the second, Internet2 one has been having trouble. Both sides are trying to connect, but it never establishes. It mainly shows OpenConfirm when I check status. Nothing has been changed on this router in years, and no one has logged into it in many months. Our provider insists that the problem in on our end and they say they're not receiving keepalives from our side, but it looks like they're being sent. Whatcan we do?

 

http://en.wikipedia.org/wiki/Border_Gateway_Protocol#Finite-state_machine

OpenConfirm State:
The peer is listening for a Keepalive message from its peer.
If a Keepalive message is received and no timer has expired before
reception of the Keepalive, BGP transitions to the Established state.
If a timer expires before a Keepalive message is received, or if an
error condition occurs, the router transitions back to the Idle state.

You sent a keepalive before transitioning from OpenSent to OpenConfirm.

Collect:
debug ip bgp

Perhaps, depending on interface etc you can put a sniffer between you and
the ISP and see what's really going on.

 

Here is 'debug ip bgp' from the other day...W.X.Y.A is the local side and W.X.Y.Z is the remote side

Dec 18 15:58:30.278: BGP: W.X.Y.A open failed: Connection timed out; remote host not responding, open active delayed 32829ms (35000ms max, 28% jitter)
Dec 18 15:59:03.110: BGP: W.X.Y.A open active, local address W.X.Y.Z
Dec 18 15:59:33.110: BGP: W.X.Y.A open failed: Connection timed out; remote host not responding, open active delayed 28160ms (35000ms max, 28% jitter)
Dec 18 15:59:56.323: BGP: W.X.Y.A passive open to W.X.Y.Z
Dec 18 15:59:56.323: BGP: W.X.Y.A went from Active to Idle
Dec 18 15:59:56.327: BGP: W.X.Y.A went from Idle to Connect
Dec 18 15:59:56.327: BGP: W.X.Y.A rcv message type 1, length (excl. header) 44
Dec 18 15:59:56.327: BGP: W.X.Y.A rcv OPEN, version 4, holdtime 90 seconds
Dec 18 15:59:56.327: BGP: W.X.Y.A went from Connect to OpenSent
Dec 18 15:59:56.327: BGP: W.X.Y.A sending OPEN, version 4, my as: 64875, holdtime 180 seconds
Dec 18 15:59:56.327: BGP: W.X.Y.A rcv OPEN w/ OPTION parameter len: 34
Dec 18 15:59:56.327: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 6
Dec 18 15:59:56.327: BGP: W.X.Y.A OPEN has CAPABILITY code: 1, length 4
Dec 18 15:59:56.327: BGP: W.X.Y.A OPEN has MP_EXT CAP for afi/safi: 1/1
Dec 18 15:59:56.327: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 2
Dec 18 15:59:56.327: BGP: W.X.Y.A OPEN has CAPABILITY code: 128, length 0
Dec 18 15:59:56.327: BGP: W.X.Y.A OPEN has ROUTE-REFRESH capability(old) for all address-families
Dec 18 15:59:56.327: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 2
Dec 18 15:59:56.327: BGP: W.X.Y.A OPEN has CAPABILITY code: 2, length 0
Dec 18 15:59:56.327: BGP: W.X.Y.A OPEN has ROUTE-REFRESH capability(new) for all address-families
Dec 18 15:59:56.327: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 8
Dec 18 15:59:56.327: BGP: W.X.Y.A OPEN has CAPABILITY code: 64, length 6
Dec 18 15:59:56.327: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 6
Dec 18 15:59:56.327: BGP: W.X.Y.A OPEN has CAPABILITY code: 65, length 4
Dec 18 15:59:56.327: BGP: W.X.Y.A unrecognized capability code: 65 - ingored
BGP: W.X.Y.A rcvd OPEN w/ remote AS 3112
Dec 18 15:59:56.327: BGP: W.X.Y.A went from OpenSent to OpenConfirm
Dec 18 15:59:56.327: BGP: W.X.Y.A send message type 1, length (incl. header) 45
Dec 18 16:01:26.327: BGP: W.X.Y.A connection timed out - has not accepted a message from us for 90000ms (hold
time), 0 messages pending transmition
Dec 18 16:01:26.327: BGP: W.X.Y.A went from OpenConfirm to Closing
Dec 18 16:01:26: %BGP-3-NOTIFICATION: sent to neighbor W.X.Y.A 4/0 (hold time expired) 0 bytes
Dec 18 16:01:26.327: BGP: W.X.Y.A send message type 3, length (incl. header) 21
Dec 18 16:01:26.327: BGP: W.X.Y.A local error close after sending NOTIFICATION
Dec 18 16:01:26.327: BGPNSF state: W.X.Y.A went from nsf_not_active to nsf_not_active
Dec 18 16:01:26.327: BGP: W.X.Y.A went from Closing to Idle
Dec 18 16:01:26.327: BGP: W.X.Y.A closing
Dec 18 16:01:27.331: BGP: W.X.Y.A went from Idle to Active
Dec 18 16:01:27.331: BGP: W.X.Y.A open active delayed 30201ms (35000ms max, 28% jitter)
Dec 18 16:01:57.535: BGP: W.X.Y.A open active, local address W.X.Y.Z
Dec 18 16:02:27.536: BGP: W.X.Y.A open failed: Connection timed out; remote host not responding, open active delayed 32450ms (35000ms max, 28% jitter)
Dec 18 16:02:59.988: BGP: W.X.Y.A open active, local address W.X.Y.Z
Dec 18 16:03:29.988: BGP: W.X.Y.A open failed: Connection timed out; remote host not responding, open active delayed 26383ms (35000ms max, 28% jitter)
Dec 18 16:03:54.328: BGP: W.X.Y.A passive open to W.X.Y.Z
Dec 18 16:03:54.328: BGP: W.X.Y.A went from Active to Idle
Dec 18 16:03:54.328: BGP: W.X.Y.A went from Idle to Connect
Dec 18 16:03:54.332: BGP: W.X.Y.A rcv message type 1, length (excl. header) 44
Dec 18 16:03:54.332: BGP: W.X.Y.A rcv OPEN, version 4, holdtime 90 seconds
Dec 18 16:03:54.332: BGP: W.X.Y.A went from Connect to OpenSent
Dec 18 16:03:54.332: BGP: W.X.Y.A sending OPEN, version 4, my as: 64875, holdtime 180 seconds
Dec 18 16:03:54.332: BGP: W.X.Y.A rcv OPEN w/ OPTION parameter len: 34
Dec 18 16:03:54.332: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 6
Dec 18 16:03:54.332: BGP: W.X.Y.A OPEN has CAPABILITY code: 1, length 4
Dec 18 16:03:54.332: BGP: W.X.Y.A OPEN has MP_EXT CAP for afi/safi: 1/1
Dec 18 16:03:54.332: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 2
Dec 18 16:03:54.332: BGP: W.X.Y.A OPEN has CAPABILITY code: 128, length 0
Dec 18 16:03:54.332: BGP: W.X.Y.A OPEN has ROUTE-REFRESH capability(old) for all address-families
Dec 18 16:03:54.332: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 2
Dec 18 16:03:54.332: BGP: W.X.Y.A OPEN has CAPABILITY code: 2, length 0
Dec 18 16:03:54.332: BGP: W.X.Y.A OPEN has ROUTE-REFRESH capability(new) for all address-families
Dec 18 16:03:54.332: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 8
Dec 18 16:03:54.332: BGP: W.X.Y.A OPEN has CAPABILITY code: 64, length 6
Dec 18 16:03:54.332: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 6
Dec 18 16:03:54.332: BGP: W.X.Y.A OPEN has CAPABILITY code: 65, length 4
Dec 18 16:03:54.332: BGP: W.X.Y.A unrecognized capability code: 65 - ingored
BGP: W.X.Y.A rcvd OPEN w/ remote AS 3112
Dec 18 16:03:54.332: BGP: W.X.Y.A went from OpenSent to OpenConfirm
Dec 18 16:03:54.332: BGP: W.X.Y.A send message type 1, length (incl. header) 45
Dec 18 16:05:24.329: BGP: W.X.Y.A rcv message type 3, length (excl. header) 2
Dec 18 16:05:24: %BGP-3-NOTIFICATION: received from neighbor W.X.Y.A 4/0 (hold time expired) 0 bytes
Dec 18 16:05:24.333: BGP: W.X.Y.A went from OpenConfirm to Closing
Dec 18 16:05:24.333: BGPNSF state: W.X.Y.A went from nsf_not_active to nsf_not_active
Dec 18 16:05:24.333: BGP: W.X.Y.A went from Closing to Idle
Dec 18 16:05:24.333: BGP: W.X.Y.A closing
Dec 18 16:05:24.433: BGP: W.X.Y.A went from Idle to Active
Dec 18 16:05:24.433: BGP: W.X.Y.A open active delayed 34021ms (35000ms max, 28% jitter)
Dec 18 16:05:58.457: BGP: W.X.Y.A open active, local address W.X.Y.Z
Dec 18 16:06:28.457: BGP: W.X.Y.A open failed: Connection timed out; remote host not responding, open active delayed 28111ms (35000ms max, 28% jitter)
Dec 18 16:06:56.570: BGP: W.X.Y.A open active, local address W.X.Y.Z
Dec 18 16:07:26.570: BGP: W.X.Y.A open failed: Connection timed out; remote host not responding, open active delayed 33307ms (35000ms max, 28% jitter)
Dec 18 16:07:52.334: BGP: W.X.Y.A passive open to W.X.Y.Z
Dec 18 16:07:52.334: BGP: W.X.Y.A went from Active to Idle
Dec 18 16:07:52.334: BGP: W.X.Y.A went from Idle to Connect
Dec 18 16:07:52.334: BGP: W.X.Y.A rcv message type 1, length (excl. header) 44
Dec 18 16:07:52.338: BGP: W.X.Y.A rcv OPEN, version 4, holdtime 90 seconds
Dec 18 16:07:52.338: BGP: W.X.Y.A went from Connect to OpenSent
Dec 18 16:07:52.338: BGP: W.X.Y.A sending OPEN, version 4, my as: 64875, holdtime 180 seconds
Dec 18 16:07:52.338: BGP: W.X.Y.A rcv OPEN w/ OPTION parameter len: 34
Dec 18 16:07:52.338: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 6
Dec 18 16:07:52.338: BGP: W.X.Y.A OPEN has CAPABILITY code: 1, length 4
Dec 18 16:07:52.338: BGP: W.X.Y.A OPEN has MP_EXT CAP for afi/safi: 1/1
Dec 18 16:07:52.338: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 2
Dec 18 16:07:52.338: BGP: W.X.Y.A OPEN has CAPABILITY code: 128, length 0
Dec 18 16:07:52.338: BGP: W.X.Y.A OPEN has ROUTE-REFRESH capability(old) for all address-families
Dec 18 16:07:52.338: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 2
Dec 18 16:07:52.338: BGP: W.X.Y.A OPEN has CAPABILITY code: 2, length 0
Dec 18 16:07:52.338: BGP: W.X.Y.A OPEN has ROUTE-REFRESH capability(new) for all address-families
Dec 18 16:07:52.338: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 8
Dec 18 16:07:52.338: BGP: W.X.Y.A OPEN has CAPABILITY code: 64, length 6
Dec 18 16:07:52.338: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 6
Dec 18 16:07:52.338: BGP: W.X.Y.A OPEN has CAPABILITY code: 65, length 4
Dec 18 16:07:52.338: BGP: W.X.Y.A unrecognized capability code: 65 - ingored
BGP: W.X.Y.A rcvd OPEN w/ remote AS 3112
Dec 18 16:07:52.338: BGP: W.X.Y.A went from OpenSent to OpenConfirm
Dec 18 16:07:52.338: BGP: W.X.Y.A send message type 1, length (incl. header)

 

Sorry... got the addresses backwards. W.X.Y.A is the remote side and W.X.Y.Z is the local side

 

Sorry, I guess you need to add "debug ip bgp keepalives" as so you see that
it is sent at around about the same time as his message.

Otherwise it looks pretty normal.

 

Sorry, I guess you need to add "debug ip bgp keepalives" as so you see that

I've tried another debug session with 'debug ip bgp keepalives' on... I canpost the results, but basically while the working session shows keepalivessent and received, the problem session shows only keepalives sent, none received (which is what they're claiming they're seeing on the ISP end).