How can a Cat 6500 cause an instantaneous router lockup?

Discussion in 'Cisco' started by John Caruso, Sep 8, 2004.

  1. John Caruso

    John Caruso Guest

    I ran into a very odd situation at a client's site today. Their internal
    network has two paired 2621 routers, sharing HSRP duties and providing
    redundant networking connections out of the building (T1 and GRE tunnels).
    For various reasons, the internal network switches are all fanned out from
    a central Cat 6500 which is under the control of another organization.
    The routers also connect to this Cat 6500.

    Today their entire network went down, and after some troubleshooting it
    became apparent that the Cat 6500 was the cause. But here's the odd part:
    when the 2621s were disconnected from the Cat 6500, they would work fine.
    But the second that they were connected to the Cat 6500 again they would
    appear to lock up completely: T1s and GRE tunnels would go down, telnet
    sessions would hang, and--most baffling--even a serial console session
    would hang. The second the Cat 6500 connection was broken again, the
    router would return to normal.

    Unfortunately since I had no access to the Cat 6500 I couldn't see what
    was happening there, and since the routers would hang while they were
    connected to the Cat 6500 I couldn't monitor them either to see what was
    going on. But I can't think of anything the Cat 6500 could have been
    doing that would cause an instantaneous, total lockup of the 2621 routers.
    Does anyone have any idea what it might be? My only guess is a bridging
    loop (which makes sense since some of the fanout switches were showing high
    activity levels and 99% CPU when they were attached to the Cat 6500), but
    it just doesn't seem possible that even that could have caused the instant
    lockups on the routers.

    - John
    John Caruso, Sep 8, 2004
    #1
    1. Advertising

  2. John Caruso

    Ivan Ostres Guest

    In article <>,
    says...
    > Unfortunately since I had no access to the Cat 6500 I couldn't see what
    > was happening there, and since the routers would hang while they were
    > connected to the Cat 6500 I couldn't monitor them either to see what was
    > going on. But I can't think of anything the Cat 6500 could have been
    > doing that would cause an instantaneous, total lockup of the 2621 routers.
    > Does anyone have any idea what it might be? My only guess is a bridging
    > loop (which makes sense since some of the fanout switches were showing high
    > activity levels and 99% CPU when they were attached to the Cat 6500), but
    > it just doesn't seem possible that even that could have caused the instant
    > lockups on the routers.
    >
    >


    I looks like a STP loop or routing loop.

    --
    -Ivan.

    *** Use Rot13 to see my eMail address ***
    Ivan Ostres, Sep 8, 2004
    #2
    1. Advertising

  3. John Caruso

    John Caruso Guest

    In article <>, Ivan Ostres wrote:
    > In article <>,
    > says...
    >> Unfortunately since I had no access to the Cat 6500 I couldn't see what
    >> was happening there, and since the routers would hang while they were
    >> connected to the Cat 6500 I couldn't monitor them either to see what was
    >> going on. But I can't think of anything the Cat 6500 could have been
    >> doing that would cause an instantaneous, total lockup of the 2621 routers.
    >> Does anyone have any idea what it might be? My only guess is a bridging
    >> loop (which makes sense since some of the fanout switches were showing high
    >> activity levels and 99% CPU when they were attached to the Cat 6500), but
    >> it just doesn't seem possible that even that could have caused the instant
    >> lockups on the routers.

    >
    > I looks like a STP loop or routing loop.


    I've seen a routing loop on one of the routers in this configuration before
    (during initial installation, before a required Null0 route was added to
    the router to prevent loops), and it caused a very gradual degradation in
    performance--not a lockup, immediate or otherwise.

    And as I said, I don't see how a bridging loop could have caused the routers
    attached to that network to lock up instantaneously (including the console).
    I suppose if the looping packets in question required process switching for
    some reason, that might have done it, especially since these were 2621s.
    But here's the show int stat output for that interface on one of the routers:

    FastEthernet0/0
    Switching path Pkts In Chars In Pkts Out Chars Out
    Processor 99922 10706243 79522 7699110
    Route cache 1831313 348192875 1182930 325567073
    Total 1931235 358899118 1262452 333266183

    So not very much process switching occurred on Fast0/0. Now, the router
    was rebooted during the problem and it also wasn't attached to the 6509
    very often after that (since we'd identified the 6509 as the source of
    unhappiness), but it still doesn't seem likely that a relatively small
    number of process-switched packets could have caused the 2621 to lock up
    as it did.

    - John
    John Caruso, Sep 8, 2004
    #3
  4. John Caruso

    Hansang Bae Guest

    In article <>,
    says...
    > I ran into a very odd situation at a client's site today. Their internal
    > network has two paired 2621 routers, sharing HSRP duties and providing
    > redundant networking connections out of the building (T1 and GRE tunnels).
    > For various reasons, the internal network switches are all fanned out from
    > a central Cat 6500 which is under the control of another organization.
    > The routers also connect to this Cat 6500.

    [snip: 2600's locking up when connected to 6500s]
    > Does anyone have any idea what it might be? My only guess is a bridging
    > loop (which makes sense since some of the fanout switches were showing high
    > activity levels and 99% CPU when they were attached to the Cat 6500), but
    > it just doesn't seem possible that even that could have caused the instant
    > lockups on the routers.


    It can lock it up quite easily. My guess is someone "thought" they
    could save some frames by disabling STP.


    --

    hsb

    "Somehow I imagined this experience would be more rewarding" Calvin
    *************** USE ROT13 TO SEE MY EMAIL ADDRESS ****************
    ********************************************************************
    Due to the volume of email that I receive, I may not not be able to
    reply to emails sent to my account. Please post a followup instead.
    ********************************************************************
    Hansang Bae, Sep 9, 2004
    #4
  5. John Caruso

    Ivan Ostres Guest

    In article <>,
    says...
    >
    > So not very much process switching occurred on Fast0/0. Now, the router
    > was rebooted during the problem and it also wasn't attached to the 6509
    > very often after that (since we'd identified the 6509 as the source of
    > unhappiness), but it still doesn't seem likely that a relatively small
    > number of process-switched packets could have caused the 2621 to lock up
    > as it did.
    >
    >


    I've seen much stronger routers than 26xx to lock up during STP loop. A
    look at 'sh int eth XX' and 'sh buffers' would be nice to see. If you're
    using cef, the table renewal would be nice to see too.

    --
    -Ivan.

    *** Use Rot13 to see my eMail address ***
    Ivan Ostres, Sep 9, 2004
    #5
  6. John Caruso

    PES Guest

    "John Caruso" <> wrote in message
    news:...
    > In article <>, Ivan Ostres
    > wrote:
    >> In article <>,
    >> says...
    >>> Unfortunately since I had no access to the Cat 6500 I couldn't see what
    >>> was happening there, and since the routers would hang while they were
    >>> connected to the Cat 6500 I couldn't monitor them either to see what was
    >>> going on. But I can't think of anything the Cat 6500 could have been
    >>> doing that would cause an instantaneous, total lockup of the 2621
    >>> routers.
    >>> Does anyone have any idea what it might be? My only guess is a bridging
    >>> loop (which makes sense since some of the fanout switches were showing
    >>> high
    >>> activity levels and 99% CPU when they were attached to the Cat 6500),
    >>> but
    >>> it just doesn't seem possible that even that could have caused the
    >>> instant
    >>> lockups on the routers.

    >>
    >> I looks like a STP loop or routing loop.

    >
    > I've seen a routing loop on one of the routers in this configuration
    > before
    > (during initial installation, before a required Null0 route was added to
    > the router to prevent loops), and it caused a very gradual degradation in
    > performance--not a lockup, immediate or otherwise.
    >
    > And as I said, I don't see how a bridging loop could have caused the
    > routers
    > attached to that network to lock up instantaneously (including the
    > console).
    > I suppose if the looping packets in question required process switching
    > for
    > some reason, that might have done it, especially since these were 2621s.
    > But here's the show int stat output for that interface on one of the
    > routers:
    >
    > FastEthernet0/0
    > Switching path Pkts In Chars In Pkts Out Chars Out
    > Processor 99922 10706243 79522 7699110
    > Route cache 1831313 348192875 1182930 325567073
    > Total 1931235 358899118 1262452 333266183
    >
    > So not very much process switching occurred on Fast0/0. Now, the router
    > was rebooted during the problem and it also wasn't attached to the 6509
    > very often after that (since we'd identified the 6509 as the source of
    > unhappiness), but it still doesn't seem likely that a relatively small
    > number of process-switched packets could have caused the 2621 to lock up
    > as it did.
    >
    > - John


    A bridging loop is much worse than a routing loop. In bridging, there is no
    ttl on a packet, therefore every packet lives for ever. In a routing loop,
    every time the packet goes through a router the ttl is decremented.
    Therefore packets in a routing loop will loop a maximum of 255 times.
    PES, Sep 9, 2004
    #6
  7. John Caruso

    AnyBody43 Guest

    Hansang Bae <> wrote
    > says...
    > > I ran into a very odd situation at a client's site today. Their internal
    > > network has two paired 2621 routers, sharing HSRP duties and providing
    > > redundant networking connections out of the building (T1 and GRE tunnels).
    > > For various reasons, the internal network switches are all fanned out from
    > > a central Cat 6500 which is under the control of another organization.
    > > The routers also connect to this Cat 6500.

    > [snip: 2600's locking up when connected to 6500s]
    > > Does anyone have any idea what it might be? My only guess is a bridging
    > > loop (which makes sense since some of the fanout switches were showing high
    > > activity levels and 99% CPU when they were attached to the Cat 6500), but
    > > it just doesn't seem possible that even that could have caused the instant
    > > lockups on the routers.

    >
    > It can lock it up quite easily. My guess is someone "thought" they
    > could save some frames by disabling STP.


    A 6500 can clearly forward at wire rate out of any (and all at
    the same time too) port. Imagine a loop in say
    a GB Ethernet path creating traffic at a level of a substantial
    proportion of the bandwidth (well > 100Mbps). If the traffic happens to
    be broadcast traffic (L2 say) then wire rate broadcasts will be sent out
    of all 10 and 100 Mbps ports in the VLAN.

    All of these frames will have to be processed by all receiving devices.
    In the case of a 2600, it will die. Period.

    The level of traffic will build up very quickly. Well, instantly, as
    far as human perception is concerned. I have created such loops in the
    lab and the effects are quite dramatic.
    AnyBody43, Sep 9, 2004
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Gary
    Replies:
    2
    Views:
    695
    Arnold Nipper
    Dec 2, 2005
  2. Skip M

    Can CF Card cause Lockup?

    Skip M, Dec 5, 2004, in forum: Digital Photography
    Replies:
    20
    Views:
    573
    Jim Morgan
    Dec 7, 2004
  3. Replies:
    0
    Views:
    954
  4. Replies:
    0
    Views:
    502
  5. Brett Cooper

    How can I tell what CAT my UTP Cat. 3, 4, 5 is?

    Brett Cooper, Jun 29, 2004, in forum: NZ Computing
    Replies:
    2
    Views:
    395
    Allistar
    Jun 29, 2004
Loading...

Share This Page