2600XM, Frame Relay, and High CPU Utilization

Discussion in 'Cisco' started by Donald Zelenak Jr., Dec 4, 2003.

  1. Hi Group..t

    I've got a Cisco 2610XM series router.. I think it's a 2610... Either way,
    it's the bottom of the line of the new XM series routers.

    Anyhow..

    It's at a datacenter and it has about 30-35 frame relay PVCs that terminate
    on it. This is bascially a hub and spoke network design, with this 2600 at
    the core and a mixture of routers at the remote sites (1720's, some old
    IBM/Synch boxes, etc..). The FR circuit comes in on a T1, to the internal
    DSU in the 2600.

    Lately, I've had sites that are complaining that they can't communicate with
    the AS/400's at the core, or other times it will freeze for a few minutes
    then come back to life. I've checked just about everything at the remote
    sites and everything seems normal. I started to suspect that the router at
    the datacenter may have something to do with it.

    This router is fairly new (about 6 weeks). It replaced some ancient Synch
    router that was at the datacenter that had a suspect bad T1 interface card.

    During the day, I can telnet into the router and sometimes it's fine, and
    sometimes I can't type commands into it because it's not responsive.
    Sometimes I can log in, but that's about it. It will just eventually time
    the session out and disconnect me. Doing a "sho processes cpu" shows that
    the 5 minute usage is like 99%/52%. Most processes are taking up little to
    no CPU except for the IP Cache Ager and the IP Input process. The IP Cache
    Ager never takes up more than 20-25%, and the IP Input process has taken up
    40-50% at it's peak. Inputting a "Scheduler Int 500" command didn't help
    either. When the router is bogged down like this, pings do not reply and
    come back maybe 1 out of 4 times. For no apparent reason, the router will
    just "snap out of it": and start behaving normally for a few minutes, then
    it will go back to 99% CPU.

    I'm beginning to wonder if I've just got too many PVCs for this router to
    handle. I realize this is a LOT of PVCs to be coming in on a single T1, but
    it wasn't my doing. I also think that if the Synch box was handling this
    load, the 2600XM should be able to do it with half it's CPU tied behind it's
    back.

    The network does use RIPV1 as the routing protocol. Nothing else fancy is
    going on. The configuration takes up less than a page.

    I'm kind of at my wits end here with this. This network is going away either
    at the end of this month or the middle of next, but until then I'd like to
    find a resolution for this issue. If not for the sake of the users out at
    the branch locations, for my sanity as well. I'd like to know what may be
    going on so I can fix it next time if the problem were to occour again.

    The router has IOS 12.2T, IP Only featureset. Out of box RAM configuration,
    I want to say 64MB. It's late and I don't have my VPN access on my laptop or
    I'd give you a console dump.

    Any ideas or suggestions that anyone has will be greatly appreciated.

    Thanks,
    Don
    Donald Zelenak Jr., Dec 4, 2003
    #1
    1. Advertising

  2. Hello, Donald!
    You wrote on Thu, 04 Dec 2003 05:44:02 GMT:

    DZJ> Sometimes I can log in, but that's about it. It will just
    DZJ> eventually time the session out and disconnect me. Doing a
    DZJ> "sho processes cpu" shows that the 5 minute usage is like
    DZJ> 99%/52%. Most processes are taking up little to no CPU except
    DZJ> for the IP Cache Ager and the IP Input process. The IP Cache
    DZJ> Ager never takes up more than 20-25%, and the IP Input
    DZJ> process has taken up 40-50% at it's peak. Inputting a
    DZJ> "Scheduler Int 500" command didn't help either. When the
    DZJ> router is bogged down like this, pings do not reply and come
    DZJ> back maybe 1 out of 4 times. For no apparent reason, the
    DZJ> router will just "snap out of it": and start behaving
    DZJ> normally for a few minutes, then it will go back to 99% CPU.

    Is CEF enabled on this box?

    Check for worm infected systems on site - you might be getting a hell lot of
    ICMP traffic from Nachi/etc.

    With best regards,
    Andrey.
    Andrey Tarasov, Dec 4, 2003
    #2
    1. Advertising

  3. "Andrey Tarasov" <> wrote in message
    news:bqmkgc$pl3$...
    > Hello, Donald!
    > You wrote on Thu, 04 Dec 2003 05:44:02 GMT:
    >
    > DZJ> Sometimes I can log in, but that's about it. It will just
    > DZJ> eventually time the session out and disconnect me. Doing a
    > DZJ> "sho processes cpu" shows that the 5 minute usage is like
    > DZJ> 99%/52%. Most processes are taking up little to no CPU except
    > DZJ> for the IP Cache Ager and the IP Input process. The IP Cache
    > DZJ> Ager never takes up more than 20-25%, and the IP Input
    > DZJ> process has taken up 40-50% at it's peak. Inputting a
    > DZJ> "Scheduler Int 500" command didn't help either. When the
    > DZJ> router is bogged down like this, pings do not reply and come
    > DZJ> back maybe 1 out of 4 times. For no apparent reason, the
    > DZJ> router will just "snap out of it": and start behaving
    > DZJ> normally for a few minutes, then it will go back to 99% CPU.
    >
    > Is CEF enabled on this box?
    >
    > Check for worm infected systems on site - you might be getting a hell lot

    of
    > ICMP traffic from Nachi/etc.
    >
    > With best regards,
    > Andrey.
    >


    Andrey,

    Thanks for the response..

    CEF is not enabled on this router.

    You may be correct with the worm suggestion. This client has a lot of PCs,
    and I know they have been battling various variants of the Agobot worm.

    I'm not sure how I can verify this, as I can only access the router via VPN
    today. Doing a general traffic debug would probably kill the connection.

    Regards,
    - Don
    Donald Zelenak Jr., Dec 4, 2003
    #3
  4. Hello, Donald!
    You wrote on Thu, 04 Dec 2003 16:31:19 GMT:

    DZJ> CEF is not enabled on this router.

    You may try to enable it. That should decrease CPU load.

    DZJ> You may be correct with the worm suggestion. This client has
    DZJ> a lot of PCs, and I know they have been battling various
    DZJ> variants of the Agobot worm.

    DZJ> I'm not sure how I can verify this, as I can only access the
    DZJ> router via VPN today. Doing a general traffic debug would
    DZJ> probably kill the connection.

    You can do that by enabling ip accounting on the router and then checking it
    periodically for multiple entries with the same source IP and very low byte
    counter per entry.

    With best regards,
    Andrey.
    Andrey Tarasov, Dec 5, 2003
    #4
  5. Donald Zelenak Jr.

    MC Guest

    You may want to configure an ACL temporarily to drop any ICMP traffic to
    test if that is it, or at least drop ICMP traffic to the router interfaces
    themselves.


    "Andrey Tarasov" <> wrote in message
    news:bqohve$bab$...
    > Hello, Donald!
    > You wrote on Thu, 04 Dec 2003 16:31:19 GMT:
    >
    > DZJ> CEF is not enabled on this router.
    >
    > You may try to enable it. That should decrease CPU load.
    >
    > DZJ> You may be correct with the worm suggestion. This client has
    > DZJ> a lot of PCs, and I know they have been battling various
    > DZJ> variants of the Agobot worm.
    >
    > DZJ> I'm not sure how I can verify this, as I can only access the
    > DZJ> router via VPN today. Doing a general traffic debug would
    > DZJ> probably kill the connection.
    >
    > You can do that by enabling ip accounting on the router and then checking

    it
    > periodically for multiple entries with the same source IP and very low

    byte
    > counter per entry.
    >
    > With best regards,
    > Andrey.
    >
    MC, Dec 5, 2003
    #5
  6. Problem found..

    Nachi.A is spreading like wildfire on their old, non updated clients. I went
    to the core and did some ICMP debugging, and clients are flooding the router
    sending echo requests to the entire class B.

    On the remote sties fortunate enough to have Cisco gear, I've implemented
    Access Lists to prevent all the ICMP traffic from getting to the core until
    we can get techs out to fix the problem. The other sites I had them either
    shut off the infected clients or they are just going to have to deal until
    the clients are cleaned.

    Also getting CPUHOG messages on the ARP Input. I can only assume the router
    is trying to ARP for all the hosts that the Nachi infected clients are
    trying to contact.

    Thanks for all the advice.

    - Don


    "MC" <> wrote in message
    news:KS7Ab.629$...
    > You may want to configure an ACL temporarily to drop any ICMP traffic to
    > test if that is it, or at least drop ICMP traffic to the router interfaces
    > themselves.
    >
    >
    > "Andrey Tarasov" <> wrote in message
    > news:bqohve$bab$...
    > > Hello, Donald!
    > > You wrote on Thu, 04 Dec 2003 16:31:19 GMT:
    > >
    > > DZJ> CEF is not enabled on this router.
    > >
    > > You may try to enable it. That should decrease CPU load.
    > >
    > > DZJ> You may be correct with the worm suggestion. This client has
    > > DZJ> a lot of PCs, and I know they have been battling various
    > > DZJ> variants of the Agobot worm.
    > >
    > > DZJ> I'm not sure how I can verify this, as I can only access the
    > > DZJ> router via VPN today. Doing a general traffic debug would
    > > DZJ> probably kill the connection.
    > >
    > > You can do that by enabling ip accounting on the router and then

    checking
    > it
    > > periodically for multiple entries with the same source IP and very low

    > byte
    > > counter per entry.
    > >
    > > With best regards,
    > > Andrey.
    > >

    >
    >
    Donald Zelenak Jr., Dec 6, 2003
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Miguel Moreno

    high CPU utilization

    Miguel Moreno, Jan 28, 2004, in forum: Cisco
    Replies:
    4
    Views:
    1,309
    Miguel Moreno
    Jan 30, 2004
  2. Brian R. Jack
    Replies:
    1
    Views:
    3,215
    Øystein Berg
    Sep 15, 2004
  3. Brian R. Jack
    Replies:
    8
    Views:
    1,658
    Hansang Bae
    Sep 14, 2004
  4. Bancal
    Replies:
    3
    Views:
    2,782
  5. Jimmy
    Replies:
    3
    Views:
    1,105
    w1llr0ut34f00d
    Oct 20, 2005
Loading...

Share This Page