Velocity Reviews

Velocity Reviews (http://www.velocityreviews.com/forums/index.php)
-   Cisco (http://www.velocityreviews.com/forums/f27-cisco.html)
-   -   2600XM, Frame Relay, and High CPU Utilization (http://www.velocityreviews.com/forums/t30700-2600xm-frame-relay-and-high-cpu-utilization.html)

Donald Zelenak Jr. 12-04-2003 05:44 AM

2600XM, Frame Relay, and High CPU Utilization
 
Hi Group..t

I've got a Cisco 2610XM series router.. I think it's a 2610... Either way,
it's the bottom of the line of the new XM series routers.

Anyhow..

It's at a datacenter and it has about 30-35 frame relay PVCs that terminate
on it. This is bascially a hub and spoke network design, with this 2600 at
the core and a mixture of routers at the remote sites (1720's, some old
IBM/Synch boxes, etc..). The FR circuit comes in on a T1, to the internal
DSU in the 2600.

Lately, I've had sites that are complaining that they can't communicate with
the AS/400's at the core, or other times it will freeze for a few minutes
then come back to life. I've checked just about everything at the remote
sites and everything seems normal. I started to suspect that the router at
the datacenter may have something to do with it.

This router is fairly new (about 6 weeks). It replaced some ancient Synch
router that was at the datacenter that had a suspect bad T1 interface card.

During the day, I can telnet into the router and sometimes it's fine, and
sometimes I can't type commands into it because it's not responsive.
Sometimes I can log in, but that's about it. It will just eventually time
the session out and disconnect me. Doing a "sho processes cpu" shows that
the 5 minute usage is like 99%/52%. Most processes are taking up little to
no CPU except for the IP Cache Ager and the IP Input process. The IP Cache
Ager never takes up more than 20-25%, and the IP Input process has taken up
40-50% at it's peak. Inputting a "Scheduler Int 500" command didn't help
either. When the router is bogged down like this, pings do not reply and
come back maybe 1 out of 4 times. For no apparent reason, the router will
just "snap out of it": and start behaving normally for a few minutes, then
it will go back to 99% CPU.

I'm beginning to wonder if I've just got too many PVCs for this router to
handle. I realize this is a LOT of PVCs to be coming in on a single T1, but
it wasn't my doing. I also think that if the Synch box was handling this
load, the 2600XM should be able to do it with half it's CPU tied behind it's
back.

The network does use RIPV1 as the routing protocol. Nothing else fancy is
going on. The configuration takes up less than a page.

I'm kind of at my wits end here with this. This network is going away either
at the end of this month or the middle of next, but until then I'd like to
find a resolution for this issue. If not for the sake of the users out at
the branch locations, for my sanity as well. I'd like to know what may be
going on so I can fix it next time if the problem were to occour again.

The router has IOS 12.2T, IP Only featureset. Out of box RAM configuration,
I want to say 64MB. It's late and I don't have my VPN access on my laptop or
I'd give you a console dump.

Any ideas or suggestions that anyone has will be greatly appreciated.

Thanks,
Don




Andrey Tarasov 12-04-2003 06:34 AM

Re: 2600XM, Frame Relay, and High CPU Utilization
 
Hello, Donald!
You wrote on Thu, 04 Dec 2003 05:44:02 GMT:

DZJ> Sometimes I can log in, but that's about it. It will just
DZJ> eventually time the session out and disconnect me. Doing a
DZJ> "sho processes cpu" shows that the 5 minute usage is like
DZJ> 99%/52%. Most processes are taking up little to no CPU except
DZJ> for the IP Cache Ager and the IP Input process. The IP Cache
DZJ> Ager never takes up more than 20-25%, and the IP Input
DZJ> process has taken up 40-50% at it's peak. Inputting a
DZJ> "Scheduler Int 500" command didn't help either. When the
DZJ> router is bogged down like this, pings do not reply and come
DZJ> back maybe 1 out of 4 times. For no apparent reason, the
DZJ> router will just "snap out of it": and start behaving
DZJ> normally for a few minutes, then it will go back to 99% CPU.

Is CEF enabled on this box?

Check for worm infected systems on site - you might be getting a hell lot of
ICMP traffic from Nachi/etc.

With best regards,
Andrey.


Donald Zelenak Jr. 12-04-2003 04:31 PM

Re: 2600XM, Frame Relay, and High CPU Utilization
 

"Andrey Tarasov" <andyvt@email.com> wrote in message
news:bqmkgc$pl3$1@news.aha.ru...
> Hello, Donald!
> You wrote on Thu, 04 Dec 2003 05:44:02 GMT:
>
> DZJ> Sometimes I can log in, but that's about it. It will just
> DZJ> eventually time the session out and disconnect me. Doing a
> DZJ> "sho processes cpu" shows that the 5 minute usage is like
> DZJ> 99%/52%. Most processes are taking up little to no CPU except
> DZJ> for the IP Cache Ager and the IP Input process. The IP Cache
> DZJ> Ager never takes up more than 20-25%, and the IP Input
> DZJ> process has taken up 40-50% at it's peak. Inputting a
> DZJ> "Scheduler Int 500" command didn't help either. When the
> DZJ> router is bogged down like this, pings do not reply and come
> DZJ> back maybe 1 out of 4 times. For no apparent reason, the
> DZJ> router will just "snap out of it": and start behaving
> DZJ> normally for a few minutes, then it will go back to 99% CPU.
>
> Is CEF enabled on this box?
>
> Check for worm infected systems on site - you might be getting a hell lot

of
> ICMP traffic from Nachi/etc.
>
> With best regards,
> Andrey.
>


Andrey,

Thanks for the response..

CEF is not enabled on this router.

You may be correct with the worm suggestion. This client has a lot of PCs,
and I know they have been battling various variants of the Agobot worm.

I'm not sure how I can verify this, as I can only access the router via VPN
today. Doing a general traffic debug would probably kill the connection.

Regards,
- Don




Andrey Tarasov 12-05-2003 12:03 AM

Re: 2600XM, Frame Relay, and High CPU Utilization
 
Hello, Donald!
You wrote on Thu, 04 Dec 2003 16:31:19 GMT:

DZJ> CEF is not enabled on this router.

You may try to enable it. That should decrease CPU load.

DZJ> You may be correct with the worm suggestion. This client has
DZJ> a lot of PCs, and I know they have been battling various
DZJ> variants of the Agobot worm.

DZJ> I'm not sure how I can verify this, as I can only access the
DZJ> router via VPN today. Doing a general traffic debug would
DZJ> probably kill the connection.

You can do that by enabling ip accounting on the router and then checking it
periodically for multiple entries with the same source IP and very low byte
counter per entry.

With best regards,
Andrey.


MC 12-05-2003 10:47 PM

Re: 2600XM, Frame Relay, and High CPU Utilization
 
You may want to configure an ACL temporarily to drop any ICMP traffic to
test if that is it, or at least drop ICMP traffic to the router interfaces
themselves.


"Andrey Tarasov" <andyvt@email.com> wrote in message
news:bqohve$bab$1@news.aha.ru...
> Hello, Donald!
> You wrote on Thu, 04 Dec 2003 16:31:19 GMT:
>
> DZJ> CEF is not enabled on this router.
>
> You may try to enable it. That should decrease CPU load.
>
> DZJ> You may be correct with the worm suggestion. This client has
> DZJ> a lot of PCs, and I know they have been battling various
> DZJ> variants of the Agobot worm.
>
> DZJ> I'm not sure how I can verify this, as I can only access the
> DZJ> router via VPN today. Doing a general traffic debug would
> DZJ> probably kill the connection.
>
> You can do that by enabling ip accounting on the router and then checking

it
> periodically for multiple entries with the same source IP and very low

byte
> counter per entry.
>
> With best regards,
> Andrey.
>




Donald Zelenak Jr. 12-06-2003 01:18 AM

Re: 2600XM, Frame Relay, and High CPU Utilization
 
Problem found..

Nachi.A is spreading like wildfire on their old, non updated clients. I went
to the core and did some ICMP debugging, and clients are flooding the router
sending echo requests to the entire class B.

On the remote sties fortunate enough to have Cisco gear, I've implemented
Access Lists to prevent all the ICMP traffic from getting to the core until
we can get techs out to fix the problem. The other sites I had them either
shut off the infected clients or they are just going to have to deal until
the clients are cleaned.

Also getting CPUHOG messages on the ARP Input. I can only assume the router
is trying to ARP for all the hosts that the Nachi infected clients are
trying to contact.

Thanks for all the advice.

- Don


"MC" <mwclarke1@yahoo.com> wrote in message
news:KS7Ab.629$IF.83@bignews4.bellsouth.net...
> You may want to configure an ACL temporarily to drop any ICMP traffic to
> test if that is it, or at least drop ICMP traffic to the router interfaces
> themselves.
>
>
> "Andrey Tarasov" <andyvt@email.com> wrote in message
> news:bqohve$bab$1@news.aha.ru...
> > Hello, Donald!
> > You wrote on Thu, 04 Dec 2003 16:31:19 GMT:
> >
> > DZJ> CEF is not enabled on this router.
> >
> > You may try to enable it. That should decrease CPU load.
> >
> > DZJ> You may be correct with the worm suggestion. This client has
> > DZJ> a lot of PCs, and I know they have been battling various
> > DZJ> variants of the Agobot worm.
> >
> > DZJ> I'm not sure how I can verify this, as I can only access the
> > DZJ> router via VPN today. Doing a general traffic debug would
> > DZJ> probably kill the connection.
> >
> > You can do that by enabling ip accounting on the router and then

checking
> it
> > periodically for multiple entries with the same source IP and very low

> byte
> > counter per entry.
> >
> > With best regards,
> > Andrey.
> >

>
>





All times are GMT. The time now is 03:57 AM.

Powered by vBulletin®. Copyright ©2000 - 2014, vBulletin Solutions, Inc.
SEO by vBSEO ©2010, Crawlability, Inc.