Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Computing > Cisco > 2650 IP Input High CPU Utilization Problem

Reply
Thread Tools

2650 IP Input High CPU Utilization Problem

 
 
Brian R. Jack
Guest
Posts: n/a
 
      09-10-2004
I have a 2650 deployed as an edge device in a distributed internet
provisioning environment. The router has 2 parallel per-packet load-shared
T1 connections with internal DSU WICs and only the single fast ethernet
interface for the LAN side that is being trunked to a Cat 4006. The config
has been in place for over 2 years without a hitch until a recent ISP change
that has created a huge problem. Basically the IP Input process
incrementally consumes more and more proc time over about 2 hours to the
point where I have to reload the router in order for traffic to pass
through.

Specifics:

IOS 12.1(5)T10
64MB RAM

25 fast ethernet subinterfaces mapped to VLANs hosted on a Cat4006, all subs
have private IP subnets assigned to them NAT is performed by the router.
There are over 25 static mappings plus a dynamic pool consisting of 7 IP
addresses

2 PPP serial connections to a carrier provided T1 channel bank. T1's are
running clean and clear, no service-module slips or framing errors noted.
Serial connections are also clear with no CRC, no i/o drops, no errors etc.
Load is not out of the ordinary for either connection (both run at about
15 - 40% tx/rx load).

CEF and fast switching is enabled on all interfaces
Netflow accounting is enabled
CEF per-packet load-sharing is enabled on serial interfaces

I have worked through all of the cisco docs regarding high cpu utilization
and found nothing out of the ordinary. Initially I concerned that this
might be an issue caused by some NAT changes that had to be put in effect
after the ISP change but my traffic levels just don't substantiate that.

At this point I am looking for any ideas that anyone else might have
regarding this issue.

Thanks,
Brian R. Jack



 
Reply With Quote
 
 
 
 
Andrey Tarasov
Guest
Posts: n/a
 
      09-10-2004
Hello, Brian!
You wrote on Fri, 10 Sep 2004 10:48:11 -0700:

BRJ> I have a 2650 deployed as an edge device in a distributed
BRJ> internet provisioning environment. The router has 2 parallel
BRJ> per-packet load-shared T1 connections with internal DSU WICs and
BRJ> only the single fast ethernet interface for the LAN side that is
BRJ> being trunked to a Cat 4006. The config has been in place for
BRJ> over 2 years without a hitch until a recent ISP change that has
BRJ> created a huge problem.

BRJ> Basically the IP Input process incrementally consumes more and more proc
BRJ> time over about 2 hours to the point where I have to reload the router in
BRJ> order for traffic to pass through.

BRJ> 25 fast ethernet subinterfaces mapped to VLANs hosted on a
BRJ> Cat4006, all subs have private IP subnets assigned to them NAT
BRJ> is performed by the router.

BRJ> CEF and fast switching is enabled on all interfaces

Well, IP Input process loading CPU and CEF doesn't fit together. Can you
provide output from show interface switching? My guess is that there is a lot of
traffic between VLAN's and CEF on same interface is not enabled or not working
properly.

With best regards,
Andrey.

 
Reply With Quote
 
 
 
 
Brian R. Jack
Guest
Posts: n/a
 
      09-10-2004
Here are the interface switching stats.

-Brian

FastEthernet0/0
Throttle count 0
Drops RP 148843 SP 0
SPD Flushes Fast 0 SSE 0
SPD Aggress Fast 0
SPD Priority Inputs 18 Drops 0

Protocol Path Pkts In Chars In Pkts Out Chars Out
Other Process 7336 790577 1907 114420
Cache misses 0
Fast 0 0 96 30706
Auton/SSE 0 0 0 0
IP Process 1318821 193519884 175635 19038357
Cache misses 0
Fast 1530740 784982950 2547933 2054973479
Auton/SSE 0 0 0 0
ARP Process 7354 441676 14793 946752
Cache misses 0
Fast 0 0 0 0
Auton/SSE 0 0 0 0
Serial0/0
Throttle count 0
Drops RP 74 SP 0
SPD Flushes Fast 0 SSE 0
SPD Aggress Fast 0
SPD Priority Inputs 0 Drops 0

Protocol Path Pkts In Chars In Pkts Out Chars Out
Other Process 2 28 179 2862
Cache misses 0
Fast 0 0 538 106456
Auton/SSE 0 0 0 0
IP Process 102607 9052386 671590 89484207
Cache misses 0
Fast 370718 289312539 219171 51002626
Auton/SSE 0 0 0 0
Serial0/1
Throttle count 0
Drops RP 34 SP 0
SPD Flushes Fast 0 SSE 0
SPD Aggress Fast 0
SPD Priority Inputs 0 Drops 0

Protocol Path Pkts In Chars In Pkts Out Chars Out
Other Process 2 28 179 2862
Cache misses 0
Fast 0 0 532 99505
Auton/SSE 0 0 0 0
IP Process 164586 14412378 672105 89645741
Cache misses 0
Fast 1089543 1061566468 219969 51049450
Auton/SSE 0 0 0 0


"Andrey Tarasov" <(E-Mail Removed)> wrote in message
news:chsrp4$80u$(E-Mail Removed)...
> Hello, Brian!
> You wrote on Fri, 10 Sep 2004 10:48:11 -0700:
>
> BRJ> I have a 2650 deployed as an edge device in a distributed
> BRJ> internet provisioning environment. The router has 2 parallel
> BRJ> per-packet load-shared T1 connections with internal DSU WICs and
> BRJ> only the single fast ethernet interface for the LAN side that is
> BRJ> being trunked to a Cat 4006. The config has been in place for
> BRJ> over 2 years without a hitch until a recent ISP change that has
> BRJ> created a huge problem.
>
> BRJ> Basically the IP Input process incrementally consumes more and more
> proc
> BRJ> time over about 2 hours to the point where I have to reload the
> router in
> BRJ> order for traffic to pass through.
>
> BRJ> 25 fast ethernet subinterfaces mapped to VLANs hosted on a
> BRJ> Cat4006, all subs have private IP subnets assigned to them NAT
> BRJ> is performed by the router.
>
> BRJ> CEF and fast switching is enabled on all interfaces
>
> Well, IP Input process loading CPU and CEF doesn't fit together. Can you
> provide output from show interface switching? My guess is that there is a
> lot of
> traffic between VLAN's and CEF on same interface is not enabled or not
> working
> properly.
>
> With best regards,
> Andrey.
>



 
Reply With Quote
 
Ivan Ostres
Guest
Posts: n/a
 
      09-11-2004
In article <Emn0d.23$(E-Mail Removed)>, http://www.velocityreviews.com/forums/(E-Mail Removed)
says...
> Here are the interface switching stats.
>
> -Brian
>
> FastEthernet0/0
> IP Process 1318821 193519884 175635 19038357
> Cache misses 0
> Fast 1530740 784982950 2547933 2054973479
> Serial0/0


> IP Process 102607 9052386 671590 89484207
> Cache misses 0
> Fast 370718 289312539 219171 51002626


As you can see here, half of your packets on FE0/0 are process switched
and one third of packets on S0/0 are process switched too. To conclude
reasons why is this going on you should post your config.


--
-Ivan.

*** Use Rot13 to see my eMail address ***
 
Reply With Quote
 
David
Guest
Posts: n/a
 
      09-13-2004
Ivan Ostres <(E-Mail Removed)> wrote in message news:<(E-Mail Removed) et>...
> In article <Emn0d.23$(E-Mail Removed)>, (E-Mail Removed)
> says...
> > Here are the interface switching stats.
> >
> > -Brian
> >
> > FastEthernet0/0
> > IP Process 1318821 193519884 175635 19038357
> > Cache misses 0
> > Fast 1530740 784982950 2547933 2054973479
> > Serial0/0

>
> > IP Process 102607 9052386 671590 89484207
> > Cache misses 0
> > Fast 370718 289312539 219171 51002626

>
> As you can see here, half of your packets on FE0/0 are process switched
> and one third of packets on S0/0 are process switched too. To conclude
> reasons why is this going on you should post your config.


How do you see that on FE0/0 half of packets are process switched
 
Reply With Quote
 
Ivan Ostres
Guest
Posts: n/a
 
      09-13-2004
In article <(E-Mail Removed) >,
(E-Mail Removed) says...
> How do you see that on FE0/0 half of packets are process switched
>


This is the original output from 'show interfaces switching'. I've cut
it a little bit.


> FastEthernet0/0
> Throttle count 0
> Drops RP 148843 SP 0
> SPD Flushes Fast 0 SSE 0
> SPD Aggress Fast 0
> SPD Priority Inputs 18 Drops 0
>
> Protocol Path Pkts In Chars In Pkts Out Chars Out
>
> Other Process 7336 790577 1907 114420
> Cache misses 0
> Fast 0 0 96 30706
> Auton/SSE 0 0 0 0
> IP Process 1318821 193519884 175635 19038357
> Cache misses 0
> Fast 1530740 784982950 2547933 2054973479


As you can see here, 1318821 packets were process switched (handled by
IP Process) and 1530740 packets were process switched - of course, input
packets. So that is the reason I concluded that half of the packets
(input) were process switched (roughly). Output packets have much better
ratio (fast/process) here.

--
-Ivan.

*** Use Rot13 to see my eMail address ***
 
Reply With Quote
 
AnyBody43
Guest
Posts: n/a
 
      09-13-2004
(E-Mail Removed) (David) wrote in message news:<(E-Mail Removed). com>...
> Ivan Ostres <(E-Mail Removed)> wrote in message news:<(E-Mail Removed) et>...
> > In article <Emn0d.23$(E-Mail Removed)>, (E-Mail Removed)
> > says...
> > > Here are the interface switching stats.
> > >
> > > -Brian
> > >
> > > FastEthernet0/0
> > > IP Process 1318821 193519884 175635 19038357
> > > Cache misses 0
> > > Fast 1530740 784982950 2547933 2054973479
> > > Serial0/0

>
> > > IP Process 102607 9052386 671590 89484207
> > > Cache misses 0
> > > Fast 370718 289312539 219171 51002626

> >
> > As you can see here, half of your packets on FE0/0 are process switched
> > and one third of packets on S0/0 are process switched too. To conclude
> > reasons why is this going on you should post your config.

>
> How do you see that on FE0/0 half of packets are process switched


1318821 / (1318821 + 1530740) = (a half) plenty near enough for me.
 
Reply With Quote
 
micke
Guest
Posts: n/a
 
      09-13-2004
hmm just a ide something in the back of my head says a lot of
fragmented ip packages and policy routed wont work with cef.

/micke


Ivan Ostres <(E-Mail Removed)> wrote in message news:<(E-Mail Removed) et>...
> In article <(E-Mail Removed) >,
> (E-Mail Removed) says...
> > How do you see that on FE0/0 half of packets are process switched
> >

>
> This is the original output from 'show interfaces switching'. I've cut
> it a little bit.
>
>
> > FastEthernet0/0
> > Throttle count 0
> > Drops RP 148843 SP 0
> > SPD Flushes Fast 0 SSE 0
> > SPD Aggress Fast 0
> > SPD Priority Inputs 18 Drops 0
> >
> > Protocol Path Pkts In Chars In Pkts Out Chars Out
> >
> > Other Process 7336 790577 1907 114420
> > Cache misses 0
> > Fast 0 0 96 30706
> > Auton/SSE 0 0 0 0
> > IP Process 1318821 193519884 175635 19038357
> > Cache misses 0
> > Fast 1530740 784982950 2547933 2054973479

>
> As you can see here, 1318821 packets were process switched (handled by
> IP Process) and 1530740 packets were process switched - of course, input
> packets. So that is the reason I concluded that half of the packets
> (input) were process switched (roughly). Output packets have much better
> ratio (fast/process) here.

 
Reply With Quote
 
Hansang Bae
Guest
Posts: n/a
 
      09-14-2004
In article <(E-Mail Removed) >,
(E-Mail Removed) says...
> How do you see that on FE0/0 half of packets are process switched


Other's answered you question. Post the output of "sho ip traffic"

Also, you should post you configs (w/o the passwords)


--

hsb

"Somehow I imagined this experience would be more rewarding" Calvin
*************** USE ROT13 TO SEE MY EMAIL ADDRESS ****************
************************************************** ******************
Due to the volume of email that I receive, I may not not be able to
reply to emails sent to my account. Please post a followup instead.
************************************************** ******************
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
High CPU utilization on C3700 ??? Jimmy Cisco 3 10-20-2005 04:59 AM
High CPU utilization on Cisco 1760 Bancal Cisco 3 09-19-2005 06:03 PM
2650 IP Input High CPU Utilization Problem Brian R. Jack Cisco 1 09-15-2004 09:41 AM
high CPU utilization Miguel Moreno Cisco 4 01-30-2004 12:39 AM
2600XM, Frame Relay, and High CPU Utilization Donald Zelenak Jr. Cisco 5 12-06-2003 01:18 AM



Advertisments