2650 IP Input High CPU Utilization Problem

Discussion in 'Cisco' started by Brian R. Jack, Sep 10, 2004.

  1. I have a 2650 deployed as an edge device in a distributed internet
    provisioning environment. The router has 2 parallel per-packet load-shared
    T1 connections with internal DSU WICs and only the single fast ethernet
    interface for the LAN side that is being trunked to a Cat 4006. The config
    has been in place for over 2 years without a hitch until a recent ISP change
    that has created a huge problem. Basically the IP Input process
    incrementally consumes more and more proc time over about 2 hours to the
    point where I have to reload the router in order for traffic to pass
    through.

    Specifics:

    IOS 12.1(5)T10
    64MB RAM

    25 fast ethernet subinterfaces mapped to VLANs hosted on a Cat4006, all subs
    have private IP subnets assigned to them NAT is performed by the router.
    There are over 25 static mappings plus a dynamic pool consisting of 7 IP
    addresses

    2 PPP serial connections to a carrier provided T1 channel bank. T1's are
    running clean and clear, no service-module slips or framing errors noted.
    Serial connections are also clear with no CRC, no i/o drops, no errors etc.
    Load is not out of the ordinary for either connection (both run at about
    15 - 40% tx/rx load).

    CEF and fast switching is enabled on all interfaces
    Netflow accounting is enabled
    CEF per-packet load-sharing is enabled on serial interfaces

    I have worked through all of the cisco docs regarding high cpu utilization
    and found nothing out of the ordinary. Initially I concerned that this
    might be an issue caused by some NAT changes that had to be put in effect
    after the ISP change but my traffic levels just don't substantiate that.

    At this point I am looking for any ideas that anyone else might have
    regarding this issue.

    Thanks,
    Brian R. Jack
     
    Brian R. Jack, Sep 10, 2004
    #1
    1. Advertising

  2. Hello, Brian!
    You wrote on Fri, 10 Sep 2004 10:48:11 -0700:

    BRJ> I have a 2650 deployed as an edge device in a distributed
    BRJ> internet provisioning environment. The router has 2 parallel
    BRJ> per-packet load-shared T1 connections with internal DSU WICs and
    BRJ> only the single fast ethernet interface for the LAN side that is
    BRJ> being trunked to a Cat 4006. The config has been in place for
    BRJ> over 2 years without a hitch until a recent ISP change that has
    BRJ> created a huge problem.

    BRJ> Basically the IP Input process incrementally consumes more and more proc
    BRJ> time over about 2 hours to the point where I have to reload the router in
    BRJ> order for traffic to pass through.

    BRJ> 25 fast ethernet subinterfaces mapped to VLANs hosted on a
    BRJ> Cat4006, all subs have private IP subnets assigned to them NAT
    BRJ> is performed by the router.

    BRJ> CEF and fast switching is enabled on all interfaces

    Well, IP Input process loading CPU and CEF doesn't fit together. Can you
    provide output from show interface switching? My guess is that there is a lot of
    traffic between VLAN's and CEF on same interface is not enabled or not working
    properly.

    With best regards,
    Andrey.
     
    Andrey Tarasov, Sep 10, 2004
    #2
    1. Advertising

  3. Here are the interface switching stats.

    -Brian

    FastEthernet0/0
    Throttle count 0
    Drops RP 148843 SP 0
    SPD Flushes Fast 0 SSE 0
    SPD Aggress Fast 0
    SPD Priority Inputs 18 Drops 0

    Protocol Path Pkts In Chars In Pkts Out Chars Out
    Other Process 7336 790577 1907 114420
    Cache misses 0
    Fast 0 0 96 30706
    Auton/SSE 0 0 0 0
    IP Process 1318821 193519884 175635 19038357
    Cache misses 0
    Fast 1530740 784982950 2547933 2054973479
    Auton/SSE 0 0 0 0
    ARP Process 7354 441676 14793 946752
    Cache misses 0
    Fast 0 0 0 0
    Auton/SSE 0 0 0 0
    Serial0/0
    Throttle count 0
    Drops RP 74 SP 0
    SPD Flushes Fast 0 SSE 0
    SPD Aggress Fast 0
    SPD Priority Inputs 0 Drops 0

    Protocol Path Pkts In Chars In Pkts Out Chars Out
    Other Process 2 28 179 2862
    Cache misses 0
    Fast 0 0 538 106456
    Auton/SSE 0 0 0 0
    IP Process 102607 9052386 671590 89484207
    Cache misses 0
    Fast 370718 289312539 219171 51002626
    Auton/SSE 0 0 0 0
    Serial0/1
    Throttle count 0
    Drops RP 34 SP 0
    SPD Flushes Fast 0 SSE 0
    SPD Aggress Fast 0
    SPD Priority Inputs 0 Drops 0

    Protocol Path Pkts In Chars In Pkts Out Chars Out
    Other Process 2 28 179 2862
    Cache misses 0
    Fast 0 0 532 99505
    Auton/SSE 0 0 0 0
    IP Process 164586 14412378 672105 89645741
    Cache misses 0
    Fast 1089543 1061566468 219969 51049450
    Auton/SSE 0 0 0 0


    "Andrey Tarasov" <> wrote in message
    news:chsrp4$80u$...
    > Hello, Brian!
    > You wrote on Fri, 10 Sep 2004 10:48:11 -0700:
    >
    > BRJ> I have a 2650 deployed as an edge device in a distributed
    > BRJ> internet provisioning environment. The router has 2 parallel
    > BRJ> per-packet load-shared T1 connections with internal DSU WICs and
    > BRJ> only the single fast ethernet interface for the LAN side that is
    > BRJ> being trunked to a Cat 4006. The config has been in place for
    > BRJ> over 2 years without a hitch until a recent ISP change that has
    > BRJ> created a huge problem.
    >
    > BRJ> Basically the IP Input process incrementally consumes more and more
    > proc
    > BRJ> time over about 2 hours to the point where I have to reload the
    > router in
    > BRJ> order for traffic to pass through.
    >
    > BRJ> 25 fast ethernet subinterfaces mapped to VLANs hosted on a
    > BRJ> Cat4006, all subs have private IP subnets assigned to them NAT
    > BRJ> is performed by the router.
    >
    > BRJ> CEF and fast switching is enabled on all interfaces
    >
    > Well, IP Input process loading CPU and CEF doesn't fit together. Can you
    > provide output from show interface switching? My guess is that there is a
    > lot of
    > traffic between VLAN's and CEF on same interface is not enabled or not
    > working
    > properly.
    >
    > With best regards,
    > Andrey.
    >
     
    Brian R. Jack, Sep 10, 2004
    #3
  4. Brian R. Jack

    Ivan Ostres Guest

    In article <Emn0d.23$>,
    says...
    > Here are the interface switching stats.
    >
    > -Brian
    >
    > FastEthernet0/0
    > IP Process 1318821 193519884 175635 19038357
    > Cache misses 0
    > Fast 1530740 784982950 2547933 2054973479
    > Serial0/0


    > IP Process 102607 9052386 671590 89484207
    > Cache misses 0
    > Fast 370718 289312539 219171 51002626


    As you can see here, half of your packets on FE0/0 are process switched
    and one third of packets on S0/0 are process switched too. To conclude
    reasons why is this going on you should post your config.


    --
    -Ivan.

    *** Use Rot13 to see my eMail address ***
     
    Ivan Ostres, Sep 11, 2004
    #4
  5. Brian R. Jack

    David Guest

    Ivan Ostres <> wrote in message news:<>...
    > In article <Emn0d.23$>,
    > says...
    > > Here are the interface switching stats.
    > >
    > > -Brian
    > >
    > > FastEthernet0/0
    > > IP Process 1318821 193519884 175635 19038357
    > > Cache misses 0
    > > Fast 1530740 784982950 2547933 2054973479
    > > Serial0/0

    >
    > > IP Process 102607 9052386 671590 89484207
    > > Cache misses 0
    > > Fast 370718 289312539 219171 51002626

    >
    > As you can see here, half of your packets on FE0/0 are process switched
    > and one third of packets on S0/0 are process switched too. To conclude
    > reasons why is this going on you should post your config.


    How do you see that on FE0/0 half of packets are process switched
     
    David, Sep 13, 2004
    #5
  6. Brian R. Jack

    Ivan Ostres Guest

    In article <>,
    says...
    > How do you see that on FE0/0 half of packets are process switched
    >


    This is the original output from 'show interfaces switching'. I've cut
    it a little bit.


    > FastEthernet0/0
    > Throttle count 0
    > Drops RP 148843 SP 0
    > SPD Flushes Fast 0 SSE 0
    > SPD Aggress Fast 0
    > SPD Priority Inputs 18 Drops 0
    >
    > Protocol Path Pkts In Chars In Pkts Out Chars Out
    >
    > Other Process 7336 790577 1907 114420
    > Cache misses 0
    > Fast 0 0 96 30706
    > Auton/SSE 0 0 0 0
    > IP Process 1318821 193519884 175635 19038357
    > Cache misses 0
    > Fast 1530740 784982950 2547933 2054973479


    As you can see here, 1318821 packets were process switched (handled by
    IP Process) and 1530740 packets were process switched - of course, input
    packets. So that is the reason I concluded that half of the packets
    (input) were process switched (roughly). Output packets have much better
    ratio (fast/process) here.

    --
    -Ivan.

    *** Use Rot13 to see my eMail address ***
     
    Ivan Ostres, Sep 13, 2004
    #6
  7. Brian R. Jack

    AnyBody43 Guest

    (David) wrote in message news:<>...
    > Ivan Ostres <> wrote in message news:<>...
    > > In article <Emn0d.23$>,
    > > says...
    > > > Here are the interface switching stats.
    > > >
    > > > -Brian
    > > >
    > > > FastEthernet0/0
    > > > IP Process 1318821 193519884 175635 19038357
    > > > Cache misses 0
    > > > Fast 1530740 784982950 2547933 2054973479
    > > > Serial0/0

    >
    > > > IP Process 102607 9052386 671590 89484207
    > > > Cache misses 0
    > > > Fast 370718 289312539 219171 51002626

    > >
    > > As you can see here, half of your packets on FE0/0 are process switched
    > > and one third of packets on S0/0 are process switched too. To conclude
    > > reasons why is this going on you should post your config.

    >
    > How do you see that on FE0/0 half of packets are process switched


    1318821 / (1318821 + 1530740) = (a half) plenty near enough for me.
     
    AnyBody43, Sep 13, 2004
    #7
  8. Brian R. Jack

    micke Guest

    hmm just a ide something in the back of my head says a lot of
    fragmented ip packages and policy routed wont work with cef.

    /micke


    Ivan Ostres <> wrote in message news:<>...
    > In article <>,
    > says...
    > > How do you see that on FE0/0 half of packets are process switched
    > >

    >
    > This is the original output from 'show interfaces switching'. I've cut
    > it a little bit.
    >
    >
    > > FastEthernet0/0
    > > Throttle count 0
    > > Drops RP 148843 SP 0
    > > SPD Flushes Fast 0 SSE 0
    > > SPD Aggress Fast 0
    > > SPD Priority Inputs 18 Drops 0
    > >
    > > Protocol Path Pkts In Chars In Pkts Out Chars Out
    > >
    > > Other Process 7336 790577 1907 114420
    > > Cache misses 0
    > > Fast 0 0 96 30706
    > > Auton/SSE 0 0 0 0
    > > IP Process 1318821 193519884 175635 19038357
    > > Cache misses 0
    > > Fast 1530740 784982950 2547933 2054973479

    >
    > As you can see here, 1318821 packets were process switched (handled by
    > IP Process) and 1530740 packets were process switched - of course, input
    > packets. So that is the reason I concluded that half of the packets
    > (input) were process switched (roughly). Output packets have much better
    > ratio (fast/process) here.
     
    micke, Sep 13, 2004
    #8
  9. Brian R. Jack

    Hansang Bae Guest

    In article <>,
    says...
    > How do you see that on FE0/0 half of packets are process switched


    Other's answered you question. Post the output of "sho ip traffic"

    Also, you should post you configs (w/o the passwords)


    --

    hsb

    "Somehow I imagined this experience would be more rewarding" Calvin
    *************** USE ROT13 TO SEE MY EMAIL ADDRESS ****************
    ********************************************************************
    Due to the volume of email that I receive, I may not not be able to
    reply to emails sent to my account. Please post a followup instead.
    ********************************************************************
     
    Hansang Bae, Sep 14, 2004
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Donald Zelenak Jr.

    2600XM, Frame Relay, and High CPU Utilization

    Donald Zelenak Jr., Dec 4, 2003, in forum: Cisco
    Replies:
    5
    Views:
    1,451
    Donald Zelenak Jr.
    Dec 6, 2003
  2. Miguel Moreno

    high CPU utilization

    Miguel Moreno, Jan 28, 2004, in forum: Cisco
    Replies:
    4
    Views:
    1,395
    Miguel Moreno
    Jan 30, 2004
  3. Brian R. Jack
    Replies:
    1
    Views:
    3,276
    Øystein Berg
    Sep 15, 2004
  4. Bancal
    Replies:
    3
    Views:
    2,899
  5. Jimmy
    Replies:
    3
    Views:
    1,141
    w1llr0ut34f00d
    Oct 20, 2005
Loading...

Share This Page