CPU utilization on the router.

Discussion in 'Cisco' started by AM, Oct 14, 2005.

  1. AM

    AM Guest

    Trying to understand which process overload the CPU I noticed even if the total is close to 90% the sum of the all
    process doesn't reach that values. Is there some utilization hidden?

    Alex.
    AM, Oct 14, 2005
    #1
    1. Advertising

  2. AM

    Guest

    YES.

    The first line of sh proc cpu shows xy/za.

    This represents

    xy total cpu.

    za Interrupt level CPU.

    The latter is NOT accounted for in the rest of the displayed details
    since
    I guess the machine is too busy forwarding packets to do accounting.
    This is the reason you bought it.

    If this does not explain your observations please come back for more:)
    , Oct 14, 2005
    #2
    1. Advertising

  3. AM

    Merv Guest

    Capture the output of

    term len 0
    sh proc cpu

    Import into excel, sort by the field runtime(ms)whi will show which
    processes are using the largest amount of CPU
    Merv, Oct 15, 2005
    #3
  4. AM

    Igor Mamuzic Guest

    Like the others said you have a problem with interrupt cpu utilization.
    Check if you have CEF enabled on this router... If you do then there is a
    lot of "CEF unsupported" packets, so they have to be fast switched using CPU
    interrupts.
    This from my experience could be caused by high number of NAT translations
    if this router is performing NAT...

    CEF must be enabled on ingress interface(s)...

    Let me know what you find out or/and if you need IOS command support...

    B.R.
    Igor





    "AM" <> wrote in message
    news:wyV3f.8677$...
    > Trying to understand which process overload the CPU I noticed even if the
    > total is close to 90% the sum of the all process doesn't reach that
    > values. Is there some utilization hidden?
    >
    > Alex.
    Igor Mamuzic, Oct 15, 2005
    #4
  5. AM

    Merv Guest

    Post the output of

    sh version

    sh int stats

    sh ip cef

    and post your config
    Merv, Oct 15, 2005
    #5
  6. AM

    AM Guest

    Merv wrote:
    > Post the output of
    >
    > sh version
    >
    > sh int stats
    >
    > sh ip cef
    >
    > and post your config
    >


    Many thanks....

    ------------------------------------------------------------------------------------------------------------------------

    Cisco Internetwork Operating System Software
    IOS (tm) 3600 Software (C3620-IK9O3S7-M), Version 12.3(13a), RELEASE SOFTWARE (fc2)
    [CUT]
    ROM: System Bootstrap, Version 11.1(7)AX [kuong (7)AX], EARLY DEPLOYMENT RELEASE SOFTWARE (fc2)
    ROM: 3600 Software (C3620-IK9O3S7-M), Version 12.3(13a), RELEASE SOFTWARE (fc2)

    Borderline uptime is 4 days, 58 minutes
    System returned to ROM by reload
    System image file is "flash:c3620-ik9o3s7-mz.123-13a.bin"

    [CUT]

    cisco 3620 (R4700) processor (revision 0x81) with 61440K/4096K bytes of memory.
    Processor board ID R5S03T1A
    R4700 CPU at 80MHz, Implementation 33, Rev 1.0
    Bridging software.
    X.25 software, Version 3.0.0.
    Basic Rate ISDN software, Version 1.1.
    2 Ethernet/IEEE 802.3 interface(s)
    1 Serial network interface(s)
    4 ISDN Basic Rate interface(s)
    DRAM configuration is 32 bits wide with parity disabled.
    29K bytes of non-volatile configuration memory.
    32768K bytes of processor board System flash (Read/Write)

    Configuration register is 0x2102

    ------------------------------------------------------------------------------------------------------------------------

    Ethernet0/0
    Switching path Pkts In Chars In Pkts Out Chars Out
    Processor 866391 61385831 832953 114705969
    Route cache 2408528 1045733747 1911060 397805101
    Total 3274919 1107119578 2744013 512511070
    Interface Serial0/0 is disabled

    Ethernet0/1
    Switching path Pkts In Chars In Pkts Out Chars Out
    Processor 342236 80784405 73984 8393092
    Route cache 1301711 340221922 1799144 982834951
    Total 1643947 421006327 1873128 991228043

    -------------------------------------------------------------------------------------------------------------------------

    Prefix Next Hop Interface
    0.0.0.0/0 xxxxxxxxxxxxx Ethernet0/0
    0.0.0.0/32 receive
    10.174.231.0/24 192.168.38.137 Ethernet0/1
    192.168.38.136/29 attached Ethernet0/1
    192.168.38.136/32 receive
    192.168.38.137/32 192.168.38.137 Ethernet0/1
    192.168.38.142/32 receive
    192.168.38.143/32 receive
    192.168.38.192/28 192.168.38.137 Ethernet0/1
    Eth0/0 LAN---/24 attached Ethernet0/0
    Eth0/0 LAN---/32 receive
    1st ISP's GW /32 1st ISP's GW Ethernet0/0
    2nd ISP's GW /32 2nd ISP's GW Ethernet0/0
    Eth0 public IP/32 receive
    Eth0/0 bcast/32 receive
    224.0.0.0/4 drop
    224.0.0.0/24 receive
    255.255.255.255/32 receive

    -------------------------------------------------------------------------------------------------------------------------

    !
    version 12.3
    service timestamps debug datetime msec
    service timestamps log datetime msec
    service password-encryption
    !
    hostname Borderline
    !
    boot-start-marker
    boot-end-marker
    !
    no logging on
    enable secret 5 ffffffffffffffffffffffffff
    !
    no aaa new-model
    ip subnet-zero
    !
    !
    ip cef
    no ip domain lookup
    ip domain name fffffffff
    ip name-server xxxxxxxxxxxxxxx
    ip name-server xxxxxxxxxxxxxxx
    !
    ip audit po max-events 100
    !
    !
    username foofoo1 password 7 fffffffffffffffff
    username foofoo2 password 7 ggggggggggggggggggggg
    !
    !
    !
    !
    !
    interface Ethernet0/0
    ip address xxxxxxxxxxxxxx 255.255.255.0
    ip nat outside
    full-duplex
    !
    interface Serial0/0
    no ip address
    shutdown
    !
    interface Ethernet0/1
    ip address 192.168.38.142 255.255.255.248
    ip nat inside
    full-duplex
    !
    interface BRI1/0
    no ip address
    shutdown
    !
    interface BRI1/1
    no ip address
    shutdown
    !
    interface BRI1/2
    no ip address
    shutdown
    !
    interface BRI1/3
    no ip address
    shutdown
    !
    ip nat inside source list 112 interface Ethernet0/0 overload
    ip nat inside source static tcp 192.168.38.137 443 interface Ethernet0/0 443
    ip nat inside source static 192.168.38.193 xxxxxxxxxxxxxxx
    ip nat inside source static 192.168.38.137 xxxxxxxxxxxxxxx
    no ip http server
    no ip http secure-server
    ip classless
    ip route 0.0.0.0 0.0.0.0 xxxxxxxxxxxxxxxxx
    ip route 10.174.231.0 255.255.255.0 192.168.38.137
    ip route 192.168.38.192 255.255.255.240 192.168.38.137
    !
    !
    !
    ip access-list extended vty-access
    permit tcp 10.174.231.0 0.0.0.255 any eq 22
    permit tcp 10.174.231.0 0.0.0.255 any eq telnet
    permit tcp xxxxxxxxxxxxxx 0.0.1.255 any eq 22
    permit tcp xxxxxxxxxxxxxx 0.0.0.31 any eq 22
    access-list 1 permit 192.168.79.0 0.0.0.255
    access-list 1 deny 10.0.0.0 0.255.255.255
    access-list 1 deny 172.0.0.0 0.31.255.255
    access-list 1 deny 192.168.0.0 0.0.255.255
    access-list 1 permit xxxxxxxxxxxx 0.0.0.31
    access-list 1 permit xxxxxxxxxxxx 0.0.1.255
    access-list 1 deny any
    access-list 100 permit udp host 192.168.38.137 eq isakmp host xxxxxxxxxxxx eq isakmp
    access-list 100 permit udp host 192.168.38.137 eq non500-isakmp host xxxxxxxxxxxx eq non500-isakmp
    access-list 100 permit esp host 192.168.38.137 host xxxxxxxxxxxx
    access-list 100 deny ip any any
    access-list 111 permit udp host 192.168.38.137 eq isakmp host xxxxxxxxxxxx eq isakmp
    access-list 111 permit udp host 192.168.38.137 eq non500-isakmp host xxxxxxxxxxxx eq non500-isakmp
    access-list 111 permit esp host 192.168.38.137 host xxxxxxxxxxxx
    access-list 112 permit ip 10.174.231.0 0.0.0.255 any
    access-list 112 permit ip 192.168.38.136 0.0.0.7 any
    access-list 112 permit ip 192.168.38.192 0.0.0.15 any
    no cdp run
    !
    route-map NAT-VPN permit 10
    match ip address 111
    match interface Ethernet0/0
    !
    !
    line con 0
    line aux 0
    line vty 0 4
    access-class vty-access in
    login local
    !
    !
    end
    AM, Oct 17, 2005
    #6
  7. AM

    Merv Guest

    1. CEF is enabled - this is a goood thing.

    2. The percentage of packet processed switched does not seem to be a
    problem


    What is the packet per second rate when CPU is at 90%.
    Sugges that you configure load-interval 30 on each Ethernet interface
    and then clear counters.

    Post:
    show process cpu
    show interface
    when router CPU utilization is high
    Merv, Oct 17, 2005
    #7
  8. AM

    Merv Guest

    Merv, Oct 17, 2005
    #8
  9. AM

    AM Guest

    AM, Oct 17, 2005
    #9
  10. AM

    Merv Guest

    Post

    show process cpu

    show interface

    when you get a chance
    Merv, Oct 17, 2005
    #10
  11. AM

    Guest

    If most of the CPU is interrupt level (fast/CEF/whatever but NOT
    process)
    switching then the router is doing what it should and there
    may not be much you can do.

    You have indicated that the CPU is not accounted for in the
    sh proc cpu list which suggests that you are indeed fast switching.
    This is confirmed by the stats that you have posted.

    I suspec that NAT is fast switched but it will still be a lot of
    extra work.

    Normal fast switching.
    Look up cache
    Get new MAC header from cache
    Decrement TTL
    Fix up IP checksum IIRC?

    Oh my head hurts. Basically if nearly all of your CPU
    is Interrupt level

    [e.g.
    CPU utilization for five seconds: 95%/92%; one minute: 90%; five
    minutes: 83%
    95% of which 92% is Interrupt level]

    then the box is full and you have to take some load off, or suffer, or
    get a bigger one.

    End of story. Period.

    Removing say NAT seems like it would help a lot but I have
    no direct experience.

    Post "sh proc cpu" please to confirm.

    You may of course be passing unwanted traffic e.g. some
    kind of DoS attack but that is another story altogether.

    Good luck.
    , Oct 17, 2005
    #11
  12. AM

    Igor Mamuzic Guest

    I had experience with such issues and there was always about NAT:) NAT is
    fast switching operation (using cpu interrupts), but I think latest IOS-es
    are using CEF for NAT since you have 'CEF translated packets" counter in the
    'show ip nat stat' command output.

    I had about 200 hosts connected to the Internet trough my 2811 box and after
    I blocked p2p clients CPU interrupt usage has been fallen on 40% (since I
    have another about 250 hosts connected to the Internet, but they are not
    NATed by this box). p2p clients are causing a lot of NAT translations by
    single host since they are connected with multiple peers - thus there are
    multiple NAT translations generated by a single host.

    Post 'show ip nat stat' output...

    B.R.
    Igor


    <> wrote in message
    news:...
    > If most of the CPU is interrupt level (fast/CEF/whatever but NOT
    > process)
    > switching then the router is doing what it should and there
    > may not be much you can do.
    >
    > You have indicated that the CPU is not accounted for in the
    > sh proc cpu list which suggests that you are indeed fast switching.
    > This is confirmed by the stats that you have posted.
    >
    > I suspec that NAT is fast switched but it will still be a lot of
    > extra work.
    >
    > Normal fast switching.
    > Look up cache
    > Get new MAC header from cache
    > Decrement TTL
    > Fix up IP checksum IIRC?
    >
    > Oh my head hurts. Basically if nearly all of your CPU
    > is Interrupt level
    >
    > [e.g.
    > CPU utilization for five seconds: 95%/92%; one minute: 90%; five
    > minutes: 83%
    > 95% of which 92% is Interrupt level]
    >
    > then the box is full and you have to take some load off, or suffer, or
    > get a bigger one.
    >
    > End of story. Period.
    >
    > Removing say NAT seems like it would help a lot but I have
    > no direct experience.
    >
    > Post "sh proc cpu" please to confirm.
    >
    > You may of course be passing unwanted traffic e.g. some
    > kind of DoS attack but that is another story altogether.
    >
    > Good luck.
    >
    Igor Mamuzic, Oct 19, 2005
    #12
  13. AM

    AM Guest

    Igor Mamuzic wrote:

    > I had experience with such issues and there was always about NAT:) NAT is
    > fast switching operation (using cpu interrupts), but I think latest IOS-es
    > are using CEF for NAT since you have 'CEF translated packets" counter in the
    > 'show ip nat stat' command output.
    >
    > I had about 200 hosts connected to the Internet trough my 2811 box and after
    > I blocked p2p clients CPU interrupt usage has been fallen on 40% (since I
    > have another about 250 hosts connected to the Internet, but they are not
    > NATed by this box). p2p clients are causing a lot of NAT translations by
    > single host since they are connected with multiple peers - thus there are
    > multiple NAT translations generated by a single host.
    >
    > Post 'show ip nat stat' output...
    >
    > B.R.
    > Igor


    Thanks Igor,

    I thought about p2p and I did "show ip nat tra" and "show ip nat sta" but p2p is not my case as I had only static
    translations shown by the output.

    I don't know what to think...

    Alex.
    AM, Oct 20, 2005
    #13
  14. AM

    Guest

    > I had about 200 hosts connected to the Internet trough my 2811 box and after
    > I blocked p2p clients CPU interrupt usage has been fallen on 40% (since I
    > have another about 250 hosts connected to the Internet, but they are not
    > NATed by this box). p2p clients are causing a lot of NAT translations by
    > single host since they are connected with multiple peers - thus there are
    > multiple NAT translations generated by a single host.


    I would be very surprised if the number of NAT
    entries affected the CPU significantly. It will use a bit
    of memory however Cisco do no memory management so
    extra memory use cannot affect the CPU other than
    the initial allocation process and a possible subsequent free.

    Algorithms exist that can do such table lookups very efficiently
    and I am sure that cisco have heard of them.

    Internal hosts doing port scans of course could generate a lot
    of NAT table activity.

    NATted traffic on the other hand will I am sure use quite a
    bit more CPU than non NATted traffic, CEF or no CEF
    however this will I believe be independent of the
    number of NAT table entries.

    One other thing that can affect the CPU of a smallish
    router very badly are broadcasts. I have in the past put

    acess-list 100 deny ip any 255.255.255.255 255.255.255.255
    acess-list 100 deny ip any local-ip-net local-ip-net-wildcard

    on the interface of a small router.

    This will of course break some things, e.g dhcp forwarding,
    RIP, but basic IP routing is OK. ARP is OK too since it
    does not use the IP protocol (0800?) but uses (0806?) instead.

    Many arp requests?

    These DoS like things will though show up in the sh proc cpu
    as non-interrupt level tasks.
    , Oct 20, 2005
    #14
  15. AM

    Igor Mamuzic Guest

    Anybody,

    I just know from my past experience that when I eliminated p2p that where
    NATed I significantly reduced my CPU interrupt load. Also, Cisco says in
    it's high cpu load troubleshooting guides that NAT is interrupt process that
    uses cache switching instead of CEF (I'm not sure that this considers latest
    IOS versions), so of course NAT is mostly memory issue, but could bring to
    the high cpu load due to interrupts...

    B.R.
    Igor






    <> wrote in message
    news:...
    >> I had about 200 hosts connected to the Internet trough my 2811 box and
    >> after
    >> I blocked p2p clients CPU interrupt usage has been fallen on 40% (since I
    >> have another about 250 hosts connected to the Internet, but they are not
    >> NATed by this box). p2p clients are causing a lot of NAT translations by
    >> single host since they are connected with multiple peers - thus there are
    >> multiple NAT translations generated by a single host.

    >
    > I would be very surprised if the number of NAT
    > entries affected the CPU significantly. It will use a bit
    > of memory however Cisco do no memory management so
    > extra memory use cannot affect the CPU other than
    > the initial allocation process and a possible subsequent free.
    >
    > Algorithms exist that can do such table lookups very efficiently
    > and I am sure that cisco have heard of them.
    >
    > Internal hosts doing port scans of course could generate a lot
    > of NAT table activity.
    >
    > NATted traffic on the other hand will I am sure use quite a
    > bit more CPU than non NATted traffic, CEF or no CEF
    > however this will I believe be independent of the
    > number of NAT table entries.
    >
    > One other thing that can affect the CPU of a smallish
    > router very badly are broadcasts. I have in the past put
    >
    > acess-list 100 deny ip any 255.255.255.255 255.255.255.255
    > acess-list 100 deny ip any local-ip-net local-ip-net-wildcard
    >
    > on the interface of a small router.
    >
    > This will of course break some things, e.g dhcp forwarding,
    > RIP, but basic IP routing is OK. ARP is OK too since it
    > does not use the IP protocol (0800?) but uses (0806?) instead.
    >
    > Many arp requests?
    >
    > These DoS like things will though show up in the sh proc cpu
    > as non-interrupt level tasks.
    >
    Igor Mamuzic, Oct 20, 2005
    #15
  16. AM

    AM Guest

    AM wrote:
    > Trying to understand which process overload the CPU I noticed even if
    > the total is close to 90% the sum of the all process doesn't reach that
    > values. Is there some utilization hidden?
    >
    > Alex.


    I'm waiting for a Contract token and then ask Cisco expert.
    I will inform you about the solution.

    Thanks to all, Alex
    AM, Oct 24, 2005
    #16
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Donald Zelenak Jr.

    2600XM, Frame Relay, and High CPU Utilization

    Donald Zelenak Jr., Dec 4, 2003, in forum: Cisco
    Replies:
    5
    Views:
    1,413
    Donald Zelenak Jr.
    Dec 6, 2003
  2. Miguel Moreno

    high CPU utilization

    Miguel Moreno, Jan 28, 2004, in forum: Cisco
    Replies:
    4
    Views:
    1,313
    Miguel Moreno
    Jan 30, 2004
  3. Brian R. Jack
    Replies:
    1
    Views:
    3,218
    Øystein Berg
    Sep 15, 2004
  4. Brian R. Jack
    Replies:
    8
    Views:
    1,661
    Hansang Bae
    Sep 14, 2004
  5. Izack
    Replies:
    21
    Views:
    1,787
    Izack
    Jan 6, 2005
Loading...

Share This Page