High CPU load on Cisco 2600

Discussion in 'Cisco' started by Syn, Feb 18, 2005.

  1. Syn

    Syn Guest

    Hello,

    We are having some very very slow response time from our Cisco 2600
    router starting since yesterday, as you can see here from show proc cpu:


    CPU utilization for five seconds: 100%/1%; one minute: 99%; five
    minutes: 80%
    29 2366684 441963 5354 97.20% 97.64% 77.52% 0 IP Input

    the IP Input process is using all CPU resources but we only have a
    2mbit/s line behind and a very small ACL. What could the problem come
    from and how can I find the source of this problem ?

    Thanks
    Regards
    Syn, Feb 18, 2005
    #1
    1. Advertising

  2. Syn

    Merv Guest

    Do you have CEF configured ?

    Post your configuration.

    Reladd the router to see if that clears the problem
    Merv, Feb 18, 2005
    #2
    1. Advertising

  3. Syn

    Syn Guest

    Merv wrote:
    > Do you have CEF configured ?
    >
    > Post your configuration.
    >
    > Reladd the router to see if that clears the problem
    >


    What is CEF ?

    I already reloaded the router, and in a few minutes it starts again.

    Here is the config:


    !RANCID-CONTENT-TYPE: cisco
    !
    !Chassis type: 2621 - a 2600 router
    !CPU: MPC860
    !
    !Memory: main 60416K/5120K
    !Memory: nvram 32K
    !
    !Power: Redundant Power System is not present.
    !
    !Image: Software: C2600-JK9S-M, 12.2(6c), RELEASE SOFTWARE (fc1)
    !Image: Compiled: Sat 02-Feb-02 01:09 by pwade
    !Image: flash:c2600-jk9s-mz.122-6c.bin
    !
    !ROM Bootstrap: Version 11.3(2)XA4, RELEASE SOFTWARE (fc1)
    !
    !
    !Flash: System flash directory:
    !Flash: File Length Name/status
    !Flash: 1 12421732 c2600-jk9s-mz.122-6c.bin
    !Flash: [12421796 bytes used, 4355420 available, 16777216 total]
    !Flash: 16384K bytes of processor board System flash (Read/Write)
    !
    !Flash: nvram: Directory of nvram:/
    !Flash: nvram: 20 -rw- 7511 <no date>
    startup-config
    !Flash: nvram: 21 ---- 24 <no date>
    private-config
    !Flash: nvram: 1 -rw- 0 <no date> ifIndex-table
    !Flash: nvram: 29688 bytes total (21077 bytes free)
    !
    !Interface: FastEthernet0/0, AMD Laguna
    !Interface: FastEthernet0/1, AMD Laguna
    !
    !Slot 0: type C2621 2FE Mainboard, 2 ports
    !
    !
    config-register 0x2102
    version 12.2
    service timestamps debug datetime
    service timestamps log datetime
    service password-encryption
    !
    hostname router
    !
    logging buffered 4096 debugging
    enable secret 5 *SECRET*
    !
    clock timezone CET 1
    clock summer-time CDT recurring
    ip subnet-zero
    !
    !
    no ip domain-lookup
    !
    ip ssh time-out 120
    ip ssh authentication-retries 3
    !
    crypto isakmp policy 100
    authentication pre-share
    !crypto isakmp key <removed> address <IP>
    !crypto isakmp key <removed> address <IP>
    !crypto isakmp key <removed> address <IP>
    !
    !
    crypto ipsec transform-set ts esp-des esp-sha-hmac
    !
    crypto map cmap 11 ipsec-isakmp
    set peer <IP>
    set transform-set ts
    set pfs group1
    match address crypto-muc-bsl
    crypto map cmap 21 ipsec-isakmp
    set peer <IP>
    set transform-set ts
    set pfs group1
    match address crypto-muc-sna
    crypto map cmap 31 ipsec-isakmp
    set peer <IP>
    set transform-set ts
    set pfs group1
    match address crypto-muc-lon
    crypto map cmap 41 ipsec-isakmp
    set peer <IP>
    set transform-set ts
    set pfs group1
    match address crypto-muc-sna2
    crypto map cmap 56 ipsec-isakmp
    set peer <IP>
    set transform-set ts
    set pfs group1
    match address crypto-muc-spa
    !
    call rsvp-sync
    !
    !
    !
    !
    !
    !
    !
    !
    interface Tunnel11
    bandwidth 2304
    ip address <IP> <IP>
    tunnel source FastEthernet0/0
    tunnel destination <IP>
    tunnel key <KEY>
    tunnel checksum
    crypto map cmap
    !
    interface Tunnel21
    bandwidth 2304
    ip address <IP> <IP>
    tunnel source FastEthernet0/0
    tunnel destination <IP>
    tunnel key <KEY>
    tunnel checksum
    crypto map cmap
    !
    interface Tunnel31
    bandwidth 2304
    ip address <IP> <IP>
    tunnel source FastEthernet0/0
    tunnel destination <IP>
    tunnel key <KEY>
    tunnel checksum
    crypto map cmap
    !
    interface Tunnel41
    bandwidth 2304
    ip address <IP> <IP>
    tunnel source FastEthernet0/0
    tunnel destination <IP>
    tunnel key <KEY>
    tunnel checksum
    crypto map cmap
    !
    interface Tunnel56
    bandwidth 2304
    ip address <IP> <IP>
    tunnel source FastEthernet0/0
    tunnel destination <IP>
    tunnel key 565656
    tunnel checksum
    crypto map cmap
    !
    interface FastEthernet0/0
    bandwidth 2304
    ip address <IP> <IP>
    ip access-group provider-in in
    ip nat outside
    duplex auto
    speed auto
    no cdp enable
    crypto map cmap
    !
    interface FastEthernet0/1
    ip address <IP> <IP>
    ip nat inside
    duplex auto
    speed auto
    no cdp enable
    !
    router eigrp 300
    network <IP>
    network <IP>
    no auto-summary
    eigrp log-neighbor-changes
    !
    ip nat inside source list nat interface FastEthernet0/0 over
    ip nat inside source static tcp <IP> 4302 <IP> 80 extendable
    ip nat inside source static tcp <IP> 5900 <IP> 5900 extendable
    ip nat inside source static tcp <IP> 5800 <IP> 5800 extendable
    ip nat inside source static tcp <IP> 22 <IP> 22 extendable
    ip nat inside source static tcp <IP> 4099 <IP> 4099 extendable
    ip nat inside source static tcp <IP> 4100 <IP> 4100 extendable
    ip nat inside source static tcp <IP> 4302 <IP> 4302 extendable
    ip nat inside source static tcp <IP> 4303 <IP> 4303 extendable
    ip nat inside source static tcp <IP> 4304 <IP> 4304 extendable
    ip nat inside source static tcp <IP> 4305 <IP> 4305 extendable
    ip classless
    ip route 0.0.0.0 0.0.0.0 <IP>
    ip route <IP> 255.255.255.0 Tunnel11
    ip route <IP> 255.255.255.0 Tunnel31
    ip route <IP> 255.255.255.0 Tunnel41
    ip route <IP> 255.255.255.0 Tunnel21
    ip route <IP> 255.255.255.252 Tunnel31
    ip route <IP> 255.255.255.252 Tunnel21
    ip http server
    ip pim bidir-enable
    !
    !
    ip access-list standard nat
    permit <IP> 0.0.0.255
    !
    ip access-list extended crypto-muc-bsl
    permit gre host <IP> host <IP>
    ip access-list extended crypto-muc-irv
    permit gre host <IP> host <IP>
    ip access-list extended crypto-muc-lon
    permit gre host <IP> host <IP>
    ip access-list extended crypto-muc-sna
    permit gre host <IP> host <IP>
    ip access-list extended crypto-muc-sna2
    permit gre host <IP> host <IP>
    ip access-list extended crypto-muc-spa
    permit gre host <IP> host <IP>
    ip access-list extended provider-in
    permit tcp any host <IP> gt 1023 established
    permit gre host <IP> host <IP>
    permit gre host <IP> host <IP>
    permit gre host <IP> host <IP>
    permit ahp host <IP> host <IP>
    permit esp host <IP> host <IP>
    permit udp host <IP> host <IP> eq isakmp
    permit ahp host <IP> host <IP>
    permit esp host <IP> host <IP>
    permit udp host <IP> host <IP> eq isakmp
    permit ahp host <IP> host <IP>
    permit esp host <IP> host <IP>
    permit udp host <IP> host <IP> eq isakmp
    permit udp any eq domain host <IP>
    permit tcp any eq domain host <IP>
    permit udp host <IP> eq ntp host <IP> eq ntp
    permit udp host <IP> eq ntp host <IP> eq ntp
    permit udp host <IP> eq ntp host <IP> eq ntp
    permit udp host <IP> eq ntp host <IP> eq ntp
    permit udp host <IP> eq ntp host <IP> eq ntp
    permit icmp any any echo
    permit icmp any any echo-reply
    permit icmp any host <IP> ttl-exceeded
    permit icmp any host <IP> port-unreachable
    permit icmp any host <IP> host-unreachable
    permit udp any host <IP> range 33434 33524
    permit tcp any host <IP> eq 5800
    permit tcp any host <IP> eq 5900
    permit tcp any host <IP> eq 4302
    permit tcp any host <IP> eq 4303
    permit tcp any host <IP> eq 4304
    permit tcp any host <IP> eq 4305
    permit tcp any host <IP> eq 4099
    permit tcp any host <IP> eq 4100
    permit tcp any host <IP> eq www
    permit tcp any host <IP> eq 22
    permit tcp host <IP> host <IP> eq telnet
    permit udp host <IP> eq isakmp host <IP> eq isakmp
    permit udp host <IP> eq 2746 host <IP> eq 2746
    deny ip 10.0.0.0 0.255.255.255 any log
    deny ip 172.16.0.0 0.15.255.255 any log
    deny ip 192.168.0.0 0.0.255.255 any log
    deny ip 224.0.0.0 31.255.255.255 any log
    deny ip any any log
    deny tcp any any log
    deny udp any any log
    deny icmp any any log
    deny gre any any log
    deny ahp any any log
    deny esp any any log
    no cdp run
    !
    snmp-server community public RO
    !
    dial-peer cor custom
    !
    !
    !
    !
    !
    line con 0
    ! password <removed>
    login
    line aux 0
    line vty 0 4
    ! password <removed>
    login
    !
    no scheduler allocate
    ntp server <IP>
    ntp server <IP>
    ntp server <IP>
    ntp server <IP>
    ntp server <IP>
    end
    Syn, Feb 18, 2005
    #3
  4. Syn

    Merv Guest


    > What is CEF ?


    CEF stands for Cisco Express Fprwarding and is the highest performance
    switching path that Cisco has.

    If your router supports it turn it on:

    conf t
    ip cef
    exit

    Post output of "sh int stat" after you enable CEF
    Merv, Feb 18, 2005
    #4
  5. Syn

    Merv Guest

    are the far end able to support AES encyption; if so I would uggest
    that you transition from DES to AES. once you sort out the CPU
    utilization issue

    also I would transition from group 1 to group 2
    Merv, Feb 18, 2005
    #5
  6. Syn

    Merv Guest

    post output of "sh ip traffic" for the 2600
    Merv, Feb 18, 2005
    #6
  7. Syn

    Ivan Ostreš Guest

    In article <4215aaaf$0$280$>,
    says...
    > Hello,
    >
    > We are having some very very slow response time from our Cisco 2600
    > router starting since yesterday, as you can see here from show proc cpu:
    >
    >
    > CPU utilization for five seconds: 100%/1%; one minute: 99%; five
    > minutes: 80%
    > 29 2366684 441963 5354 97.20% 97.64% 77.52% 0 IP Input
    >
    > the IP Input process is using all CPU resources but we only have a
    > 2mbit/s line behind and a very small ACL. What could the problem come
    > from and how can I find the source of this problem ?
    >
    > Thanks
    > Regards
    >


    Output from 'show interfaces switching' and 'show interfaces' commands
    would help a lot.

    --
    -Ivan.

    *** Use Rot13 to see my eMail address ***
    Ivan Ostreš, Feb 18, 2005
    #7
  8. Syn

    Iggy Guest

    Hello,

    I had the same problem on 2621XM router and discovered that it was about
    sasser worm outbreak... After implementing inbound ACL that blocks TCP/UDP
    135, 139, 443 on my LAN's interface CPU util. has been restored on baseline
    value... Implement this and check if there are any matches on those ACL
    entrys with show access-list [ACL number] IOS command...

    btw. which IOS version do you use? On 2811 ISR router with IOS 12.3(4)T with
    FW/IPS feature enabled I had a problem of watchdog timer expiration,
    excessive CPU load and unexpected router reloading....After upgrade to GD
    version (12.3(11)XL) it seems that problem (documented bug on Cisco.com) has
    gone...

    B.R.
    Igor



    "Syn" <> wrote in message
    news:4215aaaf$0$280$...
    > Hello,
    >
    > We are having some very very slow response time from our Cisco 2600 router
    > starting since yesterday, as you can see here from show proc cpu:
    >
    >
    > CPU utilization for five seconds: 100%/1%; one minute: 99%; five minutes:
    > 80%
    > 29 2366684 441963 5354 97.20% 97.64% 77.52% 0 IP Input
    >
    > the IP Input process is using all CPU resources but we only have a 2mbit/s
    > line behind and a very small ACL. What could the problem come from and how
    > can I find the source of this problem ?
    >
    > Thanks
    > Regards
    Iggy, Feb 18, 2005
    #8
  9. Syn

    Syn Guest

    Merv wrote:

    > CEF stands for Cisco Express Fprwarding and is the highest performance
    > switching path that Cisco has.


    > If your router supports it turn it on:
    >
    > conf t
    > ip cef
    > exit
    >
    > Post output of "sh int stat" after you enable CEF


    Hmm this feature sounds interesting, a shame I didn't know about it
    before ;)

    Here is the output of sh int stat after enabling:

    FastEthernet0/0
    Switching path Pkts In Chars In Pkts Out Chars Out
    Processor 205016 68224558 407967 74591204
    Route cache 251876 135694673 84211 11272781
    Total 456892 203919231 492178 85863985
    FastEthernet0/1
    Switching path Pkts In Chars In Pkts Out Chars Out
    Processor 241140 44859485 223526 77414664
    Route cache 84216 11273604 89690 98251225
    Total 325356 56133089 313216 175665889
    Tunnel11
    Switching path Pkts In Chars In Pkts Out Chars Out
    Processor 178744 76593262 174392 42792030
    Route cache 0 0 0 0
    Total 178744 76593262 174392 42792030
    Tunnel21
    Switching path Pkts In Chars In Pkts Out Chars Out
    Processor 0 0 17430 1614645
    Route cache 0 0 0 0
    Total 0 0 17430 1614645
    Tunnel31
    Switching path Pkts In Chars In Pkts Out Chars Out
    Processor 0 0 22487 1448679
    Route cache 0 0 0 0
    Total 0 0 22487 1448679
    Tunnel41
    Switching path Pkts In Chars In Pkts Out Chars Out
    Processor 0 0 16595 998022
    Route cache 0 0 0 0
    Total 0 0 16595 998022
    Tunnel56
    Switching path Pkts In Chars In Pkts Out Chars Out
    Processor 18101 1643966 17994 1644227
    Route cache 0 0 0 0
    Total 18101 1643966 17994 1644227
    Syn, Feb 18, 2005
    #9
  10. Syn

    Syn Guest

    Merv wrote:
    > post output of "sh ip traffic" for the 2600
    >

    Here it is:


    IP statistics:
    Rcvd: 1215686 total, 329763 local destination
    0 format errors, 0 checksum errors, 0 bad hop count
    0 unknown protocol, 1 not a gateway
    0 security failures, 0 bad options, 0 with options
    Opts: 0 end, 0 nop, 0 basic security, 0 loose source route
    0 timestamp, 0 extended security, 0 record route
    0 stream ID, 0 strict source route, 0 alert, 0 cipso, 0 ump
    0 other
    Frags: 34660 reassembled, 128 timeouts, 0 couldn't reassemble
    27466 fragmented, 777 couldn't fragment
    Bcast: 4116 received, 0 sent
    Mcast: 33445 received, 99578 sent
    Sent: 203385 generated, 615119 forwarded
    Drop: 24 encapsulation failed, 5 unresolved, 0 no adjacency
    0 no route, 0 unicast RPF, 0 forced drop

    ICMP statistics:
    Rcvd: 0 format errors, 0 checksum errors, 0 redirects, 3 unreachable
    89825 echo, 0 echo reply, 0 mask requests, 0 mask replies, 0 quench
    0 parameter, 0 timestamp, 0 info request, 0 other
    0 irdp solicitations, 0 irdp advertisements
    Sent: 0 redirects, 1365 unreachable, 0 echo, 89825 echo reply
    0 mask requests, 0 mask replies, 0 quench, 0 timestamp
    0 info reply, 111 time exceeded, 0 parameter problem
    0 irdp solicitations, 0 irdp advertisements

    UDP statistics:
    Rcvd: 5384 total, 0 checksum errors, 4543 no port
    Sent: 1039 total, 0 forwarded broadcasts

    TCP statistics:
    Rcvd: 3859 total, 0 checksum errors, 46 no port
    Sent: 11388 total

    Probe statistics:
    Rcvd: 0 address requests, 0 address replies
    0 proxy name requests, 0 where-is requests, 0 other
    Sent: 0 address requests, 0 address replies (0 proxy)
    0 proxy name replies, 0 where-is replies

    EGP statistics:
    Rcvd: 0 total, 0 format errors, 0 checksum errors, 0 no listener
    Sent: 0 total

    IGRP statistics:
    Rcvd: 0 total, 0 checksum errors
    Sent: 0 total

    OSPF statistics:
    Rcvd: 0 total, 0 checksum errors
    0 hello, 0 database desc, 0 link state req
    0 link state updates, 0 link state acks

    Sent: 0 total

    IP-IGRP2 statistics:
    Rcvd: 33522 total
    Sent: 99665 total

    PIMv2 statistics: Sent/Received
    Total: 0/0, 0 checksum errors, 0 format errors
    Registers: 0/0, Register Stops: 0/0, Hellos: 0/0
    Join/Prunes: 0/0, Asserts: 0/0, grafts: 0/0
    Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0
    State-Refresh: 0/0

    IGMP statistics: Sent/Received
    Total: 0/0, Format errors: 0/0, Checksum errors: 0/0
    Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 0/0
    DVMRP: 0/0, PIM: 0/0

    ARP statistics:
    Rcvd: 2390 requests, 4 replies, 0 reverse, 0 other
    Sent: 39 requests, 631 replies (0 proxy), 0 reverse
    Syn, Feb 18, 2005
    #10
  11. Syn

    Merv Guest

    the high CPU is due to most packets being processed switched - try
    disabling the tunnel checksum commands to see if this makes a
    difference

    You are also experience some level of fragmentation ( as seen in the
    put of show ip traffic). You need to have a look an the impact of
    GRE-ISPSEC tunnels impact on MTU. One deals with this by using
    prefragmentation or by lowering the MTU size on the tunnel interfaces

    see Cisco docs:

    http://cco.cisco.com/en/US/products/sw/iosswrel/ps1833/products_feature_guide09186a008009c92c.html

    http://cco.cisco.com/en/US/tech/tk583/tk372/technologies_configuration_example09186a0080093f85.shtml
    Merv, Feb 18, 2005
    #11
  12. Syn

    Ivan Ostreš Guest

    In article <4215c991$0$267$>,
    says...
    > Hmm this feature sounds interesting, a shame I didn't know about it
    > before ;)
    >
    > Here is the output of sh int stat after enabling:
    >
    > FastEthernet0/0
    > Switching path Pkts In Chars In Pkts Out Chars Out
    > Processor 205016 68224558 407967 74591204
    > Route cache 251876 135694673 84211 11272781
    > Total 456892 203919231 492178 85863985
    > FastEthernet0/1
    > Switching path Pkts In Chars In Pkts Out Chars Out
    > Processor 241140 44859485 223526 77414664
    > Route cache 84216 11273604 89690 98251225
    > Total 325356 56133089 313216 175665889
    > Tunnel11
    > Switching path Pkts In Chars In Pkts Out Chars Out
    > Processor 178744 76593262 174392 42792030
    > Route cache 0 0 0 0
    > Total 178744 76593262 174392 42792030
    >
    >


    As you can see here, none of the packets router trough tunnel interfaces
    is fast/cef switched. This might be due to fragmenting/reassembling
    packets. You could try to do 'ip mtu 1400' on tunnel interfaces to see
    how it goes...

    --
    -Ivan.

    *** Use Rot13 to see my eMail address ***
    Ivan Ostreš, Feb 18, 2005
    #12
  13. Syn

    Ben Guest

    Ivan Ostreš wrote:
    > In article <4215c991$0$267$>,
    > says...
    >
    >>Hmm this feature sounds interesting, a shame I didn't know about it
    >>before ;)
    >>
    >>Here is the output of sh int stat after enabling:
    >>
    >>FastEthernet0/0
    >> Switching path Pkts In Chars In Pkts Out Chars Out
    >> Processor 205016 68224558 407967 74591204
    >> Route cache 251876 135694673 84211 11272781
    >> Total 456892 203919231 492178 85863985
    >>FastEthernet0/1
    >> Switching path Pkts In Chars In Pkts Out Chars Out
    >> Processor 241140 44859485 223526 77414664
    >> Route cache 84216 11273604 89690 98251225
    >> Total 325356 56133089 313216 175665889
    >>Tunnel11
    >> Switching path Pkts In Chars In Pkts Out Chars Out
    >> Processor 178744 76593262 174392 42792030
    >> Route cache 0 0 0 0
    >> Total 178744 76593262 174392 42792030
    >>
    >>

    >
    >
    > As you can see here, none of the packets router trough tunnel interfaces
    > is fast/cef switched. This might be due to fragmenting/reassembling
    > packets. You could try to do 'ip mtu 1400' on tunnel interfaces to see
    > how it goes...
    >


    FY, this is a more healthy ratio of process to cef/fast-switched packets
    from my 1721:

    Protocol IP
    Switching path Pkts In Chars In Pkts Out Chars Out
    Process 765559 61207973 654110 62290490
    Cache misses 0 - - -
    Fast 11285043 1243496442 12932781 450470562
    Auton/SSE 0 0 0 0
    Ben, Feb 22, 2005
    #13
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Darren Parker

    High CPU load - 7200

    Darren Parker, Feb 11, 2004, in forum: Cisco
    Replies:
    1
    Views:
    576
    Gert Doering
    Feb 14, 2004
  2. hari
    Replies:
    3
    Views:
    1,747
    Hansang Bae
    Oct 5, 2004
  3. CPU Load on a 2600

    , Dec 1, 2005, in forum: Cisco
    Replies:
    9
    Views:
    1,829
  4. DavidM
    Replies:
    11
    Views:
    740
  5. Replies:
    5
    Views:
    1,095
    Aaron Leonard
    Nov 13, 2006
Loading...

Share This Page