7206 VXR high CPU usage

Discussion in 'Cisco' started by KHANS, Oct 22, 2003.

  1. KHANS

    KHANS Guest

    Dear Group.

    first of all my apologize for a long cut & paste down, We are experiencing
    high CPU utilization in our 7206 VXR Router, problem is when I supply
    command "show process cpu" it shows as an average of 62 to 70% most of the
    time, however when I calculate from each process it is not even 20%
    utilized, seems strange for me, would you please comment on it? appreciate
    all of your responses.

    =============================================================
    CPU utilization for five seconds: 62%/49%; one minute: 59%; five minutes:
    57%
    ==============================================================
    0.00% 0.00% 0.00% 0 Load Meter
    0.00% 0.00% 0.00% 0 PPP auth
    0.00% 0.15% 0.16% 0 Check heaps
    0.00% 0.00% 0.00% 0 Chunk Manager
    0.00% 0.00% 0.00% 0 Pool Manager
    0.00% 0.00% 0.00% 0 Timers
    0.00% 0.00% 0.00% 0 Serial Backgroun
    0.00% 0.00% 0.00% 0 ALARM_TRIGGER_SC
    0.00% 0.00% 0.00% 0 EnvMon
    0.00% 0.00% 0.00% 0 OIR Handler
    0.00% 0.00% 0.00% 0 IPC Zone Manager
    0.00% 0.00% 0.00% 0 IPC Periodic Tim
    0.00% 0.00% 0.00% 0 IPC Seat Manager
    1.30% 1.35% 1.36% 0 ARP Input
    0.08% 0.00% 0.00% 0 HC Counter Timer
    0.00% 0.00% 0.00% 0 DDR Timers
    0.00% 0.00% 0.00% 0 Dialer event
    0.00% 0.00% 0.00% 0 Entity MIB API
    0.00% 0.00% 0.00% 0 SERIAL A'detect
    0.00% 0.00% 0.00% 0 Critical Bkgnd
    0.08% 0.07% 0.07% 0 Net Background
    0.00% 0.00% 0.00% 0 Logger
    5Sec 1Min 5Min TTY Process
    0.00% 0.00% 0.00% 0 TTY Background
    0.08% 0.00% 0.00% 0 Per-Second Jobs
    0.00% 0.00% 0.00% 0 Inode Table Dest
    0.00% 0.00% 0.00% 0 Inode Table Refr
    0.00% 0.00% 0.00% 0 CSP Timer
    0.00% 0.00% 0.00% 0 Hawkeye Backgrou
    0.00% 0.00% 0.00% 0 VNM DSPRM MAIN
    0.00% 0.00% 0.00% 0 CES Line Conditi
    0.00% 0.00% 0.00% 0 Flash MIB Update
    0.00% 0.00% 0.00% 0 ATM OAM Input
    0.00% 0.00% 0.00% 0 ATM OAM TIMER
    0.00% 0.00% 0.00% 0 ATM Periodic
    0.00% 0.00% 0.00% 0 ATM ARP INPUT
    0.00% 0.00% 0.00% 0 ATM PA Helper
    0.00% 0.00% 0.00% 0 TurboACL
    0.00% 0.00% 0.00% 0 AAA Dictionary R
    1.30% 1.39% 1.09% 0 IP Input
    0.00% 0.00% 0.00% 0 AAA Accounting
    0.00% 0.00% 0.00% 0 PASVC create VA
    0.00% 0.00% 0.00% 0 X.25 Encaps Mana
    0.00% 0.00% 0.00% 0 PPP IP Add Route
    0.00% 0.00% 0.00% 0 SSCOP Input
    0.00% 0.00% 0.00% 0 SSCOP Output
    5Sec 1Min 5Min TTY Process
    0.00% 0.00% 0.00% 0 SSCOP Timer
    0.00% 0.01% 0.00% 0 IP Background
    0.00% 0.00% 0.00% 0 SNMP Timers
    0.00% 0.00% 0.00% 0 ILMI Input
    0.00% 0.00% 0.00% 0 ILMI Request
    0.00% 0.00% 0.00% 0 ILMI Response
    0.00% 0.00% 0.00% 0 ILMI Timer Proce
    0.00% 0.00% 0.00% 0 ATM PVC Discover
    0.00% 0.00% 0.00% 0 ATMSIG ILMI Time
    0.00% 0.00% 0.00% 0 ATMSIG DRIVERAPI
    0.00% 0.00% 0.00% 0 ATMSIG Timer
    0.00% 0.00% 0.00% 0 ATMSIG Input
    0.00% 0.00% 0.00% 0 ATMSIG Client
    0.00% 0.00% 0.00% 0 Adj Manager
    0.00% 0.00% 0.00% 0 HTTP Timer
    0.00% 0.00% 0.00% 0 TCP Timer
    0.00% 0.00% 0.00% 0 TCP Protocols
    0.00% 0.00% 0.00% 0 Probe Input
    0.00% 0.00% 0.00% 0 RARP Input
    0.00% 0.00% 0.00% 0 Socket Timers
    0.00% 0.00% 0.00% 0 DHCPD Receive
    0.00% 0.00% 0.00% 0 IP Cache Ager
    0.00% 0.00% 0.00% 0 COPS
    5Sec 1Min 5Min TTY Process
    0.00% 0.00% 0.00% 0 PAD InCall
    0.00% 0.00% 0.00% 0 X.25 Background
    0.00% 0.00% 0.00% 0 gk process
    0.00% 0.00% 0.34% 2 Virtual Exec
    0.00% 0.00% 0.00% 0 Net Serv Timer
    0.00% 0.00% 0.00% 0 Router Autoconf
    0.00% 0.00% 0.00% 0 TSP
    0.00% 0.00% 0.00% 0 QOS_MODULE_MAIN
    0.00% 0.00% 0.00% 0 CCVPM_HTSP
    0.00% 0.00% 0.00% 0 CCVPM_R2
    0.00% 0.00% 0.00% 0 CCSWVOICE
    0.00% 0.00% 0.00% 0 Background Loade
    0.00% 0.00% 0.00% 0 sssapp
    0.00% 0.00% 0.00% 0 Proxy Session Ap
    0.00% 0.00% 0.00% 0 RMON Recycle Pro
    0.00% 0.00% 0.00% 0 RMON Deferred Se
    0.00% 0.00% 0.00% 0 SYSMGT Events
    0.00% 0.00% 0.00% 0 RMON Packets
    0.00% 0.00% 0.00% 0 BUSYOUT SCAN
    0.00% 0.00% 0.00% 0 trunk conditioni
    0.00% 0.00% 0.00% 0 trunk conditioni
    0.00% 0.00% 0.00% 0 Net Input
    0.48% 0.37% 0.34% 0 Compute load avg
    5Sec 1Min 5Min TTY Process
    0.00% 0.01% 0.00% 0 Per-minute Jobs
    0.00% 0.00% 0.00% 0 CC-API_VCM
    0.00% 0.00% 0.00% 0 CCPROXY_CT
    0.00% 0.04% 0.05% 0 CEF process
    0.00% 0.00% 0.00% 0 L2F management d
    0.00% 0.00% 0.00% 0 L2F data daemon
    0.00% 0.00% 0.00% 0 L2TP data daemon
    0.00% 0.00% 0.00% 0 L2TP mgmt daemon
    0.00% 0.46% 0.44% 0 PPPOE discovery
    0.00% 0.00% 0.00% 0 PPPOE background
    0.00% 0.00% 0.00% 0 PPTP Mgmt
    0.00% 0.00% 0.00% 0 PPTP Data
    0.00% 0.00% 0.00% 0 TCP Driver
    0.00% 0.00% 0.00% 0 TCP Listener
    0.00% 0.00% 0.00% 0 IP Flow Backgrou
    1.03% 2.52% 2.68% 0 HyBridge Input P
    0.00% 0.00% 0.00% 0 Tbridge Monitor
    0.08% 0.02% 0.00% 0 Spanning Tree
    0.00% 0.02% 0.00% 0 CEF Scanner
    0.00% 0.00% 0.00% 0 CCP manager
    0.24% 0.06% 0.06% 0 PPP manager
    0.00% 0.00% 0.00% 0 Multilink PPP
    0.00% 0.00% 0.00% 0 Multilink PPP ou
    5Sec 1Min 5Min TTY Process
    0.00% 0.00% 0.00% 0 Multilink event
    0.00% 0.00% 0.00% 0 VPDN failure man
    0.00% 0.18% 0.09% 0 IP SNMP
    0.00% 0.13% 0.06% 0 PDU DISPATCHER
    0.00% 0.86% 0.54% 0 SNMP ENGINE
    0.00% 0.00% 0.00% 0 SNMP ConfCopyPro
    0.00% 0.00% 0.00% 0 SNMP Traps
    0.08% 0.00% 0.00% 0 RADIUS
    0.00% 0.00% 0.00% 0 NTP
    0.00% 0.00% 0.00% 0 Time Range Proce
    0.00% 0.00% 0.00% 0 xcpa-driver
    0.81% 0.59% 0.49% 0 VTEMPLATE Backgr
    0.00% 0.00% 0.00% 0 DHCPD Timer
    0.00% 0.00% 0.00% 0 DHCPD Database
    0.00% 0.00% 0.00% 0 BGP Router
    0.00% 0.00% 0.00% 0 BGP I/O
    0.00% 0.07% 0.09% 0 BGP Scanner


    Thanks.
     
    KHANS, Oct 22, 2003
    #1
    1. Advertisements

  2. KHANS

    Zero kool Guest

    Post the configuration if possible. What kind of links are coming into the
    router? Also what switching method are you using??
     
    Zero kool, Oct 22, 2003
    #2
    1. Advertisements

  3. This is a good question. The CPU utilization statistics you see at
    the top are:

    CPU utilization for five seconds: x%/y%
    x = total processor utilization for five seconds (including
    interrupts)
    y = interrupt level processor utilization for five seconds

    one minute: z%
    z = total processor utilization average over one minute (inluding
    interrupts)

    five: w%
    w = total processor utilization average over five minute (inluding
    interrupts)

    When you look at the individual processes, this shows the stats for
    those processes only, interrupts are not accounted for here at all.
    So it is the interrupts that throw you off on the totals. The five
    second total cpu (x) should be the sum of the interrupt (y) and the
    total of your five second process utilizations.

    So, basically what you are seeing is a lot of CPU interrupts. Check
    to see if have ip route-caching turned off on your interfaces, that
    will cause high cpu based on interrupts. There are other things that
    will cause this, but you should check with TAC.

    HTH,

    Mike
     
    Mike Gallagher, Oct 22, 2003
    #3
  4. As Mike mentioned, the number after the slash is the CPU usage caused
    by interrupts, which doesn't show up as a process in the command
    output. How much traffic is going through the device? Packet
    forwarding is done in interrupt context when the forwarding method is
    via the route-cache or CEF, so your CPU usage due to interrupts will
    increase as the traffic load increases.

    -Terry
     
    Terry Baranski, Oct 23, 2003
    #4
  5. KHANS

    N. Hall Guest

    Could be virus related. Run a sniffer on it and look for exceptionially
    high numbers of ARP or ICMP traffic especially when it is sequentially going
    through ip addresses.

    Just today by using a sniffer (Etherreal is free) to find about 20 Nachi
    infected PC's, I took our utilization on our 4006 from 25% down to 0% -
    1% -- I was seeing about 400 ARP requests per second at the height of it.
     
    N. Hall, Oct 24, 2003
    #5
  6. Well, in the below output you can see the "ARP input" process using
    1.30% CPU, and the majority of the usage is interrupt, which means
    packet forwarding.

    That said, the traffic could be a worm "attaching" something, but then
    it would not be broadcast traffic, and he would need to configure a span
    port on a switch, and put the sniffer there.

    But changes are, it's just "normal" traffic, I'd not be concerned
    about 50% CPU usage for interrupt.

    /Jesper
     
    Jesper Skriver, Oct 24, 2003
    #6
  7. KHANS

    KHANS Guest

    hi,

    This router is our gateway router and we have configured PBR with CEF in
    this router to load balance our L2tp traffic between two routers. I am only
    concerned with it's high cpu usage, we do have some DSL bridge & PPPOE
    customers on this router as well but I could not find still the source of
    the problem, however one more noticeable thing is the other two routers
    (Load balancer where most of L2tp traffic is terminating) are also having
    high cpu once this spikes up.

    thanks for your inputs.
     
    KHANS, Oct 25, 2003
    #7
  8. If you posted the entire configuration, show version, and gave us an idea on how
    many packets per second you route, we might be able to tell more.
     
    Jesper Skriver, Oct 25, 2003
    #8
  9. Hello Khans and Jesper!

    25 Oct 2003 13:08:51 GMT, Jesper Skriver wrote in comp.dcom.sys.cisco:


    JS> If you posted the entire configuration, show version, and gave us
    JS> an idea on how many packets per second you route, we might be able
    JS> to tell more.

    I've got similar problem with 7206VXR (NPE-G1). IOS C7200-IS-M, V 12.2(15)T8
    We have configured PBR with CEF. Suddenly NPE-G1 halt for few minutes with
    extremely high CPU load. Some output caught in the middle follows:

    #show proc cpu | exc 0.00% 0.00%
    CPU utilization for five seconds: 100%/89%; one minute: 98%; five minutes: 85%
    PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
    4 36048 1805 19971 0.00% 0.07% 0.22% 0 Check heaps
    5 920 1187 775 0.07% 0.02% 0.00% 0 Pool Manager
    13 4180 5577 749 0.00% 0.05% 0.01% 0 EnvMon
    19 620 981 632 0.00% 0.01% 0.00% 0 ARP Input
    27 8168 1809 4515 0.22% 0.08% 0.06% 0 Net Background
    30 4568 5585 817 0.07% 0.08% 0.07% 0 Per-Second Jobs
    45 1076516 295408 3644 8.85% 8.23% 13.56% 0 IP Input
    55 12044 150 80293 0.00% 0.08% 0.07% 0 IP Background
    61 7516 8099 928 0.00% 0.08% 0.06% 0 CEF process
    62 2728 4668 584 0.00% 0.04% 0.02% 0 TCP Timer
    68 53672 96 559083 0.98% 0.12% 0.10% 0 IP Cache Ager
    98 4764 1913 2490 0.00% 0.03% 0.04% 0 Exec
    113 540 51843 10 0.07% 0.01% 0.00% 0 CCPROXY_CT
    128 58016 36784 1577 0.00% 0.15% 0.12% 0 BGP Router
    130 257868 1113 231687 0.00% 0.64% 1.60% 0 BGP Scanner

    What is CCPROXY_CT process?
    How to stop CCPROXY_CT?

    #sh int gi0/2
    GigabitEthernet0/2 is up, line protocol is up
    Hardware is BCM1250 Internal MAC, address is 000c.cf5b.181a (bia 000c.cf5b.18
    Description: DG_backbone
    Internet address is 213.186.193.3/27
    MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
    reliability 255/255, txload 7/255, rxload 3/255
    Encapsulation ARPA, loopback not set
    Keepalive set (10 sec)
    Full-duplex, 1000Mb/s, media type is RJ45
    output flow-control is XON, input flow-control is XON
    ARP type: ARPA, ARP Timeout 04:00:00
    Last input 00:00:00, output 00:00:00, output hang never
    Last clearing of "show interface" counters never
    Input queue: 0/4096/9109/598679 (size/max/drops/flushes); Total output drops: 129
    Queueing strategy: random early detection(RED)
    5 minute input rate 14203000 bits/sec, 8183 packets/sec
    5 minute output rate 28539000 bits/sec, 7128 packets/sec
    47937335 packets input, 1563038313 bytes, 26154 no buffer
    Received 0 broadcasts, 0 runts, 0 giants, 297 throttles
    0 input errors, 0 CRC, 0 frame, 5779 overrun, 0 ignored
    0 watchdog, 1 multicast, 0 pause input
    0 input packets with dribble condition detected
    41959425 packets output, 830831245 bytes, 0 underruns
    0 output errors, 0 collisions, 5 interface resets

    Interface GigabitEthernet0/2 connected with one host only.
    So all L2 traffic is correct:

    # sh int gi0/2 mac
    GigabitEthernet0/2 DG_backbone
    Input (511 free)
    00e0.8121.03e3(160): 48713381 packets, 10331M bytes, last: 0ms ago
    Total: 48713381 packets, 10331M bytes
    Output (511 free)
    00e0.8121.03e3(160): 41988788 packets, 22312M bytes, last: 0ms ago
    Total: 41988788 packets, 22312M bytes

    But we have large RP drops & throttles:

    # sh int gi0/2 switching
    GigabitEthernet0/2 DG_backbone
    Throttle count 876
    Drops RP 12418 SP 0
    SPD Flushes Fast 939842 SSE 0
    SPD Aggress Fast 0
    SPD Priority Inputs 535 Drops 0

    Protocol Path Pkts In Chars In Pkts Out Chars Out
    Other Process 0 0 1086 65160
    Cache misses 0
    Fast 0 0 0 0
    Auton/SSE 0 0 0 0
    IP Process 3721073 360664177 31768 18516445
    Cache misses 2131
    Fast 94584148 3336935082 85507443 2450591316
    Auton/SSE 0 0 0 0
    ARP Process 471 28260 257 15420
    Cache misses 0
    Fast 0 0 0 0
    Auton/SSE 0 0 0 0
    CDP Process 0 0 183 65880
    Cache misses 0
    Fast 0 0 0 0

    # show ip cef
    Prefix Next Hop Interface
    0.0.0.0/0 drop Null0 (default route handler entry)
    0.0.0.0/32 receive
    3.0.0.0/8 212.44.92.2 (?)
    4.0.0.0/8 212.44.92.2 (?)
    4.17.225.0/24 212.44.92.2 (?)
    4.17.226.0/23 212.44.92.2 (?)
    4.17.251.0/24 212.44.92.2 (?)
    4.17.252.0/23 212.44.92.2 (?)

    CEF forgot info about exit interface!!!

    # show ip sockets
    Proto Remote Port Local Port In Out Stat TTY OutputIF
    17 10.99.99.24 514 10.99.99.35 54267 0 0 20 0
    17 10.99.99.24 49 10.99.99.35 49 0 0 11 0
    17 213.186.192.24 162 213.186.193.3 51750 0 0 0 0
    17 10.99.99.24 47090 10.99.99.35 161 0 0 1 0
    17 --listen-- 213.186.193.182 162 0 0 11 0
    17 --listen-- 213.186.193.182 57994 0 0 11 0
    17 0.0.0.0 0 213.186.193.182 67 0 0 2211 0
    17 --listen-- 213.186.193.182 123 0 0 1 0

    What is 57994 UDP port?
     
    Sergi Polischuk, Oct 27, 2003
    #9
  10. I don't know - but it's unlikely that is where your problem is, as
    it's not using much cpu.

    Again, most of it is interrupt, that is packet forwarding, including the
    features applied to the packets.
    A symptom of a overloaded box, packets are arriving faster than they can
    be processed.
    If you only have one host, no need to enable MAC accounting is there, it
    all uses CPU ...
    No they are recursive routes, and you havn't got a route to the next-hop
    212.44.92.2 - so they will be unresolved.

    Please provide the configuration, it could be that a simple
    configuration change could fix it for you.
     
    Jesper Skriver, Oct 27, 2003
    #10
  11. KHANS

    leraph

    Joined:
    Feb 6, 2009
    Messages:
    1
    Likes Received:
    0
     
    leraph, Feb 6, 2009
    #11
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.