Input Drops With An Empty Input Queue

Discussion in 'Cisco' started by Spiz, May 9, 2005.

  1. Spiz

    Spiz Guest

    Hello,

    I have a Cisco 7206 (NPE200) with a serial interface that is hooked up
    to a T3. I have this strange problem where I am getting occassional
    input drops. They come in bursts of about 100, around 3 to 6 times a
    day, during periods of both little activity and peak usage. The input
    queue is never full or even half full, it is usually at 0. CPU
    utilization never goes above about 25% or so. We use only about 10% of
    our T3 during peak. There are no other errors on this or any other
    interface and no queues are being filled on any other interface (there
    are 3 other fastethernet interfaces).

    I've tried increasing the hold-queue in parameter for the interface and
    I've had mixed results. The higher I go, the less input drops I see
    until around a queue of about 200. Once I set it around that value or
    higher, I start to see slight increments in the no buffer and ignored
    counters on the interface. I've tried tweaking the buffers (show
    buffers output showed failures) following Cisco's guidelines
    (http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a00800a7b80.shtml)
    but I haven't had much luck their either unless I'm missing
    something.

    Any suggestions?

    Thanks,

    Spencer
     
    Spiz, May 9, 2005
    #1
    1. Advertising

  2. Spiz

    Guest

    My guess would be a burst of traffic hitting some interface.

    You will get drops if you are getting buffer failures and you may
    get drops with buffer misses (IIRC).

    If you have free memory you could increase
    the buffers until you stop getting any drops or misses.

    Either increase the permanent (which I prefer) or the min
    free. Do be careful that you have sufficient memory.
    Look at the "lowest" and the "largest".

    This router (all may do this) helpfully
    shows the peak number of actual buffers so increasing the
    permanent buffers to more this value will reduce the amount of
    allocation and free activity.

    sh buff
    buffers small permanent 100
    buffers small min-free 50
    buffers middle permanent 75
    buffers middle min-free 30


    sh ver
    uptime is 15 weeks, 1 day, 7 hours, 31 minutes

    sh buff
    Public buffer pools:
    Small buffers, 104 bytes (total 100, permanent 100, peak 136 @ 3w6d):
    100 in free list (50 min, 150 max allowed)
    48808491 hits, 1523 misses, 463 trims, 463 created
    0 failures (0 no memory)
    Middle buffers, 600 bytes (total 75, permanent 75, peak 105 @ 7w0d):
    73 in free list (30 min, 150 max allowed)
    16234811 hits, 700 misses, 160 trims, 160 created
    0 failures (0 no memory)
    Big buffers, 1536 bytes (total 50, permanent 50, peak 62 @ 2w0d):
    49 in free list (5 min, 150 max allowed)
    38306730 hits, 304 misses, 64 trims, 64 created
    49 failures (0 no memory)
    VeryBig buffers, 4520 bytes (total 10, permanent 10, peak 46 @ 7w0d):
    10 in free list (0 min, 100 max allowed)
    67340 hits, 40 misses, 80 trims, 80 created
    0 failures (0 no memory)


    #sh mem
    Head Total(b) Used(b) Free(b) Lowest(b)
    Largest(b)
    Processor 816A6990 24064624 10776824 13287800 13020740
    11456432
    I/O 2D99C00 2515968 780784 1735184 1565264
    1735020



    sh int | inc drop
    Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops:
    662
    Input queue: 1/75/1621/0 (size/max/drops/flushes); Total output
    drops: 0
    Output queue: 0/1000/64/0 (size/max total/threshold/drops)
    Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
    Input queue: 0/75/0/1057 (size/max/drops/flushes); Total output
    drops: 0
    Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
    Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
    Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
    Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
    Input queue: 0/4096/0/0 (size/max/drops/flushes); Total output drops:
    0
    Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

    ATM0 is up, line protocol is up
    Dialer1 is up, line protocol is up (spoofing)
    Virtual-Access2 is up, line protocol is up
    Ethernet0 is up, line protocol is up
    FastEthernet1 is up, line protocol is up
    FastEthernet2 is down, line protocol is down
    FastEthernet3 is down, line protocol is down
    FastEthernet4 is down, line protocol is down
    Virtual-Access1 is up, line protocol is up
    Virtual-Access2 is up, line protocol is up



    ########
    As you can see I have done some work but there is still
    some to go. This is a router with a single Ethernet and an ADSL.

    I may rack the buffers up some more today.
     
    , May 10, 2005
    #2
    1. Advertising

  3. Spiz

    Spiz Guest

    I've messed with the buffers with mixed results. I didn't track very
    well so I'm starting from square one again. I've set my buffers to the
    following:

    buffers small permanent 310
    buffers small max-free 400
    buffers small min-free 90
    buffers middle permanent 230
    buffers middle max-free 300
    buffers middle min-free 70
    buffers big permanent 100
    buffers big max-free 140
    buffers big min-free 30
    buffers verybig permanent 25
    buffers verybig max-free 40
    buffers verybig min-free 10
    buffers large permanent 6
    buffers large max-free 10
    buffers large min-free 2
    buffers huge permanent 6
    buffers huge max-free 10
    buffers huge min-free 2


    This seemed to work OK. I followed Cisco guidelines and eventually
    reached those numbers after a few days of adjustments. I didn't see
    any failures or misses with these settings but if I remember correctly,
    I still got slight increments in the ignored counter on the serial
    interface. I'm going to monitor the router with these buffer settings
    and the current input queue of 150 (probably going to end up bumping
    this into the 250 range like before if drops continue) and I'll post my
    results.

    Thanks for the response, much appreciated.

    Spencer
     
    Spiz, May 10, 2005
    #3
  4. Spiz

    Spiz Guest

    I'm seeing the ignored counter increase with these buffer settings.
    The "show buffers" output does not show incrementing failures/misses
    for the public buffer pools either. I was still seeing input drops
    with an input queue of 175 so I've bumped it to 200.

    I'm not sure what is going on here...ungh.
     
    Spiz, May 11, 2005
    #4
  5. Spiz

    Guest

    Read the definitions of throttles and ignored.

    I can't recall right now.

    There at least three different ways that input packets can be lost

    Throttles (Interrupt throttles) [I guess a CPU limitation]
    Ignores ???
    Input Drops ???

    Throttles may not always be listed in the sh int. Have a
    look at sh controllers if it's not.

    If this is a real user affecting issue you may need
    to get your wallet out and get a beefier router.
    Get your suplier to supply against your requirement so that
    they have to fix it if it doesn't work.

    Consider looking at scheduller allocate command.
    Also "sheduler interval" ?
    The CPU may be too busy with something to deal with the packets.
    Aim to free up the router to do a variety of tasks
    as opposed to keeping it's head down finishing one this
    before going on to another.
    Think of Windows 'Server mode' vs 'workstation mode'
    optimisations.
    MyComputer/Properties/Advanced/PerformanceSettings/
    Advanced/ProcessorScheduling/
    programs vs Background services.

    Go for programs.


    Reducing for example the min timeslice that non interrupt tasks
    get may free time for packet processing. You may of course
    stop the router from working at all:)

    sh proc cpu
    CPU utilization for five seconds: 12%/6%; one minute: 11%; five
    minutes: 9%
    PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

    130 16719024 80474 207762 0.00% 0.17% 0.19% 0 crypto
    sw pk pro

    The "207762" above is the average time in microseconds
    that this process runs for on each
    invocation. I suspect that this is the root cause of my
    packet loss on this router since I suspect that it will
    not be able to do anything else during this period.

    It is however not affecting the users significantly so
    we are living with it. (bit of web browsing).

    TCP retransmissions are wonderful<g>.
     
    , May 13, 2005
    #5
  6. Spiz

    Spiz Guest

    I'm not seeing any throttles on the serial interface, just the ignored
    and occassional drop at this point. I'm still tuning the buffers, I
    have things working better but I'm still seeing an occassional small
    burst of ignored and drop errors, sometimes on average only one burst
    per 24 hours. I'm hoping this is just a buffer tuning/input queue
    tuning issue.

    A little background info on the interface that has me wonder sometimes
    if it has something to do with cabling. When I first took over for
    this router, the serial interface was spewing tons of input errors and
    crc errors. I ended up swapping out the DS3 cable and the errors went
    away but then I noticed input queue flushes. I started increasing the
    hold-queue in value for the serial interface and the flushes went away
    but then the input drops started coming through with the ignored
    errors.

    I'm going to continue tweakin the buffers and hold-queue and see where
    I can get. I don't think this problem is having really any impact on
    our operations but it would be nice to eliminate this annoyance.

    Fun fun fun,

    Spencer
     
    Spiz, May 13, 2005
    #6
  7. Spiz

    Guest

    There are few options to work around problem of CPU overload and
    thereby resulting in input drops:

    1. Using CEF.
    2. Using SPD headroom tuning.

    Would be worth checking these options too.
    HTH

    Guru Prashant



    wrote:
    > Read the definitions of throttles and ignored.
    >
    > I can't recall right now.
    >
    > There at least three different ways that input packets can be lost
    >
    > Throttles (Interrupt throttles) [I guess a CPU limitation]
    > Ignores ???
    > Input Drops ???
    >
    > Throttles may not always be listed in the sh int. Have a
    > look at sh controllers if it's not.
    >
    > If this is a real user affecting issue you may need
    > to get your wallet out and get a beefier router.
    > Get your suplier to supply against your requirement so that
    > they have to fix it if it doesn't work.
    >
    > Consider looking at scheduller allocate command.
    > Also "sheduler interval" ?
    > The CPU may be too busy with something to deal with the packets.
    > Aim to free up the router to do a variety of tasks
    > as opposed to keeping it's head down finishing one this
    > before going on to another.
    > Think of Windows 'Server mode' vs 'workstation mode'
    > optimisations.
    > MyComputer/Properties/Advanced/PerformanceSettings/
    > Advanced/ProcessorScheduling/
    > programs vs Background services.
    >
    > Go for programs.
    >
    >
    > Reducing for example the min timeslice that non interrupt tasks
    > get may free time for packet processing. You may of course
    > stop the router from working at all:)
    >
    > sh proc cpu
    > CPU utilization for five seconds: 12%/6%; one minute: 11%; five
    > minutes: 9%
    > PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY

    Process
    >
    > 130 16719024 80474 207762 0.00% 0.17% 0.19% 0 crypto
    > sw pk pro
    >
    > The "207762" above is the average time in microseconds
    > that this process runs for on each
    > invocation. I suspect that this is the root cause of my
    > packet loss on this router since I suspect that it will
    > not be able to do anything else during this period.
    >
    > It is however not affecting the users significantly so
    > we are living with it. (bit of web browsing).
    >
    > TCP retransmissions are wonderful<g>.
     
    , May 14, 2005
    #7
  8. Spiz

    Guest

    One thing I forgot.

    You could consider applying access lists to
    block unwanted traffic. This is a good _first_step.
    Sorry I forgot it.


    I am not sure what you might want on your serial
    interface however block directed IP broadcasts at
    least.

    http://www.cisco.com/en/US/products/hw/routers/ps167/products_tech_note09186a0080094320.shtml

    Says:
    "As mentioned above, the only packets that are sent
    to the input queue are the ones that are destined to
    the router itself."

    It does not seem to say this above, it does say:

    "When a packet enters the router, the router
    attempts to forward it at interrupt level. If a match
    cannot be found in an appropriate cache table, the
    packet is queued in the incoming interface's input
    queue for processing."

    Very confusing.

    You might consider the following which may
    identify the nature of the problem traffic.

    Create an access list to deny known traffic
    other than valid inbound traffic on the serial interface.
    Disable - console, terminal, trap logging.
    Enable - buffered logging
    Create a large logging buffer.
    Set up some monitoring of the input drops
    so that the logging buffer can be captured when
    the event occurs before the buffer wraps.

    Enable "deb ip pac xxx"

    Clearly this may well crash the router or maybe it won't.
    You will have to investigate this yourself however the
    steps above will I believe minimise the impact on the router.

    Initially leave fast switching enabled. This should mean that
    _only_ process switched traffic will be logged but this might
    do the trick.

    If you don't get to see the nature of the traffic you could then
    try disabling fast switching.

    Depends on how much you want to worry about it.

    It does not sound as if you are experiencing user
    visible problems. Interesting though.
     
    , May 15, 2005
    #8
  9. Spiz

    Spiz Guest

    Thanks for the input, I'm going to follow up on those suggestions. In
    the meantime, since I left Friday, here is what I saw on my serial T3
    interface in question:

    Serial4/0 is up, line protocol is up
    Hardware is M1T-T3 pa
    MTU 4470 bytes, BW 44210 Kbit, DLY 200 usec,
    reliability 255/255, txload 7/255, rxload 7/255
    Encapsulation FRAME-RELAY IETF, crc 16, loopback not set
    Keepalive set (8 sec)
    Restart-Delay is 0 secs
    LMI enq sent 28894, LMI stat recvd 28894, LMI upd recvd 0, DTE LMI
    up
    LMI enq recvd 0, LMI stat sent 0, LMI upd sent 0
    LMI DLCI 0 LMI type is ANSI Annex D frame relay DTE
    Broadcast queue 0/256, broadcasts sent/dropped 3852/0, interface
    broadcasts 0
    Last input 00:00:00, output 00:00:00, output hang never
    Last clearing of "show interface" counters 2d16h
    Input queue: 0/210/49/0 (size/max/drops/flushes); Total output drops:
    0
    Queueing strategy: fifo
    Output queue: 0/40 (size/max)
    5 minute input rate 1223000 bits/sec, 252 packets/sec
    5 minute output rate 1319000 bits/sec, 258 packets/sec
    114387931 packets input, 1071347993 bytes, 0 no buffer
    Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
    0 parity
    0 input errors, 0 CRC, 0 frame, 0 overrun, 201 ignored, 0 abort
    119572070 packets output, 583407540 bytes, 0 underruns
    0 output errors, 0 applique, 0 interface resets
    0 output buffer failures, 0 output buffers swapped out
    0 carrier transitions
    rxLOS inactive, rxLOF inactive, rxAIS inactive
    txAIS inactive, rxRAI inactive, txRAI inactive

    I'm no longer seeing the no buffer errors but still a slight increment
    in drops and ignored. I compared output from the "show buffers"
    command from Friday and today and there were no failures/misses/trims
    for the public buffer pools.

    I log CPU utilization using MRTG/SiteScope and I haven't seen any
    utilization above 35% for the week. I'm using CEF. I think I've
    disabled fast switching before and it didn't work out very well, if I
    recall I saw tons of errors.

    You mentioned the ACL and that made me think of something. I took over
    this router from the previous admin and I have always questioned the
    ACL configuration. Basically everything is pretty much allowed through
    the serial T3 interface THEN everything is blocked on the interface
    servicing our network that is accessible by the public. We only allow
    web/ftp type traffic to certain servers on that public network. Would
    it be better to have this on the serial T3 interface?

    Spencer
     
    Spiz, May 16, 2005
    #9
  10. Spiz

    Spiz Guest

    I moved the ACL from the inside interface servicing our public network
    to the outside, serial T3 interface. I didn't see any drops or ignored
    errors since making the change. I noticed that the previous admin also
    allowed ICMP traffic to hit all our public servers. I switched that
    off and only allowed echo-replies back in so people inside our network
    can ping out. Hopefully the interface will continue to show the same
    stats.

    Spencer
     
    Spiz, May 17, 2005
    #10
  11. Spiz

    Guest

    Well done.

    Prevention is better than cure.

    As you have found out it is usually best to deny traffic as early as
    possible.

    I am not much in favour of denying all diagnostics, but it clearly
    depends
    on the circumstances. e.g. Cisco let you ping them.

    I bumped into this today when looking for something else.

    http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a0080094791.shtml

    It has some interesting material that I have not seen before.

    e.g.

    show buffers input-interface serial 0/0

    show buffers input-interface [interface type] [interface number] header


    router#show interfaces ethernet 0/0 mac-accounting
    Ethernet0/0
    Input(494 free)
    0000.0c5d.92f9(58 ): 1 packets, 106 bytes, last: 4038ms ago
    0004.c059.c060(61 ): 0 packets, 0 bytes, last: 2493135ms ago
    00b0.64bc.4860(64 ): 1 packets, 106 bytes, last: 20165ms ago
    0090.f2c9.cc00(103): 12 packets, 720 bytes, last: 3117ms ago
    Total: 14 packets, 932 bytes
    Output (511 free)
    0090.f2c9.cc00(103): 8 packets, 504 bytes, last: 4311ms ago
    Total: 8 packets, 504 bytes


    Could be handy one day.
     
    , May 17, 2005
    #11
  12. Spiz

    aservin Guest

    Good document.

    We found input/output queue drops in our Internet routers. It was
    because the LAN Interface was receiving too much traffic to be handled
    for the wan Interface (FE vs E3). We usually saw that when we had
    infected PC in our network trying to spread the virus.

    If after permit the echo in your net you saw a dicrease of the
    drops, I could be that you have some infected machines sending echos to
    the Internet. I would check which ip address are sending pings (or
    echos) and how frequently.

    -as


    wrote:
    > Well done.
    >
    > Prevention is better than cure.
    >
    > As you have found out it is usually best to deny traffic as early as
    > possible.
    >
    > I am not much in favour of denying all diagnostics, but it clearly
    > depends
    > on the circumstances. e.g. Cisco let you ping them.
    >
    > I bumped into this today when looking for something else.
    >
    >

    http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a0080094791.shtml
    >
    > It has some interesting material that I have not seen before.
    >
    > e.g.
    >
    > show buffers input-interface serial 0/0
    >
    > show buffers input-interface [interface type] [interface number]

    header
    >
    >
    > router#show interfaces ethernet 0/0 mac-accounting
    > Ethernet0/0
    > Input(494 free)
    > 0000.0c5d.92f9(58 ): 1 packets, 106 bytes, last: 4038ms ago
    > 0004.c059.c060(61 ): 0 packets, 0 bytes, last: 2493135ms ago
    > 00b0.64bc.4860(64 ): 1 packets, 106 bytes, last: 20165ms ago
    > 0090.f2c9.cc00(103): 12 packets, 720 bytes, last: 3117ms ago
    > Total: 14 packets, 932 bytes
    > Output (511 free)
    > 0090.f2c9.cc00(103): 8 packets, 504 bytes, last: 4311ms ago
    > Total: 8 packets, 504 bytes
    >
    >
    > Could be handy one day.
     
    aservin, May 18, 2005
    #12
  13. Spiz

    Spiz Guest

    I can say with confidence that this problem is resolved. I haven't
    seen any drops or ignored errors since making the changes. Thanks for
    the info anybod. Aservin, fortunately we don't have any rogue machines
    within the network. I'm logging echo-replies coming back in as I allow
    pings (only out from within our network) and I don't see anything
    strange. We were gettin hammered by all the people portscanning and
    what not our class-c range, when echos to our public network servers
    were allowed before.

    Serial4/0 is up, line protocol is up
    Hardware is M1T-T3 pa
    MTU 4470 bytes, BW 44210 Kbit, DLY 200 usec,
    reliability 255/255, txload 6/255, rxload 5/255
    Encapsulation FRAME-RELAY IETF, crc 16, loopback not set
    Keepalive set (8 sec)
    Restart-Delay is 0 secs
    LMI enq sent 19909, LMI stat recvd 19909, LMI upd recvd 0, DTE LMI
    up
    LMI enq recvd 0, LMI stat sent 0, LMI upd sent 0
    LMI DLCI 0 LMI type is ANSI Annex D frame relay DTE
    Broadcast queue 0/256, broadcasts sent/dropped 2654/0, interface
    broadcasts 0
    Last input 00:00:00, output 00:00:00, output hang never
    Last clearing of "show interface" counters 1d20h
    Input queue: 0/210/0/0 (size/max/drops/flushes); Total output drops:
    0
    Queueing strategy: fifo
    Output queue: 0/40 (size/max)
    5 minute input rate 928000 bits/sec, 197 packets/sec
    5 minute output rate 1113000 bits/sec, 199 packets/sec
    79925516 packets input, 2547962089 bytes, 0 no buffer
    Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
    0 parity
    0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
    85124305 packets output, 2471220331 bytes, 0 underruns
    0 output errors, 0 applique, 0 interface resets
    0 output buffer failures, 0 output buffers swapped out
    0 carrier transitions
    rxLOS inactive, rxLOF inactive, rxAIS inactive
    txAIS inactive, rxRAI inactive, txRAI inactive

    Aaahhhh...nice and clean.

    Spencer
     
    Spiz, May 18, 2005
    #13
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. blu_aqua
    Replies:
    5
    Views:
    4,392
    AnyBody43
    Nov 30, 2004
  2. Anthrax

    Re: Queue Drops

    Anthrax, Dec 9, 2005, in forum: Cisco
    Replies:
    3
    Views:
    3,232
  3. hack.bac

    if input / output drops

    hack.bac, Apr 27, 2007, in forum: Cisco
    Replies:
    3
    Views:
    1,079
  4. garywi

    Wireless Connection Drops, then connects, drops...

    garywi, Feb 12, 2009, in forum: Wireless Networking
    Replies:
    1
    Views:
    701
    Robert L. \(MS-MVP\)
    Feb 12, 2009
  5. 8ball meme
    Replies:
    7
    Views:
    1,978
    8ball meme
    Nov 18, 2010
Loading...

Share This Page