Input Drops With An Empty Input Queue

Hello,

I have a Cisco 7206 (NPE200) with a serial interface that is hooked up
to a T3. I have this strange problem where I am getting occassional
input drops. They come in bursts of about 100, around 3 to 6 times a
day, during periods of both little activity and peak usage. The input
queue is never full or even half full, it is usually at 0. CPU
utilization never goes above about 25% or so. We use only about 10% of
our T3 during peak. There are no other errors on this or any other
interface and no queues are being filled on any other interface (there
are 3 other fastethernet interfaces).

I've tried increasing the hold-queue in parameter for the interface and
I've had mixed results. The higher I go, the less input drops I see
until around a queue of about 200. Once I set it around that value or
higher, I start to see slight increments in the no buffer and ignored
counters on the interface. I've tried tweaking the buffers (show
buffers output showed failures) following Cisco's guidelines
(http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a00800a7b80.shtml)
but I haven't had much luck their either unless I'm missing
something.

Any suggestions?

Thanks,

Spencer

 

My guess would be a burst of traffic hitting some interface.

You will get drops if you are getting buffer failures and you may
get drops with buffer misses (IIRC).

If you have free memory you could increase
the buffers until you stop getting any drops or misses.

Either increase the permanent (which I prefer) or the min
free. Do be careful that you have sufficient memory.
Look at the "lowest" and the "largest".

This router (all may do this) helpfully
shows the peak number of actual buffers so increasing the
permanent buffers to more this value will reduce the amount of
allocation and free activity.

sh buff
buffers small permanent 100
buffers small min-free 50
buffers middle permanent 75
buffers middle min-free 30


sh ver
uptime is 15 weeks, 1 day, 7 hours, 31 minutes

sh buff
Public buffer pools:
Small buffers, 104 bytes (total 100, permanent 100, peak 136 @ 3w6d):
100 in free list (50 min, 150 max allowed)
48808491 hits, 1523 misses, 463 trims, 463 created
0 failures (0 no memory)
Middle buffers, 600 bytes (total 75, permanent 75, peak 105 @ 7w0d):
73 in free list (30 min, 150 max allowed)
16234811 hits, 700 misses, 160 trims, 160 created
0 failures (0 no memory)
Big buffers, 1536 bytes (total 50, permanent 50, peak 62 @ 2w0d):
49 in free list (5 min, 150 max allowed)
38306730 hits, 304 misses, 64 trims, 64 created
49 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 10, permanent 10, peak 46 @ 7w0d):
10 in free list (0 min, 100 max allowed)
67340 hits, 40 misses, 80 trims, 80 created
0 failures (0 no memory)


#sh mem
Head Total(b) Used(b) Free(b) Lowest(b)
Largest(b)
Processor 816A6990 24064624 10776824 13287800 13020740
11456432
I/O 2D99C00 2515968 780784 1735184 1565264
1735020



sh int | inc drop
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops:
662
Input queue: 1/75/1621/0 (size/max/drops/flushes); Total output
drops: 0
Output queue: 0/1000/64/0 (size/max total/threshold/drops)
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Input queue: 0/75/0/1057 (size/max/drops/flushes); Total output
drops: 0
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Input queue: 0/4096/0/0 (size/max/drops/flushes); Total output drops:
0
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

ATM0 is up, line protocol is up
Dialer1 is up, line protocol is up (spoofing)
Virtual-Access2 is up, line protocol is up
Ethernet0 is up, line protocol is up
FastEthernet1 is up, line protocol is up
FastEthernet2 is down, line protocol is down
FastEthernet3 is down, line protocol is down
FastEthernet4 is down, line protocol is down
Virtual-Access1 is up, line protocol is up
Virtual-Access2 is up, line protocol is up



########
As you can see I have done some work but there is still
some to go. This is a router with a single Ethernet and an ADSL.

I may rack the buffers up some more today.

 

I've messed with the buffers with mixed results. I didn't track very
well so I'm starting from square one again. I've set my buffers to the
following:

buffers small permanent 310
buffers small max-free 400
buffers small min-free 90
buffers middle permanent 230
buffers middle max-free 300
buffers middle min-free 70
buffers big permanent 100
buffers big max-free 140
buffers big min-free 30
buffers verybig permanent 25
buffers verybig max-free 40
buffers verybig min-free 10
buffers large permanent 6
buffers large max-free 10
buffers large min-free 2
buffers huge permanent 6
buffers huge max-free 10
buffers huge min-free 2


This seemed to work OK. I followed Cisco guidelines and eventually
reached those numbers after a few days of adjustments. I didn't see
any failures or misses with these settings but if I remember correctly,
I still got slight increments in the ignored counter on the serial
interface. I'm going to monitor the router with these buffer settings
and the current input queue of 150 (probably going to end up bumping
this into the 250 range like before if drops continue) and I'll post my
results.

Thanks for the response, much appreciated.

Spencer

 

I'm seeing the ignored counter increase with these buffer settings.
The "show buffers" output does not show incrementing failures/misses
for the public buffer pools either. I was still seeing input drops
with an input queue of 175 so I've bumped it to 200.

I'm not sure what is going on here...ungh.

 

Read the definitions of throttles and ignored.

I can't recall right now.

There at least three different ways that input packets can be lost

Throttles (Interrupt throttles) [I guess a CPU limitation]
Ignores ???
Input Drops ???

Throttles may not always be listed in the sh int. Have a
look at sh controllers if it's not.

If this is a real user affecting issue you may need
to get your wallet out and get a beefier router.
Get your suplier to supply against your requirement so that
they have to fix it if it doesn't work.

Consider looking at scheduller allocate command.
Also "sheduler interval" ?
The CPU may be too busy with something to deal with the packets.
Aim to free up the router to do a variety of tasks
as opposed to keeping it's head down finishing one this
before going on to another.
Think of Windows 'Server mode' vs 'workstation mode'
optimisations.
MyComputer/Properties/Advanced/PerformanceSettings/
Advanced/ProcessorScheduling/
programs vs Background services.

Go for programs.


Reducing for example the min timeslice that non interrupt tasks
get may free time for packet processing. You may of course
stop the router from working at all

sh proc cpu
CPU utilization for five seconds: 12%/6%; one minute: 11%; five
minutes: 9%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

130 16719024 80474 207762 0.00% 0.17% 0.19% 0 crypto
sw pk pro

The "207762" above is the average time in microseconds
that this process runs for on each
invocation. I suspect that this is the root cause of my
packet loss on this router since I suspect that it will
not be able to do anything else during this period.

It is however not affecting the users significantly so
we are living with it. (bit of web browsing).

TCP retransmissions are wonderful<g>.

 

I'm not seeing any throttles on the serial interface, just the ignored
and occassional drop at this point. I'm still tuning the buffers, I
have things working better but I'm still seeing an occassional small
burst of ignored and drop errors, sometimes on average only one burst
per 24 hours. I'm hoping this is just a buffer tuning/input queue
tuning issue.

A little background info on the interface that has me wonder sometimes
if it has something to do with cabling. When I first took over for
this router, the serial interface was spewing tons of input errors and
crc errors. I ended up swapping out the DS3 cable and the errors went
away but then I noticed input queue flushes. I started increasing the
hold-queue in value for the serial interface and the flushes went away
but then the input drops started coming through with the ignored
errors.

I'm going to continue tweakin the buffers and hold-queue and see where
I can get. I don't think this problem is having really any impact on
our operations but it would be nice to eliminate this annoyance.

Fun fun fun,

Spencer

 

There are few options to work around problem of CPU overload and
thereby resulting in input drops:

1. Using CEF.
2. Using SPD headroom tuning.

Would be worth checking these options too.
HTH

Guru Prashant

 

One thing I forgot.

You could consider applying access lists to
block unwanted traffic. This is a good _first_step.
Sorry I forgot it.


I am not sure what you might want on your serial
interface however block directed IP broadcasts at
least.

http://www.cisco.com/en/US/products/hw/routers/ps167/products_tech_note09186a0080094320.shtml

Says:
"As mentioned above, the only packets that are sent
to the input queue are the ones that are destined to
the router itself."

It does not seem to say this above, it does say:

"When a packet enters the router, the router
attempts to forward it at interrupt level. If a match
cannot be found in an appropriate cache table, the
packet is queued in the incoming interface's input
queue for processing."

Very confusing.

You might consider the following which may
identify the nature of the problem traffic.

Create an access list to deny known traffic
other than valid inbound traffic on the serial interface.
Disable - console, terminal, trap logging.
Enable - buffered logging
Create a large logging buffer.
Set up some monitoring of the input drops
so that the logging buffer can be captured when
the event occurs before the buffer wraps.

Enable "deb ip pac xxx"

Clearly this may well crash the router or maybe it won't.
You will have to investigate this yourself however the
steps above will I believe minimise the impact on the router.

Initially leave fast switching enabled. This should mean that
_only_ process switched traffic will be logged but this might
do the trick.

If you don't get to see the nature of the traffic you could then
try disabling fast switching.

Depends on how much you want to worry about it.

It does not sound as if you are experiencing user
visible problems. Interesting though.

 

Thanks for the input, I'm going to follow up on those suggestions. In
the meantime, since I left Friday, here is what I saw on my serial T3
interface in question:

Serial4/0 is up, line protocol is up
Hardware is M1T-T3 pa
MTU 4470 bytes, BW 44210 Kbit, DLY 200 usec,
reliability 255/255, txload 7/255, rxload 7/255
Encapsulation FRAME-RELAY IETF, crc 16, loopback not set
Keepalive set (8 sec)
Restart-Delay is 0 secs
LMI enq sent 28894, LMI stat recvd 28894, LMI upd recvd 0, DTE LMI
up
LMI enq recvd 0, LMI stat sent 0, LMI upd sent 0
LMI DLCI 0 LMI type is ANSI Annex D frame relay DTE
Broadcast queue 0/256, broadcasts sent/dropped 3852/0, interface
broadcasts 0
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 2d16h
Input queue: 0/210/49/0 (size/max/drops/flushes); Total output drops:
0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 1223000 bits/sec, 252 packets/sec
5 minute output rate 1319000 bits/sec, 258 packets/sec
114387931 packets input, 1071347993 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 parity
0 input errors, 0 CRC, 0 frame, 0 overrun, 201 ignored, 0 abort
119572070 packets output, 583407540 bytes, 0 underruns
0 output errors, 0 applique, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
rxLOS inactive, rxLOF inactive, rxAIS inactive
txAIS inactive, rxRAI inactive, txRAI inactive

I'm no longer seeing the no buffer errors but still a slight increment
in drops and ignored. I compared output from the "show buffers"
command from Friday and today and there were no failures/misses/trims
for the public buffer pools.

I log CPU utilization using MRTG/SiteScope and I haven't seen any
utilization above 35% for the week. I'm using CEF. I think I've
disabled fast switching before and it didn't work out very well, if I
recall I saw tons of errors.

You mentioned the ACL and that made me think of something. I took over
this router from the previous admin and I have always questioned the
ACL configuration. Basically everything is pretty much allowed through
the serial T3 interface THEN everything is blocked on the interface
servicing our network that is accessible by the public. We only allow
web/ftp type traffic to certain servers on that public network. Would
it be better to have this on the serial T3 interface?

Spencer

 

I moved the ACL from the inside interface servicing our public network
to the outside, serial T3 interface. I didn't see any drops or ignored
errors since making the change. I noticed that the previous admin also
allowed ICMP traffic to hit all our public servers. I switched that
off and only allowed echo-replies back in so people inside our network
can ping out. Hopefully the interface will continue to show the same
stats.

Spencer

 

Well done.

Prevention is better than cure.

As you have found out it is usually best to deny traffic as early as
possible.

I am not much in favour of denying all diagnostics, but it clearly
depends
on the circumstances. e.g. Cisco let you ping them.

I bumped into this today when looking for something else.

http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a0080094791.shtml

It has some interesting material that I have not seen before.

e.g.

show buffers input-interface serial 0/0

show buffers input-interface [interface type] [interface number] header


router#show interfaces ethernet 0/0 mac-accounting
Ethernet0/0
Input(494 free)
0000.0c5d.92f9(58 ): 1 packets, 106 bytes, last: 4038ms ago
0004.c059.c060(61 ): 0 packets, 0 bytes, last: 2493135ms ago
00b0.64bc.4860(64 ): 1 packets, 106 bytes, last: 20165ms ago
0090.f2c9.cc00(103): 12 packets, 720 bytes, last: 3117ms ago
Total: 14 packets, 932 bytes
Output (511 free)
0090.f2c9.cc00(103): 8 packets, 504 bytes, last: 4311ms ago
Total: 8 packets, 504 bytes


Could be handy one day.

 

Good document.

We found input/output queue drops in our Internet routers. It was
because the LAN Interface was receiving too much traffic to be handled
for the wan Interface (FE vs E3). We usually saw that when we had
infected PC in our network trying to spread the virus.

If after permit the echo in your net you saw a dicrease of the
drops, I could be that you have some infected machines sending echos to
the Internet. I would check which ip address are sending pings (or
echos) and how frequently.

-as

 

I can say with confidence that this problem is resolved. I haven't
seen any drops or ignored errors since making the changes. Thanks for
the info anybod. Aservin, fortunately we don't have any rogue machines
within the network. I'm logging echo-replies coming back in as I allow
pings (only out from within our network) and I don't see anything
strange. We were gettin hammered by all the people portscanning and
what not our class-c range, when echos to our public network servers
were allowed before.

Serial4/0 is up, line protocol is up
Hardware is M1T-T3 pa
MTU 4470 bytes, BW 44210 Kbit, DLY 200 usec,
reliability 255/255, txload 6/255, rxload 5/255
Encapsulation FRAME-RELAY IETF, crc 16, loopback not set
Keepalive set (8 sec)
Restart-Delay is 0 secs
LMI enq sent 19909, LMI stat recvd 19909, LMI upd recvd 0, DTE LMI
up
LMI enq recvd 0, LMI stat sent 0, LMI upd sent 0
LMI DLCI 0 LMI type is ANSI Annex D frame relay DTE
Broadcast queue 0/256, broadcasts sent/dropped 2654/0, interface
broadcasts 0
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 1d20h
Input queue: 0/210/0/0 (size/max/drops/flushes); Total output drops:
0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 928000 bits/sec, 197 packets/sec
5 minute output rate 1113000 bits/sec, 199 packets/sec
79925516 packets input, 2547962089 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 parity
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
85124305 packets output, 2471220331 bytes, 0 underruns
0 output errors, 0 applique, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
rxLOS inactive, rxLOF inactive, rxAIS inactive
txAIS inactive, rxRAI inactive, txRAI inactive

Aaahhhh...nice and clean.

Spencer