«

I'm experiencing a strange NAT problem with a Cisco 2514 running 12.0(26).
This router is configured with two NAT inside Ethernet interfaces for the
LANs and one NAT outside serial interface for Internet. There are a few
static NAT entries for servers and a pool of 1 IP address for dynamic NAT.

About 1 to 3 times a month, no workstation using dynamic NAT on either
Ethernet interface is able to get out to the web. When I do a "show ip nat
trans", all I see listed are the static translations, no dynamic ones.
Servers setup with static NAT go in and out fine. The only solution I've
found is doing a "reload". Everything works after that.

No config changes are made when these problems appear, and they seem to
always occur first thing in the morning. The LAN has no more than 45
simultaneous users. The NAT setup hasn't changed for over 2 to 3 years, and
this only started happening in the past year.

Any ideas?

Thanks,

--
Yoann Roman

On Mon, 28 Aug 2006 15:54:13 +0000, Yoann Roman wrote:

> I'm experiencing a strange NAT problem with a Cisco 2514 running 12.0(26).
> This router is configured with two NAT inside Ethernet interfaces for the
> LANs and one NAT outside serial interface for Internet. There are a few
> static NAT entries for servers and a pool of 1 IP address for dynamic NAT.
>
> About 1 to 3 times a month, no workstation using dynamic NAT on either
> Ethernet interface is able to get out to the web. When I do a "show ip nat
> trans", all I see listed are the static translations, no dynamic ones.
> Servers setup with static NAT go in and out fine. The only solution I've
> found is doing a "reload". Everything works after that.
>
> No config changes are made when these problems appear, and they seem to
> always occur first thing in the morning. The LAN has no more than 45
> simultaneous users. The NAT setup hasn't changed for over 2 to 3 years, and
> this only started happening in the past year.
>
> Any ideas?
>
> Thanks,

It sounds like your pool of port numbers is emptied.
Nothing changed to the router, but has the number of LAN user increased
compared to two years ago?

How are the ip nat translation time-out values set?


FW

> On Mon, 28 Aug 2006 15:54:13 +0000, Yoann Roman wrote:
>
>> I'm experiencing a strange NAT problem with a Cisco 2514 running
>> 12.0(26). This router is configured with two NAT inside Ethernet
>> interfaces for the LANs and one NAT outside serial interface for
>> Internet. There are a few static NAT entries for servers and a pool
>> of 1 IP address for dynamic NAT.
>>
>> About 1 to 3 times a month, no workstation using dynamic NAT on
>> either Ethernet interface is able to get out to the web. When I do a
>> "show ip nat trans", all I see listed are the static translations,
>> no dynamic ones. Servers setup with static NAT go in and out fine.
>> The only solution I've found is doing a "reload". Everything works
>> after that.
>>
>> No config changes are made when these problems appear, and they seem
>> to always occur first thing in the morning. The LAN has no more than
>> 45 simultaneous users. The NAT setup hasn't changed for over 2 to 3
>> years, and this only started happening in the past year.
>>
>> Any ideas?
>>
>> Thanks,
>
> It sounds like your pool of port numbers is emptied.
> Nothing changed to the router, but has the number of LAN user
> increased compared to two years ago?
>
> How are the ip nat translation time-out values set?
>
>
> FW

The number of LAN users has probably increased from 30 to 45, at most, over
the past 2 years. The timeout values are at their defaults, which I can
lookup if needed.

I have read about cases where the pool of port numbers is emptied or the NAT
table is filled up when the timeout values are too great, but I would
presume there should be at least a few dynamic NAT entries when doing a
"show ip nat trans" if that were the case. Instead, I'm not seeing anything
at all...

Thanks,

--
Yoann Roman

>> On Mon, 28 Aug 2006 15:54:13 +0000, Yoann Roman wrote:
>>
>>> [snip]
>>
>> It sounds like your pool of port numbers is emptied.
>> Nothing changed to the router, but has the number of LAN user
>> increased compared to two years ago?
>>
>> How are the ip nat translation time-out values set?
>>
>>
>> FW
>
> The number of LAN users has probably increased from 30 to 45, at
> most, over the past 2 years. The timeout values are at their
> defaults, which I can lookup if needed.
>
> I have read about cases where the pool of port numbers is emptied or
> the NAT table is filled up when the timeout values are too great, but
> I would presume there should be at least a few dynamic NAT entries
> when doing a "show ip nat trans" if that were the case. Instead, I'm
> not seeing anything at all...

Anyone other ideas on this? Thanks.

--
Yoann Roman

Yoann Roman wrote:
> >> On Mon, 28 Aug 2006 15:54:13 +0000, Yoann Roman wrote:
> >>
> >>> [snip]
> >>
> >> It sounds like your pool of port numbers is emptied.
> >> Nothing changed to the router, but has the number of LAN user
> >> increased compared to two years ago?
> >>
> >> How are the ip nat translation time-out values set?
> >>
> >>
> >> FW
> >
> > The number of LAN users has probably increased from 30 to 45, at
> > most, over the past 2 years. The timeout values are at their
> > defaults, which I can lookup if needed.
> >
> > I have read about cases where the pool of port numbers is emptied or
> > the NAT table is filled up when the timeout values are too great, but
> > I would presume there should be at least a few dynamic NAT entries
> > when doing a "show ip nat trans" if that were the case. Instead, I'm
> > not seeing anything at all...
>
> Anyone other ideas on this? Thanks.

Well it looks like a bug or just maybe you are legitimately
out of memory. i.e. by design.

If you are lucky you may be able to do something about it.

Please post
sh ver
sh mem ! First few lines.
sh proc mem ! ? I forget exactly, the one that lists memory stats by
process
sh buff

when you have a failure and after a reboot.

> Yoann Roman wrote:
>>>> On Mon, 28 Aug 2006 15:54:13 +0000, Yoann Roman wrote:
>>>>
>>>>> [snip]
>>>>
>>>> It sounds like your pool of port numbers is emptied.
>>>> Nothing changed to the router, but has the number of LAN user
>>>> increased compared to two years ago?
>>>>
>>>> How are the ip nat translation time-out values set?
>>>>
>>>>
>>>> FW
>>>
>>> The number of LAN users has probably increased from 30 to 45, at
>>> most, over the past 2 years. The timeout values are at their
>>> defaults, which I can lookup if needed.
>>>
>>> I have read about cases where the pool of port numbers is emptied or
>>> the NAT table is filled up when the timeout values are too great,
>>> but I would presume there should be at least a few dynamic NAT
>>> entries when doing a "show ip nat trans" if that were the case.
>>> Instead, I'm not seeing anything at all...
>>
>> Anyone other ideas on this? Thanks.
>
> Well it looks like a bug or just maybe you are legitimately
> out of memory. i.e. by design.
>
> If you are lucky you may be able to do something about it.
>
> Please post
> sh ver
> sh mem ! First few lines.
> sh proc mem ! ? I forget exactly, the one that lists memory stats by
> process
> sh buff
>
> when you have a failure and after a reboot.

I'll make a note of these commands to run them the next time the failure
happens. Given the rare occurence of this problem, this may not be for
another month or so...

Thanks for the help.

--
Yoann Roman

>> Well it looks like a bug or just maybe you are legitimately
>> out of memory. i.e. by design.
>>
>> If you are lucky you may be able to do something about it.
>>
>> Please post
>> sh ver
>> sh mem ! First few lines.
>> sh proc mem ! ? I forget exactly, the one that lists memory stats by
>> process
>> sh buff
>>
>> when you have a failure and after a reboot.

The failure occurred again this past Sunday. Exactly the same failure...
Below is the output from the above commands. Let me know if this sheds any
light on this problem.

Router#sh ver
Cisco Internetwork Operating System Software
IOS (tm) 2500 Software (C2500-I-L), Version 12.0(26), RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2003 by cisco Systems, Inc.
Compiled Mon 31-Mar-03 18:33 by srani
Image text-base: 0x0302F634, data-base: 0x00001000

ROM: System Bootstrap, Version 5.2(5), RELEASE SOFTWARE
BOOTFLASH: 3000 Bootstrap Software (IGS-RXBOOT), Version 10.2(5), RELEASE
SOFTWA
RE (fc1)

Router uptime is 2 weeks, 6 days, 9 hours, 20 minutes
System restarted by reload at 09:17:42 EDT Mon Aug 28 2006
System image file is "flash:c2500-i-l.120-26.bin"

cisco 2500 (68030) processor (revision D) with 4096K/2048K bytes of memory.
Processor board ID 01716848, with hardware revision 00000000
Bridging software.
X.25 software, Version 3.0.0.
2 Ethernet/IEEE 802.3 interface(s)
2 Serial network interface(s)
32K bytes of non-volatile configuration memory.
8192K bytes of processor board System flash (Read ONLY)

Configuration register is 0x2102

Router#sh mem
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 7C4C0 3681088 1161484 2519604 2319620 2456252
I/O 400000 2097152 392024 1705128 1632252 1699580

Router#sh proc mem
Total: 5778240, Used: 1553584, Free: 4224656
PID TTY Allocated Freed Holding Getbufs Retbufs Process
0 0 36308 1852 1292872 0 0 *Init*
0 0 460 52380 460 0 0 *Sched*
0 0 2443584 972616 12812 377700 0 *Dead*
1 0 268 268 1748 0 0 Load Meter
2 2 1268 0 6016 0 0 Virtual Exec
3 0 0 0 4748 0 0 Check heaps
4 0 829620 0 4844 621600 0 Pool Manager
5 0 268 268 4748 0 0 Timers
6 0 268 268 4748 0 0 Serial
Backgroun
7 0 16468 412612 7796 0 0 ARP Input
8 0 268 268 4748 0 0 DDR Timers
9 0 4700 1076 8372 0 0 Entity MIB
API
10 0 96 0 4844 0 0 SERIAL
A'detect
11 0 35311556 253432 25244 72720 0 IP Input
13 0 244 0 4992 0 0 PPP IP Add
Route
14 0 272 0 5020 0 0 X.25 Encaps
Mana
15 0 0 7588 6748 0 0 TCP Timer
16 0 49976 0 8256 0 0 TCP
Protocols
17 0 340 0 5088 0 0 Probe Input
18 0 96 0 4844 0 0 RARP Input
19 0 1943216 1941180 5984 0 0 BOOTP Server
20 0 228 21988 5976 0 0 IP
Background
21 0 0 27862472 4748 0 0 IP Cache
Ager
22 0 244 0 4992 0 0 PAD InCall
23 0 364 268 6844 0 0 X.25
Background
24 0 0 0 4748 0 0 Socket
Timers
25 0 0 0 4748 0 0 ISDN Timer
27 0 0 0 4748 0 0 CallMIB
Backgrou
28 0 0 0 4748 0 0 ISDNMIB
Backgrou
29 0 96 0 6844 0 0 SNMP
ConfCopyPro
30 0 96 0 4844 0 0 Critical
Bkgnd
31 0 37312 22708 7108 0 0 Net
Background
32 0 448 268 6928 0 0 Logger
33 0 268 420 4748 0 0 TTY
Background
34 0 0 172 5748 0 0 Per-Second
Jobs
35 0 192 0 4940 0 0 Net Input
36 0 268 268 4748 0 0 Compute load
avg
37 0 6720 195256 4748 5040 715948 Per-minute
Jobs
38 0 1948 6674660 6344 0 0 IP NAT Ager
39 0 0 0 4748 0 0 IP RACL Ager
40 0 0 0 4748 0 0 SNMP Timers
41 0 1368 268 7848 0 0 IP SNMP
42 0 96 0 4844 0 0 SNMP Traps
43 0 1512 268 5992 0 0 NTP
1551664 Total

Router#sh buff
Buffer elements:
499 in free list (500 max allowed)
61398697 hits, 0 misses, 0 created

Public buffer pools:
Small buffers, 104 bytes (total 50, permanent 50):
49 in free list (20 min, 150 max allowed)
565056 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Middle buffers, 600 bytes (total 25, permanent 25):
23 in free list (10 min, 150 max allowed)
75370 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Big buffers, 1524 bytes (total 50, permanent 50):
50 in free list (5 min, 150 max allowed)
845807 hits, 1868 misses, 373 trims, 373 created
584 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 10, permanent 10):
10 in free list (0 min, 100 max allowed)
0 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Large buffers, 5024 bytes (total 0, permanent 0):
0 in free list (0 min, 10 max allowed)
0 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Huge buffers, 18024 bytes (total 0, permanent 0):
0 in free list (0 min, 4 max allowed)
2 hits, 2 misses, 4 trims, 4 created
0 failures (0 no memory)

Interface buffer pools:
Ethernet0 buffers, 1524 bytes (total 32, permanent 32):
8 in free list (0 min, 32 max allowed)
593197 hits, 825857 fallbacks
8 max cache size, 5 in cache
Ethernet1 buffers, 1524 bytes (total 32, permanent 32):
8 in free list (0 min, 32 max allowed)
1324 hits, 376 fallbacks
8 max cache size, 8 in cache
Serial0 buffers, 1524 bytes (total 32, permanent 32):
7 in free list (0 min, 32 max allowed)
24659 hits, 17223 fallbacks
8 max cache size, 8 in cache
Serial1 buffers, 1524 bytes (total 32, permanent 32):
7 in free list (0 min, 32 max allowed)
25 hits, 0 fallbacks
8 max cache size, 8 in cache

Thanks!

--
Yoann Roman

Yoann Roman wrote:
> >> Well it looks like a bug or just maybe you are legitimately
> >> out of memory. i.e. by design.
> >>
> >> If you are lucky you may be able to do something about it.
> >>
> >> Please post
> >> sh ver
> >> sh mem ! First few lines.
> >> sh proc mem ! ? I forget exactly, the one that lists memory stats by
> >> process
> >> sh buff
> >>
> >> when you have a failure and after a reboot.

> The failure occurred again this past Sunday. Exactly the same failure...
> Below is the output from the above commands. Let me know if this sheds any
> light on this problem.

Comments inline.

Warning - All of this is clutching at straws really but you
may just fix it you never know.


> Router#sh ver

> Router uptime is 2 weeks, 6 days, 9 hours, 20 minutes

> Router#sh mem
> Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
> Processor 7C4C0 3681088 1161484 2519604 2319620 2456252
> I/O 400000 2097152 392024 1705128 1632252 1699580

Memory OK.

Largest ~= Free ~= Lowest (more or less)
Memory not fragmented and you have never run out.


> Router#sh buff
> Buffer elements:
> 499 in free list (500 max allowed)
> 61398697 hits, 0 misses, 0 created
>
> Public buffer pools:
> Small buffers, 104 bytes (total 50, permanent 50):
> 49 in free list (20 min, 150 max allowed)
> 565056 hits, 0 misses, 0 trims, 0 created
> 0 failures (0 no memory)
> Middle buffers, 600 bytes (total 25, permanent 25):
> 23 in free list (10 min, 150 max allowed)
> 75370 hits, 0 misses, 0 trims, 0 created
> 0 failures (0 no memory)

Zero misses is unusual but good!

> Big buffers, 1524 bytes (total 50, permanent 50):
> 50 in free list (5 min, 150 max allowed)
> 845807 hits, 1868 misses, 373 trims, 373 created
> 584 failures (0 no memory)

This is really, really clutching at straws but you may be lucky.
The idea of the following is to try to give the router
the best opportunity to cope with what may be an overloaded
condition.

Failures we don't want. Lets try to get rid of them.
There are several options here you could try for example.

conf t
buffers big min-free 20 ! 20 * 1524 = 30000 ish

Above will use a bit more than 30k of RAM and you have enough.

You will have to balance the memory that you have with the
number of buffers that you allocate.


> Huge buffers, 18024 bytes (total 0, permanent 0):
> 0 in free list (0 min, 4 max allowed)
> 2 hits, 2 misses, 4 trims, 4 created
> 0 failures (0 no memory)

> Interface buffer pools:
> Ethernet0 buffers, 1524 bytes (total 32, permanent 32):
> 8 in free list (0 min, 32 max allowed)
> 593197 hits, 825857 fallbacks
> 8 max cache size, 5 in cache

Quite a lot of fallbacks, I think that these occur when the interface
queues are full and the router allocates main memory for more
queued packets. Something may be a bit busy.

How is the CPU?

I suggest that you could monitor these buffer failures to see if
they occur regularly or maybe in a burst that could be swamping
the router and resulting in the failure.

A 2500 is a pretty marginal device in a modern LAN. All it
would need is a few broadcasts and it would be filled up for a while.

About the smallest routers that you can get today
from Cisco that have not had end-of-life announced are the 850/870.
they do 25000/10000 packets per second. A 2500 does 4400 pps.


I have in the past applied access lists to try to protect routers
from Windows broadcasts. Search the group for the thread
"too many input drops in a 1721 router"
"Queue Drops"
"Input Drops With An Empty Input Queue"

Did you have to reboot or did "clear ip nat tr *" fix it?

Please also post sh int after failure.
If you have another grab a show tech, sorry should have
suggested that before.

Sorry that I can't be of more precise assistance.