Dynamic NAT Failure

Discussion in 'Cisco' started by Yoann Roman, Aug 28, 2006.

  1. Yoann Roman

    Yoann Roman Guest

    I'm experiencing a strange NAT problem with a Cisco 2514 running 12.0(26).
    This router is configured with two NAT inside Ethernet interfaces for the
    LANs and one NAT outside serial interface for Internet. There are a few
    static NAT entries for servers and a pool of 1 IP address for dynamic NAT.

    About 1 to 3 times a month, no workstation using dynamic NAT on either
    Ethernet interface is able to get out to the web. When I do a "show ip nat
    trans", all I see listed are the static translations, no dynamic ones.
    Servers setup with static NAT go in and out fine. The only solution I've
    found is doing a "reload". Everything works after that.

    No config changes are made when these problems appear, and they seem to
    always occur first thing in the morning. The LAN has no more than 45
    simultaneous users. The NAT setup hasn't changed for over 2 to 3 years, and
    this only started happening in the past year.

    Any ideas?

    Thanks,

    --
    Yoann Roman
     
    Yoann Roman, Aug 28, 2006
    #1
    1. Advertising

  2. Yoann Roman

    NO_spamm Guest

    On Mon, 28 Aug 2006 15:54:13 +0000, Yoann Roman wrote:

    > I'm experiencing a strange NAT problem with a Cisco 2514 running 12.0(26).
    > This router is configured with two NAT inside Ethernet interfaces for the
    > LANs and one NAT outside serial interface for Internet. There are a few
    > static NAT entries for servers and a pool of 1 IP address for dynamic NAT.
    >
    > About 1 to 3 times a month, no workstation using dynamic NAT on either
    > Ethernet interface is able to get out to the web. When I do a "show ip nat
    > trans", all I see listed are the static translations, no dynamic ones.
    > Servers setup with static NAT go in and out fine. The only solution I've
    > found is doing a "reload". Everything works after that.
    >
    > No config changes are made when these problems appear, and they seem to
    > always occur first thing in the morning. The LAN has no more than 45
    > simultaneous users. The NAT setup hasn't changed for over 2 to 3 years, and
    > this only started happening in the past year.
    >
    > Any ideas?
    >
    > Thanks,


    It sounds like your pool of port numbers is emptied.
    Nothing changed to the router, but has the number of LAN user increased
    compared to two years ago?

    How are the ip nat translation time-out values set?


    FW
     
    NO_spamm, Aug 28, 2006
    #2
    1. Advertising

  3. Yoann Roman

    Yoann Roman Guest

    > On Mon, 28 Aug 2006 15:54:13 +0000, Yoann Roman wrote:
    >
    >> I'm experiencing a strange NAT problem with a Cisco 2514 running
    >> 12.0(26). This router is configured with two NAT inside Ethernet
    >> interfaces for the LANs and one NAT outside serial interface for
    >> Internet. There are a few static NAT entries for servers and a pool
    >> of 1 IP address for dynamic NAT.
    >>
    >> About 1 to 3 times a month, no workstation using dynamic NAT on
    >> either Ethernet interface is able to get out to the web. When I do a
    >> "show ip nat trans", all I see listed are the static translations,
    >> no dynamic ones. Servers setup with static NAT go in and out fine.
    >> The only solution I've found is doing a "reload". Everything works
    >> after that.
    >>
    >> No config changes are made when these problems appear, and they seem
    >> to always occur first thing in the morning. The LAN has no more than
    >> 45 simultaneous users. The NAT setup hasn't changed for over 2 to 3
    >> years, and this only started happening in the past year.
    >>
    >> Any ideas?
    >>
    >> Thanks,

    >
    > It sounds like your pool of port numbers is emptied.
    > Nothing changed to the router, but has the number of LAN user
    > increased compared to two years ago?
    >
    > How are the ip nat translation time-out values set?
    >
    >
    > FW


    The number of LAN users has probably increased from 30 to 45, at most, over
    the past 2 years. The timeout values are at their defaults, which I can
    lookup if needed.

    I have read about cases where the pool of port numbers is emptied or the NAT
    table is filled up when the timeout values are too great, but I would
    presume there should be at least a few dynamic NAT entries when doing a
    "show ip nat trans" if that were the case. Instead, I'm not seeing anything
    at all...

    Thanks,

    --
    Yoann Roman
     
    Yoann Roman, Aug 28, 2006
    #3
  4. Yoann Roman

    Yoann Roman Guest

    >> On Mon, 28 Aug 2006 15:54:13 +0000, Yoann Roman wrote:
    >>
    >>> [snip]

    >>
    >> It sounds like your pool of port numbers is emptied.
    >> Nothing changed to the router, but has the number of LAN user
    >> increased compared to two years ago?
    >>
    >> How are the ip nat translation time-out values set?
    >>
    >>
    >> FW

    >
    > The number of LAN users has probably increased from 30 to 45, at
    > most, over the past 2 years. The timeout values are at their
    > defaults, which I can lookup if needed.
    >
    > I have read about cases where the pool of port numbers is emptied or
    > the NAT table is filled up when the timeout values are too great, but
    > I would presume there should be at least a few dynamic NAT entries
    > when doing a "show ip nat trans" if that were the case. Instead, I'm
    > not seeing anything at all...


    Anyone other ideas on this? Thanks.

    --
    Yoann Roman
     
    Yoann Roman, Sep 1, 2006
    #4
  5. Yoann Roman

    Guest

    Yoann Roman wrote:
    > >> On Mon, 28 Aug 2006 15:54:13 +0000, Yoann Roman wrote:
    > >>
    > >>> [snip]
    > >>
    > >> It sounds like your pool of port numbers is emptied.
    > >> Nothing changed to the router, but has the number of LAN user
    > >> increased compared to two years ago?
    > >>
    > >> How are the ip nat translation time-out values set?
    > >>
    > >>
    > >> FW

    > >
    > > The number of LAN users has probably increased from 30 to 45, at
    > > most, over the past 2 years. The timeout values are at their
    > > defaults, which I can lookup if needed.
    > >
    > > I have read about cases where the pool of port numbers is emptied or
    > > the NAT table is filled up when the timeout values are too great, but
    > > I would presume there should be at least a few dynamic NAT entries
    > > when doing a "show ip nat trans" if that were the case. Instead, I'm
    > > not seeing anything at all...

    >
    > Anyone other ideas on this? Thanks.


    Well it looks like a bug or just maybe you are legitimately
    out of memory. i.e. by design.

    If you are lucky you may be able to do something about it.

    Please post
    sh ver
    sh mem ! First few lines.
    sh proc mem ! ? I forget exactly, the one that lists memory stats by
    process
    sh buff

    when you have a failure and after a reboot.
     
    , Sep 2, 2006
    #5
  6. Yoann Roman

    Yoann Roman Guest

    > Yoann Roman wrote:
    >>>> On Mon, 28 Aug 2006 15:54:13 +0000, Yoann Roman wrote:
    >>>>
    >>>>> [snip]
    >>>>
    >>>> It sounds like your pool of port numbers is emptied.
    >>>> Nothing changed to the router, but has the number of LAN user
    >>>> increased compared to two years ago?
    >>>>
    >>>> How are the ip nat translation time-out values set?
    >>>>
    >>>>
    >>>> FW
    >>>
    >>> The number of LAN users has probably increased from 30 to 45, at
    >>> most, over the past 2 years. The timeout values are at their
    >>> defaults, which I can lookup if needed.
    >>>
    >>> I have read about cases where the pool of port numbers is emptied or
    >>> the NAT table is filled up when the timeout values are too great,
    >>> but I would presume there should be at least a few dynamic NAT
    >>> entries when doing a "show ip nat trans" if that were the case.
    >>> Instead, I'm not seeing anything at all...

    >>
    >> Anyone other ideas on this? Thanks.

    >
    > Well it looks like a bug or just maybe you are legitimately
    > out of memory. i.e. by design.
    >
    > If you are lucky you may be able to do something about it.
    >
    > Please post
    > sh ver
    > sh mem ! First few lines.
    > sh proc mem ! ? I forget exactly, the one that lists memory stats by
    > process
    > sh buff
    >
    > when you have a failure and after a reboot.


    I'll make a note of these commands to run them the next time the failure
    happens. Given the rare occurence of this problem, this may not be for
    another month or so...

    Thanks for the help.

    --
    Yoann Roman
     
    Yoann Roman, Sep 6, 2006
    #6
  7. Yoann Roman

    Yoann Roman Guest

    >> Well it looks like a bug or just maybe you are legitimately
    >> out of memory. i.e. by design.
    >>
    >> If you are lucky you may be able to do something about it.
    >>
    >> Please post
    >> sh ver
    >> sh mem ! First few lines.
    >> sh proc mem ! ? I forget exactly, the one that lists memory stats by
    >> process
    >> sh buff
    >>
    >> when you have a failure and after a reboot.


    The failure occurred again this past Sunday. Exactly the same failure...
    Below is the output from the above commands. Let me know if this sheds any
    light on this problem.

    Router#sh ver
    Cisco Internetwork Operating System Software
    IOS (tm) 2500 Software (C2500-I-L), Version 12.0(26), RELEASE SOFTWARE (fc1)
    Copyright (c) 1986-2003 by cisco Systems, Inc.
    Compiled Mon 31-Mar-03 18:33 by srani
    Image text-base: 0x0302F634, data-base: 0x00001000

    ROM: System Bootstrap, Version 5.2(5), RELEASE SOFTWARE
    BOOTFLASH: 3000 Bootstrap Software (IGS-RXBOOT), Version 10.2(5), RELEASE
    SOFTWA
    RE (fc1)

    Router uptime is 2 weeks, 6 days, 9 hours, 20 minutes
    System restarted by reload at 09:17:42 EDT Mon Aug 28 2006
    System image file is "flash:c2500-i-l.120-26.bin"

    cisco 2500 (68030) processor (revision D) with 4096K/2048K bytes of memory.
    Processor board ID 01716848, with hardware revision 00000000
    Bridging software.
    X.25 software, Version 3.0.0.
    2 Ethernet/IEEE 802.3 interface(s)
    2 Serial network interface(s)
    32K bytes of non-volatile configuration memory.
    8192K bytes of processor board System flash (Read ONLY)

    Configuration register is 0x2102

    Router#sh mem
    Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
    Processor 7C4C0 3681088 1161484 2519604 2319620 2456252
    I/O 400000 2097152 392024 1705128 1632252 1699580

    Router#sh proc mem
    Total: 5778240, Used: 1553584, Free: 4224656
    PID TTY Allocated Freed Holding Getbufs Retbufs Process
    0 0 36308 1852 1292872 0 0 *Init*
    0 0 460 52380 460 0 0 *Sched*
    0 0 2443584 972616 12812 377700 0 *Dead*
    1 0 268 268 1748 0 0 Load Meter
    2 2 1268 0 6016 0 0 Virtual Exec
    3 0 0 0 4748 0 0 Check heaps
    4 0 829620 0 4844 621600 0 Pool Manager
    5 0 268 268 4748 0 0 Timers
    6 0 268 268 4748 0 0 Serial
    Backgroun
    7 0 16468 412612 7796 0 0 ARP Input
    8 0 268 268 4748 0 0 DDR Timers
    9 0 4700 1076 8372 0 0 Entity MIB
    API
    10 0 96 0 4844 0 0 SERIAL
    A'detect
    11 0 35311556 253432 25244 72720 0 IP Input
    13 0 244 0 4992 0 0 PPP IP Add
    Route
    14 0 272 0 5020 0 0 X.25 Encaps
    Mana
    15 0 0 7588 6748 0 0 TCP Timer
    16 0 49976 0 8256 0 0 TCP
    Protocols
    17 0 340 0 5088 0 0 Probe Input
    18 0 96 0 4844 0 0 RARP Input
    19 0 1943216 1941180 5984 0 0 BOOTP Server
    20 0 228 21988 5976 0 0 IP
    Background
    21 0 0 27862472 4748 0 0 IP Cache
    Ager
    22 0 244 0 4992 0 0 PAD InCall
    23 0 364 268 6844 0 0 X.25
    Background
    24 0 0 0 4748 0 0 Socket
    Timers
    25 0 0 0 4748 0 0 ISDN Timer
    27 0 0 0 4748 0 0 CallMIB
    Backgrou
    28 0 0 0 4748 0 0 ISDNMIB
    Backgrou
    29 0 96 0 6844 0 0 SNMP
    ConfCopyPro
    30 0 96 0 4844 0 0 Critical
    Bkgnd
    31 0 37312 22708 7108 0 0 Net
    Background
    32 0 448 268 6928 0 0 Logger
    33 0 268 420 4748 0 0 TTY
    Background
    34 0 0 172 5748 0 0 Per-Second
    Jobs
    35 0 192 0 4940 0 0 Net Input
    36 0 268 268 4748 0 0 Compute load
    avg
    37 0 6720 195256 4748 5040 715948 Per-minute
    Jobs
    38 0 1948 6674660 6344 0 0 IP NAT Ager
    39 0 0 0 4748 0 0 IP RACL Ager
    40 0 0 0 4748 0 0 SNMP Timers
    41 0 1368 268 7848 0 0 IP SNMP
    42 0 96 0 4844 0 0 SNMP Traps
    43 0 1512 268 5992 0 0 NTP
    1551664 Total

    Router#sh buff
    Buffer elements:
    499 in free list (500 max allowed)
    61398697 hits, 0 misses, 0 created

    Public buffer pools:
    Small buffers, 104 bytes (total 50, permanent 50):
    49 in free list (20 min, 150 max allowed)
    565056 hits, 0 misses, 0 trims, 0 created
    0 failures (0 no memory)
    Middle buffers, 600 bytes (total 25, permanent 25):
    23 in free list (10 min, 150 max allowed)
    75370 hits, 0 misses, 0 trims, 0 created
    0 failures (0 no memory)
    Big buffers, 1524 bytes (total 50, permanent 50):
    50 in free list (5 min, 150 max allowed)
    845807 hits, 1868 misses, 373 trims, 373 created
    584 failures (0 no memory)
    VeryBig buffers, 4520 bytes (total 10, permanent 10):
    10 in free list (0 min, 100 max allowed)
    0 hits, 0 misses, 0 trims, 0 created
    0 failures (0 no memory)
    Large buffers, 5024 bytes (total 0, permanent 0):
    0 in free list (0 min, 10 max allowed)
    0 hits, 0 misses, 0 trims, 0 created
    0 failures (0 no memory)
    Huge buffers, 18024 bytes (total 0, permanent 0):
    0 in free list (0 min, 4 max allowed)
    2 hits, 2 misses, 4 trims, 4 created
    0 failures (0 no memory)

    Interface buffer pools:
    Ethernet0 buffers, 1524 bytes (total 32, permanent 32):
    8 in free list (0 min, 32 max allowed)
    593197 hits, 825857 fallbacks
    8 max cache size, 5 in cache
    Ethernet1 buffers, 1524 bytes (total 32, permanent 32):
    8 in free list (0 min, 32 max allowed)
    1324 hits, 376 fallbacks
    8 max cache size, 8 in cache
    Serial0 buffers, 1524 bytes (total 32, permanent 32):
    7 in free list (0 min, 32 max allowed)
    24659 hits, 17223 fallbacks
    8 max cache size, 8 in cache
    Serial1 buffers, 1524 bytes (total 32, permanent 32):
    7 in free list (0 min, 32 max allowed)
    25 hits, 0 fallbacks
    8 max cache size, 8 in cache

    Thanks!

    --
    Yoann Roman
     
    Yoann Roman, Sep 18, 2006
    #7
  8. Yoann Roman

    Guest

    Yoann Roman wrote:
    > >> Well it looks like a bug or just maybe you are legitimately
    > >> out of memory. i.e. by design.
    > >>
    > >> If you are lucky you may be able to do something about it.
    > >>
    > >> Please post
    > >> sh ver
    > >> sh mem ! First few lines.
    > >> sh proc mem ! ? I forget exactly, the one that lists memory stats by
    > >> process
    > >> sh buff
    > >>
    > >> when you have a failure and after a reboot.


    > The failure occurred again this past Sunday. Exactly the same failure...
    > Below is the output from the above commands. Let me know if this sheds any
    > light on this problem.


    Comments inline.

    Warning - All of this is clutching at straws really but you
    may just fix it you never know.


    > Router#sh ver


    > Router uptime is 2 weeks, 6 days, 9 hours, 20 minutes


    > Router#sh mem
    > Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
    > Processor 7C4C0 3681088 1161484 2519604 2319620 2456252
    > I/O 400000 2097152 392024 1705128 1632252 1699580


    Memory OK.

    Largest ~= Free ~= Lowest (more or less)
    Memory not fragmented and you have never run out.


    > Router#sh buff
    > Buffer elements:
    > 499 in free list (500 max allowed)
    > 61398697 hits, 0 misses, 0 created
    >
    > Public buffer pools:
    > Small buffers, 104 bytes (total 50, permanent 50):
    > 49 in free list (20 min, 150 max allowed)
    > 565056 hits, 0 misses, 0 trims, 0 created
    > 0 failures (0 no memory)
    > Middle buffers, 600 bytes (total 25, permanent 25):
    > 23 in free list (10 min, 150 max allowed)
    > 75370 hits, 0 misses, 0 trims, 0 created
    > 0 failures (0 no memory)


    Zero misses is unusual but good!

    > Big buffers, 1524 bytes (total 50, permanent 50):
    > 50 in free list (5 min, 150 max allowed)
    > 845807 hits, 1868 misses, 373 trims, 373 created
    > 584 failures (0 no memory)


    This is really, really clutching at straws but you may be lucky.
    The idea of the following is to try to give the router
    the best opportunity to cope with what may be an overloaded
    condition.

    Failures we don't want. Lets try to get rid of them.
    There are several options here you could try for example.

    conf t
    buffers big min-free 20 ! 20 * 1524 = 30000 ish

    Above will use a bit more than 30k of RAM and you have enough.

    You will have to balance the memory that you have with the
    number of buffers that you allocate.


    > Huge buffers, 18024 bytes (total 0, permanent 0):
    > 0 in free list (0 min, 4 max allowed)
    > 2 hits, 2 misses, 4 trims, 4 created
    > 0 failures (0 no memory)


    > Interface buffer pools:
    > Ethernet0 buffers, 1524 bytes (total 32, permanent 32):
    > 8 in free list (0 min, 32 max allowed)
    > 593197 hits, 825857 fallbacks
    > 8 max cache size, 5 in cache


    Quite a lot of fallbacks, I think that these occur when the interface
    queues are full and the router allocates main memory for more
    queued packets. Something may be a bit busy.

    How is the CPU?

    I suggest that you could monitor these buffer failures to see if
    they occur regularly or maybe in a burst that could be swamping
    the router and resulting in the failure.

    A 2500 is a pretty marginal device in a modern LAN. All it
    would need is a few broadcasts and it would be filled up for a while.

    About the smallest routers that you can get today
    from Cisco that have not had end-of-life announced are the 850/870.
    they do 25000/10000 packets per second. A 2500 does 4400 pps.


    I have in the past applied access lists to try to protect routers
    from Windows broadcasts. Search the group for the thread
    "too many input drops in a 1721 router"
    "Queue Drops"
    "Input Drops With An Empty Input Queue"

    Did you have to reboot or did "clear ip nat tr *" fix it?

    Please also post sh int after failure.
    If you have another grab a show tech, sorry should have
    suggested that before.

    Sorry that I can't be of more precise assistance.
     
    , Sep 18, 2006
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Sharqy_5
    Replies:
    0
    Views:
    3,961
    Sharqy_5
    Jul 20, 2003
  2. c
    Replies:
    2
    Views:
    836
  3. Hans-Peter Walter
    Replies:
    3
    Views:
    1,173
    Joe Bloggs
    Jan 21, 2004
  4. yadap

    acl+Static nat+Dynamic Nat

    yadap, Aug 31, 2006, in forum: Cisco
    Replies:
    0
    Views:
    679
    yadap
    Aug 31, 2006
  5. Diego Balgera
    Replies:
    5
    Views:
    7,792
    Johann Lo
    Feb 8, 2008
Loading...

Share This Page