iptables NAT forwarding adding 75-100ms

Discussion in 'Linux Networking' started by Mike Lovell, Apr 29, 2012.

  1. Mike Lovell

    Mike Lovell Guest

    I have a strange occurrence of lag on my local networking. I have a
    cable modem that plugs into a Debian server, then that Debian server is
    plugged into a switch that all other machines in the house connect to.

    So something like:

    wan0 -> wan
    eth0 -> lan

    The relevant forwarding/NAT rules are:

    #iptables -A FORWARD -i wan0 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
    #iptables -A FORWARD -i eth0 -o wan0 -j ACCEPT

    #iptables -t nat -A POSTROUTING -o wan0 -j MASQUERADE

    So pretty standard boring NAT.

    Lag is occurring (between 75ms and 100ms) on all forwarding rules. Apart
    from the lag they function fine (no connectivity issues).

    Ping: LAN Machine -> Debian Router = ~0.7ms
    Ping: Debian Router -> Google = ~20ms
    Ping: LAN Machine -> Google = ~121ms !!!

    The Debian server has plenty of free RAM, the load is showing as low,
    it's (at this time) entirely dedicated to routing - Why is it
    introducing 100ms of lag into forwarded traffic???

    Anyone else seen similar to this???

    I get great speed from LAN machines, just high latency.

    ~ Mike
    Mike Lovell, Apr 29, 2012
    1. Advertisements

  2. Mike Lovell

    ein Guest

    How many FORWARD rules u have?
    Do above rules are in beginning of FORWARD chain? If no, please switch
    them as far of begin as u can. Is lag time changed?
    How do u check that?
    Wrong! Please 'ping' nearest machine after your router for example your
    ISP's gateway or ISP's DNS servers. Please have in mind that your ISP's
    router have more important things to do, than respond to ICMP echo
    request messages.
    What version of Debian is it?
    How much forward traffic u have?
    Do you have some QoS at this machine?
    ein, Apr 30, 2012
    1. Advertisements

  3. Mike Lovell

    ein Guest

    I meaned as close of the FORWARD begin as possible. :)
    ein, Apr 30, 2012
  4. Mike Lovell

    Chris Davies Guest

    Are you pinging by name or by address? If it's by name can you please
    repeat the tests with IP addresses. (This will eliminate any delay caused
    by slow DNS lookups.)

    Chris Davies, Apr 30, 2012
  5. The time= field in ping output measures the round trip of the ICMP
    packet; it doesn't include the DNS lookup.
    Richard Kettlewell, Apr 30, 2012
  6. Bad connection on an RJ-45 connector
    between eth0 and LAN switch?

    Also, do *not* rely on autosense in
    switchers for any crossover/patch cable
    detection. If you know it ought to be a
    crossover, use a crossover cable.

    Alastair Black, Apr 30, 2012
  7. Mike Lovell

    Mike Lovell Guest

    Just rules for the NAT, that's it:

    Chain FORWARD (policy DROP)
    target prot opt source destination
    ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
    ACCEPT all -- anywhere anywhere
    LOG all -- anywhere anywhere LOG level warning prefix "IPv4-forward "
    Interesting, much better:

    --- ping statistics ---
    10 packets transmitted, 9 received, 10% packet loss, time 9017ms
    rtt min/avg/max/mdev = 28.711/57.428/109.753/28.626 ms

    Even if I firewall all other traffic but a single test machine, I still
    get the latency problem there.

    So, based on the better ping to the next hop after my router, why (how)
    would something be distinguishing between the router carrying out the
    ping, and something behind the router?

    Can I mask it from doing so (if that's the problem)?


    ~ Mike
    Mike Lovell, Apr 30, 2012
  8. Mike Lovell

    Mike Lovell Guest

    But ping/connectivity *to* the router (eth0) from the LAN is fine. The
    problem only occurs when going through NAT/FORWARDING to something
    behind it.

    Although it appear (from a previously suggested test) that it *Doesn't*
    occur on the first hop after my router.

    So something strange is happening...
    Mike Lovell, Apr 30, 2012
  9. Mike Lovell

    Rick Jones Guest

    Is there a ping utility out there which puts a DNS lookup in the
    timing between the sending of the ICMP Echo Request and the receipt of
    the ICMP Echo Reply?

    Sure, DNS lookups could delay sending the first or subsequent ICMP
    Echo Requests, but any ping utility that sticks just about anything
    between the sending of the request and the receipt of the reply is
    doing something it shouldn't.

    rick jones
    Rick Jones, Apr 30, 2012
  10. Mike Lovell

    Chris Davies Guest

    Ooops, yes. I was thinking of the (really old) ping that (wrongly)
    used to do this, despite knowing that recent ping utilities do it
    right. I'm going to blame transient brain-fade on this one and hope
    no-one objects :-/

    Chris Davies, Apr 30, 2012
  11. Hello,

    Mike Lovell a écrit :
    What is wrong ?

    Indeed. So why do you suggest to ping the ISP's gateway, which is a
    router ? By the way, AFAICS, the OP did ping his own Debian router and
    a Google server, not any ISP's router.
    What is much better ?
    What is that address ?
    The average RTT is still ~60 ms.
    Better than what ?
    In the old days some ISPs have been known to try to detect multiple
    hosts behind NAT with variations of the TTL in packets.
    You can rewrite the TTL with iptables. Note that traceroute won't work
    any more.
    Pascal Hambourg, May 1, 2012
  12. Mike Lovell a écrit :
    Better ping a host than a router. As ein wrote, "routers have more
    important things to do, than respond to ICMP echo request messages".
    I wish I were...
    Note that the TTL of forwarded packets is decremented by 1 after the
    PREROUTING chains, so you shouldn't set the same TTL in OUTPUT if you
    want the same TTL when locally-generated and forwarded packets leave
    your router.

    Or you can set the TTL in POSTROUTING for both locally-generated and
    forwarded packets.
    Pascal Hambourg, May 1, 2012
  13. Mike Lovell a écrit :
    I guess so. Or just :

    iptables -t mangle -A POSTROUTING -o wan0 -j TTL --ttl-set 56
    Pascal Hambourg, May 1, 2012
  14. Mike Lovell

    Mike Lovell Guest

    Gah I definitely celebrated too soon.

    Must have been (possibly) related to rebooting the Debian router, it's
    returned to the same delay as before now (even with the TTL changes).

    There's something else fishy here.
    Mike Lovell, May 1, 2012
  15. Mike Lovell a écrit :
    Can you reproduce this ?
    Pascal Hambourg, May 1, 2012
  16. Ian Zimmerman a écrit :
    Because it is simple and effective.
    Not simpler, as this has to be done on all hosts instead of one router.
    Besides, it works only for Linux hosts, and some packets may not use the
    default TTL.
    Pascal Hambourg, May 1, 2012
  17. Mike Lovell

    ein Guest

    IMHO Mike's way to attempt to diagnose the problem.
    Correlation is simple: more hops = more things can change while
    connection or even while transmitting different packets to the same
    machine in this connection. Each ICMP echo request message can go over
    Internet with different path.

    In my country small ISPs are usesd to queue everything and decide for u
    what traffic is more important. ICMP "connections" are at the bottom of
    I still think the OP ISP is doing something nasty.

    Mike: can u connect to your (while observing the latency problem)
    router's WAN some other device for which u have access to?
    It will divide the problem to smaller parts. It will answer u that the
    problem is actually in forwarding at debian router.

    ein, May 2, 2012
  18. Mike Lovell

    ein Guest

    [email protected]:~# time ping -c1 google.ch
    PING google.ch ( 56(84) bytes of data.
    64 bytes from muc03s01-in-f24.1e100.net ( icmp_req=1
    ttl=49 time=48.5 ms

    --- google.ch ping statistics ---
    1 packets transmitted, 1 received, 0% packet loss, time 0ms
    rtt min/avg/max/mdev = 48.550/48.550/48.550/0.000 ms

    real 0m0.139s
    user 0m0.001s
    sys 0m0.001s
    [email protected]:~# time ping -c1
    PING ( 56(84) bytes of data.
    64 bytes from icmp_req=1 ttl=49 time=47.1 ms

    --- ping statistics ---
    1 packets transmitted, 1 received, 0% packet loss, time 0ms
    rtt min/avg/max/mdev = 47.155/47.155/47.155/0.000 ms

    real 0m0.048s
    user 0m0.000s
    sys 0m0.001s

    ein, May 2, 2012
  19. Mike Lovell

    Rick Jones Guest

    That is the overall difference in how long the ping command takes to
    run, but there is no difference in the RTTs of the actual ICMP Echo
    Request/Reply pairs. IE, there is a DNS lookup before the ICMP Echo
    Request is sent, and one after the ICMP Echo Reply is received, but
    they are outside the timestamping inside ping to measure the RTTs.

    Roughly speaking:

    DNS lookup name
    Get starting timestamp
    Send ICMP Echo Request
    ....time passes...
    Receive ICMP Echo Reply
    Get ending timestamp
    DNS lookup source IP of ICMP Echo Reply

    So, DNS delays can cause even a well-written ping utility to take
    longer to execute overall, but they will not cause the RTTs reported
    by that utility to be longer.

    rick jones
    Rick Jones, May 2, 2012
  20. Comcast Cable Communications, Inc. JUMPSTART-2 (NET-68-80-0-0-1) -
    Comcast Cable Communications, Inc. COMCAST-16 (NET-68-86-64-0-1) -

    As told by `whois` :)

    To the OP: It's always a good idea to ping an address you
    a) know
    b) know the distance to
    c) maybe use `traceroute -n` instead

    As the ghoul for instance has many IP addresses for the same
    domai name, so the DNS is not the problem but just by pinging
    e.g. google.com you're probably pinging a dozen different IP's...

    No offense, just my 0.02 Drachmen :p

    Ralph Spitzner, May 4, 2012
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.