TCP connections slow to 1 packet, every 120 seconds

Discussion in 'Cisco' started by Royal Frazier, Oct 12, 2010.

  1. Have an unusual condition and trying to find a mechanism that would
    cause this.

    Have a Internet location where download wilsl randomly slow until near
    complete stop. Not everyone, by a too high percentage. All major
    sites show issue... Microsoft, McAFee, CNet, Youtube, anything.

    We've done Wireshark captures. There are no retransmissions, no
    corrupt packets, no loss. The delay between packets will
    logarithmically increase until it quickly reaches 120 seconds. From
    the client's perspective

    Received packet from Microsoft.
    ACK sent immediately
    EXACTLY 120 seconds
    Received packet from Microsoft.
    ACK sent immediately
    EXACTLY 120 seconds
    Received packet from Microsoft.
    ACK sent immediately


    The delay incremented in one test as so
    rapdi exchange of packet for the first two seconds, then
    @2.1 seconds
    @2.3 seconds
    @2.5 seconds
    @2.9 seconds
    @3.1 seconds
    @3.9 seconds
    @4.2 seconds
    @5.9 seconds
    @9.4 seconds
    @16.4 seconds
    @30.3 seconds
    @57.8 seconds
    @112.5 seconds
    @221.7 seconds
    @342.0 seconds
    @462.4 seconds
    @582.6 seconds
    ... every 120 seconds for next half hour--then capture closed

    120 seconnds is clearly a timer of some sort, but I can't figure out
    what would cause a service to hold off sending the next packet for 120
    seconds... Or what device would buffer such a thing for 120 secs.


    Again, no re-transmissions. Each packet is a new packet, different.
    the transfer continues at a rate of one 1,514 byte packet every 120
    seconds. This is happening to servers, desktops & laptops across a
    site.

    It only occurs with TCP transfers. We have UDP connections between
    the sites that are unaffected. Not affecting local LAN-to-LAN. SO the
    problem is either on our Cisco 2821 (WAN/Internet router) or within
    our ISP. ISP has been doing good faith tests and investigation
    without results.

    Just fishing for ideas of what would hold back the next packet for 120
    seconds, yet not be re-transmissions, corrupt packets, or loss of
    packets.
     
    Royal Frazier, Oct 12, 2010
    #1
    1. Advertisements

  2. Royal Frazier

    D. Stussy Guest

    A TCP TARPIT.
     
    D. Stussy, Oct 12, 2010
    #2
    1. Advertisements

  3. Royal Frazier

    bod43 Guest

    Firstly are the sequence numbers incrementing
    with each new received packet? If not then it is simply
    that your acks are not getting through.

    This seems by far the most likely.

    Below is some other stuff I had already written
    which may be worth a look if the above does not explain the behaviour.

    What TCP options are being used? Look in the SYN
    packets in wireshark and post options from both sides
    here. What are *your* receive windows? i.e. what
    receive windows are being offered by you in acks
    to the far hosts?

    The sending TCP should do something like send one packet
    extra for every ack received. For some reason this is not
    happening.

    Also set up a couple of instances of say pingplotter
    (30 day free trial) and ping a couple or three big fancy
    popular web sites
    with full size packets every second for ever. Look
    for anomalies. May as well do small packets too.

    Could ALSO consider TCP with pingplotter too. It
    does close the session properly so will not leave
    1,000's of dangling connections sucking the life out
    of your target.

    My first idea is that your ISP or your firewall is likely
    doing something that is affecting the service
    and this is driving the far end TCPs into a state from
    which it is difficult or impossible for them to recover.

    You have rebooted your firewall?
    Recent firwall software upgrades/config changes.

    I forget the definition now but I will look it
    up, Slow Start is a TCP state:) This of course
    won't help you since it is whatever is getting them into
    this behaviour that is the real problem.

    It could of course be a problem on your site too.

    Let's have some more details of your site and internet
    connection. Whatever you think might be relevant.
    Also - do you do any QoS, rate limiting, traffic
    shaping or anything like that?

    The email in the header works, I don't normally use it
    but if you like send a wireshark file and I can have a look
    to see if I can spot it in all the spam:)
     
    bod43, Oct 13, 2010
    #3
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.