TCP connections slow to 1 packet, every 120 seconds

Have an unusual condition and trying to find a mechanism that would
cause this.

Have a Internet location where download wilsl randomly slow until near
complete stop. Not everyone, by a too high percentage. All major
sites show issue... Microsoft, McAFee, CNet, Youtube, anything.

We've done Wireshark captures. There are no retransmissions, no
corrupt packets, no loss. The delay between packets will
logarithmically increase until it quickly reaches 120 seconds. From
the client's perspective

Received packet from Microsoft.
ACK sent immediately
EXACTLY 120 seconds
Received packet from Microsoft.
ACK sent immediately
EXACTLY 120 seconds
Received packet from Microsoft.
ACK sent immediately


The delay incremented in one test as so
rapdi exchange of packet for the first two seconds, then
@2.1 seconds
@2.3 seconds
@2.5 seconds
@2.9 seconds
@3.1 seconds
@3.9 seconds
@4.2 seconds
@5.9 seconds
@9.4 seconds
@16.4 seconds
@30.3 seconds
@57.8 seconds
@112.5 seconds
@221.7 seconds
@342.0 seconds
@462.4 seconds
@582.6 seconds
... every 120 seconds for next half hour--then capture closed

120 seconnds is clearly a timer of some sort, but I can't figure out
what would cause a service to hold off sending the next packet for 120
seconds... Or what device would buffer such a thing for 120 secs.


Again, no re-transmissions. Each packet is a new packet, different.
the transfer continues at a rate of one 1,514 byte packet every 120
seconds. This is happening to servers, desktops & laptops across a
site.

It only occurs with TCP transfers. We have UDP connections between
the sites that are unaffected. Not affecting local LAN-to-LAN. SO the
problem is either on our Cisco 2821 (WAN/Internet router) or within
our ISP. ISP has been doing good faith tests and investigation
without results.

Just fishing for ideas of what would hold back the next packet for 120
seconds, yet not be re-transmissions, corrupt packets, or loss of
packets.

 

A TCP TARPIT.

 

Firstly are the sequence numbers incrementing
with each new received packet? If not then it is simply
that your acks are not getting through.

This seems by far the most likely.

Below is some other stuff I had already written
which may be worth a look if the above does not explain the behaviour.

What TCP options are being used? Look in the SYN
packets in wireshark and post options from both sides
here. What are *your* receive windows? i.e. what
receive windows are being offered by you in acks
to the far hosts?

The sending TCP should do something like send one packet
extra for every ack received. For some reason this is not
happening.

Also set up a couple of instances of say pingplotter
(30 day free trial) and ping a couple or three big fancy
popular web sites
with full size packets every second for ever. Look
for anomalies. May as well do small packets too.

Could ALSO consider TCP with pingplotter too. It
does close the session properly so will not leave
1,000's of dangling connections sucking the life out
of your target.

My first idea is that your ISP or your firewall is likely
doing something that is affecting the service
and this is driving the far end TCPs into a state from
which it is difficult or impossible for them to recover.

You have rebooted your firewall?
Recent firwall software upgrades/config changes.

I forget the definition now but I will look it
up, Slow Start is a TCP state This of course
won't help you since it is whatever is getting them into
this behaviour that is the real problem.

It could of course be a problem on your site too.

Let's have some more details of your site and internet
connection. Whatever you think might be relevant.
Also - do you do any QoS, rate limiting, traffic
shaping or anything like that?

The email in the header works, I don't normally use it
but if you like send a wireshark file and I can have a look
to see if I can spot it in all the spam