Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Computing > Cisco > TCP 3 Way Handshake

Reply
Thread Tools

TCP 3 Way Handshake

 
 
Darren Green
Guest
Posts: n/a
 
      02-09-2008
I have an instance where a remote ADSL site (PPPOA) wants to talk to a
server in a Co-Lo but the 3 x way handshake doesn't fully complete. PC
at remote site is Linux as is the Server in the Co-Lo.

When the traffic traverses the ADSL it comes into the clients main
office and jumps across a couple of Ethernet circuits to the Co-Lo.

I've taken off the remote acl and firewall from the site router. The
Hosts gets all the way to Established but the server stays in the
SYN_REC state indicating that:

The originating PC sent the ACK to the server. The server sent the SYN
Ack back to to client. The client moves to Established but the last ACK
never hists the server to open up the communication.

I don't think it's a fragmentation issue. We can ping with DF bit set to
1500 bytes (1472 + headers) between source and destination (haven't
tested the other way though). There are no other filtering devices in
the way. The path (traceroute) from site to site isn't exactly the same.
Could out of sequence traffic somehow be the issue, or maybe
fragmentation could be the issue on the way back as I haven't tested the
return path.

We have tried a HTTP request, FTP and SSH and none work, so it isn't an
issue restricted to HTTP.

Any helpful debugs - I have seen 'debug tcp transactions' to test the 3
x way handshake. I know this will help but the Co-Lo router crunches a
lot of data and I am concerned that the debug may have a negative
effect. Anyone got any other good debugs or traps they can suggest so
that I can determine where the 3 x way communication is breaking down.

Regards

Darren
 
Reply With Quote
 
 
 
 
Bod43@hotmail.co.uk
Guest
Posts: n/a
 
      02-09-2008
On 9 Feb, 09:15, Darren Green <(E-Mail Removed)> wrote:
> I have an instance where a remote ADSL site (PPPOA) wants to talk to a
> server in a Co-Lo but the 3 x way handshake doesn't fully complete. PC
> at remote site is Linux as is the Server in the Co-Lo.
>
> When the traffic traverses the ADSL it comes into the clients main
> office and jumps across a couple of Ethernet circuits to the Co-Lo.
>
> I've taken off the remote acl and firewall from the site router. The
> Hosts gets all the way to Established but the server stays in the
> SYN_REC state indicating that:
>
> The originating PC sent the ACK to the server. The server sent the SYN
> Ack back to to client. The client moves to Established but the last ACK
> never hists the server to open up the communication.
>
> I don't think it's a fragmentation issue. We can ping with DF bit set to
> * 1500 bytes (1472 + headers) between source and destination (haven't
> tested the other way though). There are no other filtering devices in
> the way. The path (traceroute) from site to site isn't exactly the same.
> Could out of sequence traffic somehow be the issue, or maybe
> fragmentation could be the issue on the way back as I haven't tested the
> return path.
>
> We have tried a HTTP request, FTP and SSH and none work, so it isn't an
> issue restricted to HTTP.
>
> Any helpful debugs - I have seen 'debug tcp transactions' to test the 3
> x way handshake. I know this will help but the Co-Lo router crunches a
> lot of data and I am concerned that the debug may have a negative
> effect. Anyone got any other good debugs or traps they can suggest so
> that I can determine where the 3 x way communication is breaking down.

 
Reply With Quote
 
 
 
 
Bod43@hotmail.co.uk
Guest
Posts: n/a
 
      02-09-2008
On 9 Feb, 09:15, Darren Green <(E-Mail Removed)> wrote:
> I have an instance where a remote ADSL site (PPPOA) wants to talk to a
> server in a Co-Lo but the 3 x way handshake doesn't fully complete. PC
> at remote site is Linux as is the Server in the Co-Lo.
>
> When the traffic traverses the ADSL it comes into the clients main
> office and jumps across a couple of Ethernet circuits to the Co-Lo.
>
> I've taken off the remote acl and firewall from the site router. The
> Hosts gets all the way to Established but the server stays in the
> SYN_REC state indicating that:
>
> The originating PC sent the ACK to the server. The server sent the SYN
> Ack back to to client. The client moves to Established but the last ACK
> never hists the server to open up the communication.
>
> I don't think it's a fragmentation issue. We can ping with DF bit set to
> * 1500 bytes (1472 + headers) between source and destination (haven't
> tested the other way though). There are no other filtering devices in
> the way. The path (traceroute) from site to site isn't exactly the same.
> Could out of sequence traffic somehow be the issue, or maybe
> fragmentation could be the issue on the way back as I haven't tested the
> return path.
>
> We have tried a HTTP request, FTP and SSH and none work, so it isn't an
> issue restricted to HTTP.
>
> Any helpful debugs - I have seen 'debug tcp transactions' to test the 3
> x way handshake. I know this will help but the Co-Lo router crunches a
> lot of data and I am concerned that the debug may have a negative
> effect. Anyone got any other good debugs or traps they can suggest so
> that I can determine where the 3 x way communication is breaking down.

On 9 Feb, 09:15, Darren Green <(E-Mail Removed)> wrote:
> I have an instance where a remote ADSL site (PPPOA) wants to talk to a
> server in a Co-Lo but the 3 x way handshake doesn't fully complete. PC
> at remote site is Linux as is the Server in the Co-Lo.


> When the traffic traverses the ADSL it comes into the clients main
> office and jumps across a couple of Ethernet circuits to the Co-Lo.
>
> I've taken off the remote acl and firewall from the site router. The
> Hosts gets all the way to Established but the server stays in the
> SYN_REC state indicating that:
>
> The originating PC sent the ACK to the server. The server sent the SYN
> Ack back to to client. The client moves to Established but the last ACK
> never hists the server to open up the communication.
>
> I don't think it's a fragmentation issue. We can ping with DF bit set to
> 1500 bytes (1472 + headers) between source and destination (haven't
> tested the other way though). There are no other filtering devices in
> the way. The path (traceroute) from site to site isn't exactly the same.
> Could out of sequence traffic somehow be the issue, or maybe
> fragmentation could be the issue on the way back as I haven't tested the
> return path.
>
> We have tried a HTTP request, FTP and SSH and none work, so it isn't an
> issue restricted to HTTP.
>
> Any helpful debugs - I have seen 'debug tcp transactions' to test the 3
> x way handshake. I know this will help but the Co-Lo router crunches a
> lot of data and I am concerned that the debug may have a negative
> effect. Anyone got any other good debugs or traps they can suggest so
> that I can determine where the 3 x way communication is breaking down.


You seem to have thought about this a decent bit so
I have too. Here is a brain splurge.

I presume this is over the internet with no VPN.

It's not a fragmentation or packet order issue
since the initial packets are very small and are
sent one at a time.

By the way:
The DF bit test it not a guarantee that you are not
getting fragmentation since some equipment can be configured
to ignore the bit. You need to check the actual packets
with say tcpdump. The test is probably valid though and
as stated this cannot be a fragmentation issue.

You are saying that ping between the hosts in question works
but TCP does not.

Here are some thinks to consider
- Some kind of firewall on the client. ##
- Can the client make ANY tcp conenction?
To itself or other hosts?
- That you do not have a duplicate IP address
run sniffer and look for ARPs.
- RUn tcpdump on the Client to check for
received unreachables.
- Worry about NAT - don't know what exactly but
check it out.
- Perhaps you have two internet connections and
load balancing and two NATters?
- Bad client - use an indepdent sniffer to check
for the "lost" packets.
- A "deny tcp src dst established" ACL
on the path from server to client
would block this exact traffic.
- run a lot of pings to check that
link loss rate is not too high.


Be aware that some debugs do not work with
non-process-switched traffic.

"deb ip packet" is one of them.

"no ip route-cache" on interfaces turns of fast swithing
but router capacity is reduced by about x 10 (factor
of 10) on non hardwre platforms (not 4500/6500) and
I would guess MORE on hardware based platforms if indeed
it did turn off hardware switching.

Good debug practise if you are worried about CPU
no logg con ! should just be off anyway really.
no logg mon
no logg trap
logg buffered
logg buffered 100000 ! or whatever value you like best or have
memory for.

STILL worry about CPU

On a busy or not so busy router some debugs will
kill it - DEAD.

One reasonably CPU friendly way of looking for packets
is to create an ACL that PERMITS the traffic that you
want to look for and checking the counters on the ACLs.


ip access-l 100 permit tcp 1.1.1.1 2.2.2.2 established
ip access-l 100 permit tcp 1.1.1.1 2.2.2.2
ip access-l 100 permit ip any any

ip access-group .....

sh ip access-l 100

ACL lets ALL tarffic through but the first line lets you
see if packets match it.

I think this should be reliable on non hardware
switched platforms (not 4500 6500)


Good luck.

 
Reply With Quote
 
Darren Green
Guest
Posts: n/a
 
      02-09-2008
http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:

Hi Bod,

Thanks for this, you have given me a few things here to try / consider,
really appreciate the help.

> You seem to have thought about this a decent bit so
> I have too. Here is a brain splurge.
>
> I presume this is over the internet with no VPN.


No, this is private ADSL into an MPLS core. When the traffic reaches the
Central site (the HQ) there are Ethernet links onwards to the Co-Lo. The
Co-LO has a path out to the Internet when clients require breakout.
>
> It's not a fragmentation or packet order issue
> since the initial packets are very small and are
> sent one at a time.
>
> By the way:
> The DF bit test it not a guarantee that you are not
> getting fragmentation since some equipment can be configured
> to ignore the bit. You need to check the actual packets
> with say tcpdump. The test is probably valid though and
> as stated this cannot be a fragmentation issue.


Good point. I have before had to configure routers to reset the df-bit,
old fragmentation problem for another client. Not used tcpdump before I
will take a look but I think this is similar to Ethereal which I am
familiar with.

>
> You are saying that ping between the hosts in question works
> but TCP does not.
>

Yes.

> Here are some thinks to consider
> - Some kind of firewall on the client. ##


I asked the Sys Admin at the customer whether IPTables on the Linux
machine(s) we blocking connection. He says not.
BTW a connection from his Central Office over Ethernet WAN circuits to
the Co-Lo works fine.

> - Can the client make ANY tcp conenction?
> To itself or other hosts?
> - That you do not have a duplicate IP address


Don't think so, there are only a handful of hosts in the Co-Lo. I asked
him to check connectivity from other machines at site to a number of
Co-Lo servers, various ports, nothing worked. I also asked him to try
from another site, various ports and that never worked either.

> run sniffer and look for ARPs.
> - RUn tcpdump on the Client to check for
> received unreachables.
> - Worry about NAT - don't know what exactly but
> check it out.


Hmmmm, the router where the server sits does some NATing for some of the
Co-Lo hosts. The server concerned though is not subject to NAT. I'll
double check may facts but I am sure that there is no translation for
this device.

> - Perhaps you have two internet connections and
> load balancing and two NATters?
> - Bad client - use an indepdent sniffer to check
> for the "lost" packets.
> - A "deny tcp src dst established" ACL
> on the path from server to client
> would block this exact traffic.


Nice one, I'll do that.

> - run a lot of pings to check that
> link loss rate is not too high.
>
>
> Be aware that some debugs do not work with
> non-process-switched traffic.
>
> "deb ip packet" is one of them.
>

I am glad you mentioned that, I had forgot.

> "no ip route-cache" on interfaces turns of fast swithing
> but router capacity is reduced by about x 10 (factor
> of 10) on non hardwre platforms (not 4500/6500) and
> I would guess MORE on hardware based platforms if indeed
> it did turn off hardware switching.
>
> Good debug practise if you are worried about CPU
> no logg con ! should just be off anyway really.
> no logg mon
> no logg trap
> logg buffered
> logg buffered 100000 ! or whatever value you like best or have
> memory for.
>
> STILL worry about CPU
>
> On a busy or not so busy router some debugs will
> kill it - DEAD.
>
> One reasonably CPU friendly way of looking for packets
> is to create an ACL that PERMITS the traffic that you
> want to look for and checking the counters on the ACLs.
>
>
> ip access-l 100 permit tcp 1.1.1.1 2.2.2.2 established
> ip access-l 100 permit tcp 1.1.1.1 2.2.2.2
> ip access-l 100 permit ip any any
>
> ip access-group .....
>
> sh ip access-l 100
>
> ACL lets ALL tarffic through but the first line lets you
> see if packets match it.


What's weird is that when I had previously enabled ip inspect and and
outside ACL on the remote router and saw that I was receiving hits on
this ACL. NB I had the entry in there as the client was trying to telnet
on port 80 and ip inspect was dropping the session (ip inspect had
entries for tcp, udp and icmp only not HTTP). I therefore reckoned
application inspection was dropping the traffic as it didn't recongise
this as true HTTP traffic hence my outside ACL entry to allow it in.

Point being that I saw the counter increment but no data would flow.
Client then tried a wget and same result.
>
> I think this should be reliable on non hardware
> switched platforms (not 4500 6500)
>
>
> Good luck.
>


Again thanks. I will see what the ACL's / debugs turn up.

Regards

Darren
 
Reply With Quote
 
Thrill5
Guest
Posts: n/a
 
      02-11-2008

"Darren Green" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed)...
>I have an instance where a remote ADSL site (PPPOA) wants to talk to a
>server in a Co-Lo but the 3 x way handshake doesn't fully complete. PC at
>remote site is Linux as is the Server in the Co-Lo.
>
> When the traffic traverses the ADSL it comes into the clients main office
> and jumps across a couple of Ethernet circuits to the Co-Lo.
>
> I've taken off the remote acl and firewall from the site router. The Hosts
> gets all the way to Established but the server stays in the SYN_REC state
> indicating that:
>
> The originating PC sent the ACK to the server. The server sent the SYN Ack
> back to to client. The client moves to Established but the last ACK never
> hists the server to open up the communication.
>
> I don't think it's a fragmentation issue. We can ping with DF bit set to
> 1500 bytes (1472 + headers) between source and destination (haven't tested
> the other way though). There are no other filtering devices in the way.
> The path (traceroute) from site to site isn't exactly the same. Could out
> of sequence traffic somehow be the issue, or maybe fragmentation could be
> the issue on the way back as I haven't tested the return path.
>
> We have tried a HTTP request, FTP and SSH and none work, so it isn't an
> issue restricted to HTTP.
>
> Any helpful debugs - I have seen 'debug tcp transactions' to test the 3 x
> way handshake. I know this will help but the Co-Lo router crunches a lot
> of data and I am concerned that the debug may have a negative effect.
> Anyone got any other good debugs or traps they can suggest so that I can
> determine where the 3 x way communication is breaking down.
>
> Regards
>
> Darren


You have a firewall or ACL issue. Fragmentation is not an issue because all
of the three-way handshake packets are only 40 bytes.
You NEVER run packet debugs on production routers or switches, that is what
sniffers are for. Do a traceroute and check the configs on every single
device to verify that ACL's are not your problem, then check them again, and
then a third time. Sometimes you can't see the forest because of the trees,
so yes check them three times. Check the logs on any firewall that also may
be in the path.for packets to the destination. If they don't have logging
of dropped packets, have them check the ruleset. If all else fails, setup
captures at points along the path to track down where the packets are being
dropped. Tedious, but probably the only way to find the offending device.





 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
PIX configuration -- 3 way handshake incomplete brickwalls19 Cisco 2 07-30-2009 02:53 PM
Raw socket three way handshake Marco C Programming 5 06-04-2006 11:12 PM
4-Way Handshake Alexandr Mishagin Wireless Networking 1 05-10-2005 05:40 PM
Windows XP SP2 Supplicant ==> 4-Way Handshake in IBSS Alexandr Mishagin Wireless Networking 3 04-28-2005 06:01 AM
Two Way Handshake Verification wrytat ASP .Net Security 0 04-14-2005 12:53 AM



Advertisments