TCP 3 Way Handshake

Discussion in 'Cisco' started by Darren Green, Feb 9, 2008.

  1. Darren Green

    Darren Green Guest

    I have an instance where a remote ADSL site (PPPOA) wants to talk to a
    server in a Co-Lo but the 3 x way handshake doesn't fully complete. PC
    at remote site is Linux as is the Server in the Co-Lo.

    When the traffic traverses the ADSL it comes into the clients main
    office and jumps across a couple of Ethernet circuits to the Co-Lo.

    I've taken off the remote acl and firewall from the site router. The
    Hosts gets all the way to Established but the server stays in the
    SYN_REC state indicating that:

    The originating PC sent the ACK to the server. The server sent the SYN
    Ack back to to client. The client moves to Established but the last ACK
    never hists the server to open up the communication.

    I don't think it's a fragmentation issue. We can ping with DF bit set to
    1500 bytes (1472 + headers) between source and destination (haven't
    tested the other way though). There are no other filtering devices in
    the way. The path (traceroute) from site to site isn't exactly the same.
    Could out of sequence traffic somehow be the issue, or maybe
    fragmentation could be the issue on the way back as I haven't tested the
    return path.

    We have tried a HTTP request, FTP and SSH and none work, so it isn't an
    issue restricted to HTTP.

    Any helpful debugs - I have seen 'debug tcp transactions' to test the 3
    x way handshake. I know this will help but the Co-Lo router crunches a
    lot of data and I am concerned that the debug may have a negative
    effect. Anyone got any other good debugs or traps they can suggest so
    that I can determine where the 3 x way communication is breaking down.

    Regards

    Darren
     
    Darren Green, Feb 9, 2008
    #1
    1. Advertising

  2. Darren Green

    Guest

    On 9 Feb, 09:15, Darren Green <> wrote:
    > I have an instance where a remote ADSL site (PPPOA) wants to talk to a
    > server in a Co-Lo but the 3 x way handshake doesn't fully complete. PC
    > at remote site is Linux as is the Server in the Co-Lo.
    >
    > When the traffic traverses the ADSL it comes into the clients main
    > office and jumps across a couple of Ethernet circuits to the Co-Lo.
    >
    > I've taken off the remote acl and firewall from the site router. The
    > Hosts gets all the way to Established but the server stays in the
    > SYN_REC state indicating that:
    >
    > The originating PC sent the ACK to the server. The server sent the SYN
    > Ack back to to client. The client moves to Established but the last ACK
    > never hists the server to open up the communication.
    >
    > I don't think it's a fragmentation issue. We can ping with DF bit set to
    >   1500 bytes (1472 + headers) between source and destination (haven't
    > tested the other way though). There are no other filtering devices in
    > the way. The path (traceroute) from site to site isn't exactly the same.
    > Could out of sequence traffic somehow be the issue, or maybe
    > fragmentation could be the issue on the way back as I haven't tested the
    > return path.
    >
    > We have tried a HTTP request, FTP and SSH and none work, so it isn't an
    > issue restricted to HTTP.
    >
    > Any helpful debugs - I have seen 'debug tcp transactions' to test the 3
    > x way handshake. I know this will help but the Co-Lo router crunches a
    > lot of data and I am concerned that the debug may have a negative
    > effect. Anyone got any other good debugs or traps they can suggest so
    > that I can determine where the 3 x way communication is breaking down.
     
    , Feb 9, 2008
    #2
    1. Advertising

  3. Darren Green

    Guest

    On 9 Feb, 09:15, Darren Green <> wrote:
    > I have an instance where a remote ADSL site (PPPOA) wants to talk to a
    > server in a Co-Lo but the 3 x way handshake doesn't fully complete. PC
    > at remote site is Linux as is the Server in the Co-Lo.
    >
    > When the traffic traverses the ADSL it comes into the clients main
    > office and jumps across a couple of Ethernet circuits to the Co-Lo.
    >
    > I've taken off the remote acl and firewall from the site router. The
    > Hosts gets all the way to Established but the server stays in the
    > SYN_REC state indicating that:
    >
    > The originating PC sent the ACK to the server. The server sent the SYN
    > Ack back to to client. The client moves to Established but the last ACK
    > never hists the server to open up the communication.
    >
    > I don't think it's a fragmentation issue. We can ping with DF bit set to
    >   1500 bytes (1472 + headers) between source and destination (haven't
    > tested the other way though). There are no other filtering devices in
    > the way. The path (traceroute) from site to site isn't exactly the same.
    > Could out of sequence traffic somehow be the issue, or maybe
    > fragmentation could be the issue on the way back as I haven't tested the
    > return path.
    >
    > We have tried a HTTP request, FTP and SSH and none work, so it isn't an
    > issue restricted to HTTP.
    >
    > Any helpful debugs - I have seen 'debug tcp transactions' to test the 3
    > x way handshake. I know this will help but the Co-Lo router crunches a
    > lot of data and I am concerned that the debug may have a negative
    > effect. Anyone got any other good debugs or traps they can suggest so
    > that I can determine where the 3 x way communication is breaking down.

    On 9 Feb, 09:15, Darren Green <> wrote:
    > I have an instance where a remote ADSL site (PPPOA) wants to talk to a
    > server in a Co-Lo but the 3 x way handshake doesn't fully complete. PC
    > at remote site is Linux as is the Server in the Co-Lo.


    > When the traffic traverses the ADSL it comes into the clients main
    > office and jumps across a couple of Ethernet circuits to the Co-Lo.
    >
    > I've taken off the remote acl and firewall from the site router. The
    > Hosts gets all the way to Established but the server stays in the
    > SYN_REC state indicating that:
    >
    > The originating PC sent the ACK to the server. The server sent the SYN
    > Ack back to to client. The client moves to Established but the last ACK
    > never hists the server to open up the communication.
    >
    > I don't think it's a fragmentation issue. We can ping with DF bit set to
    > 1500 bytes (1472 + headers) between source and destination (haven't
    > tested the other way though). There are no other filtering devices in
    > the way. The path (traceroute) from site to site isn't exactly the same.
    > Could out of sequence traffic somehow be the issue, or maybe
    > fragmentation could be the issue on the way back as I haven't tested the
    > return path.
    >
    > We have tried a HTTP request, FTP and SSH and none work, so it isn't an
    > issue restricted to HTTP.
    >
    > Any helpful debugs - I have seen 'debug tcp transactions' to test the 3
    > x way handshake. I know this will help but the Co-Lo router crunches a
    > lot of data and I am concerned that the debug may have a negative
    > effect. Anyone got any other good debugs or traps they can suggest so
    > that I can determine where the 3 x way communication is breaking down.


    You seem to have thought about this a decent bit so
    I have too. Here is a brain splurge.

    I presume this is over the internet with no VPN.

    It's not a fragmentation or packet order issue
    since the initial packets are very small and are
    sent one at a time.

    By the way:
    The DF bit test it not a guarantee that you are not
    getting fragmentation since some equipment can be configured
    to ignore the bit. You need to check the actual packets
    with say tcpdump. The test is probably valid though and
    as stated this cannot be a fragmentation issue.

    You are saying that ping between the hosts in question works
    but TCP does not.

    Here are some thinks to consider
    - Some kind of firewall on the client. ##
    - Can the client make ANY tcp conenction?
    To itself or other hosts?
    - That you do not have a duplicate IP address
    run sniffer and look for ARPs.
    - RUn tcpdump on the Client to check for
    received unreachables.
    - Worry about NAT - don't know what exactly but
    check it out.
    - Perhaps you have two internet connections and
    load balancing and two NATters?
    - Bad client - use an indepdent sniffer to check
    for the "lost" packets.
    - A "deny tcp src dst established" ACL
    on the path from server to client
    would block this exact traffic.
    - run a lot of pings to check that
    link loss rate is not too high.


    Be aware that some debugs do not work with
    non-process-switched traffic.

    "deb ip packet" is one of them.

    "no ip route-cache" on interfaces turns of fast swithing
    but router capacity is reduced by about x 10 (factor
    of 10) on non hardwre platforms (not 4500/6500) and
    I would guess MORE on hardware based platforms if indeed
    it did turn off hardware switching.

    Good debug practise if you are worried about CPU
    no logg con ! should just be off anyway really.
    no logg mon
    no logg trap
    logg buffered
    logg buffered 100000 ! or whatever value you like best or have
    memory for.

    STILL worry about CPU

    On a busy or not so busy router some debugs will
    kill it - DEAD.

    One reasonably CPU friendly way of looking for packets
    is to create an ACL that PERMITS the traffic that you
    want to look for and checking the counters on the ACLs.


    ip access-l 100 permit tcp 1.1.1.1 2.2.2.2 established
    ip access-l 100 permit tcp 1.1.1.1 2.2.2.2
    ip access-l 100 permit ip any any

    ip access-group .....

    sh ip access-l 100

    ACL lets ALL tarffic through but the first line lets you
    see if packets match it.

    I think this should be reliable on non hardware
    switched platforms (not 4500 6500)


    Good luck.
     
    , Feb 9, 2008
    #3
  4. Darren Green

    Darren Green Guest

    wrote:

    Hi Bod,

    Thanks for this, you have given me a few things here to try / consider,
    really appreciate the help.

    > You seem to have thought about this a decent bit so
    > I have too. Here is a brain splurge.
    >
    > I presume this is over the internet with no VPN.


    No, this is private ADSL into an MPLS core. When the traffic reaches the
    Central site (the HQ) there are Ethernet links onwards to the Co-Lo. The
    Co-LO has a path out to the Internet when clients require breakout.
    >
    > It's not a fragmentation or packet order issue
    > since the initial packets are very small and are
    > sent one at a time.
    >
    > By the way:
    > The DF bit test it not a guarantee that you are not
    > getting fragmentation since some equipment can be configured
    > to ignore the bit. You need to check the actual packets
    > with say tcpdump. The test is probably valid though and
    > as stated this cannot be a fragmentation issue.


    Good point. I have before had to configure routers to reset the df-bit,
    old fragmentation problem for another client. Not used tcpdump before I
    will take a look but I think this is similar to Ethereal which I am
    familiar with.

    >
    > You are saying that ping between the hosts in question works
    > but TCP does not.
    >

    Yes.

    > Here are some thinks to consider
    > - Some kind of firewall on the client. ##


    I asked the Sys Admin at the customer whether IPTables on the Linux
    machine(s) we blocking connection. He says not.
    BTW a connection from his Central Office over Ethernet WAN circuits to
    the Co-Lo works fine.

    > - Can the client make ANY tcp conenction?
    > To itself or other hosts?
    > - That you do not have a duplicate IP address


    Don't think so, there are only a handful of hosts in the Co-Lo. I asked
    him to check connectivity from other machines at site to a number of
    Co-Lo servers, various ports, nothing worked. I also asked him to try
    from another site, various ports and that never worked either.

    > run sniffer and look for ARPs.
    > - RUn tcpdump on the Client to check for
    > received unreachables.
    > - Worry about NAT - don't know what exactly but
    > check it out.


    Hmmmm, the router where the server sits does some NATing for some of the
    Co-Lo hosts. The server concerned though is not subject to NAT. I'll
    double check may facts but I am sure that there is no translation for
    this device.

    > - Perhaps you have two internet connections and
    > load balancing and two NATters?
    > - Bad client - use an indepdent sniffer to check
    > for the "lost" packets.
    > - A "deny tcp src dst established" ACL
    > on the path from server to client
    > would block this exact traffic.


    Nice one, I'll do that.

    > - run a lot of pings to check that
    > link loss rate is not too high.
    >
    >
    > Be aware that some debugs do not work with
    > non-process-switched traffic.
    >
    > "deb ip packet" is one of them.
    >

    I am glad you mentioned that, I had forgot.

    > "no ip route-cache" on interfaces turns of fast swithing
    > but router capacity is reduced by about x 10 (factor
    > of 10) on non hardwre platforms (not 4500/6500) and
    > I would guess MORE on hardware based platforms if indeed
    > it did turn off hardware switching.
    >
    > Good debug practise if you are worried about CPU
    > no logg con ! should just be off anyway really.
    > no logg mon
    > no logg trap
    > logg buffered
    > logg buffered 100000 ! or whatever value you like best or have
    > memory for.
    >
    > STILL worry about CPU
    >
    > On a busy or not so busy router some debugs will
    > kill it - DEAD.
    >
    > One reasonably CPU friendly way of looking for packets
    > is to create an ACL that PERMITS the traffic that you
    > want to look for and checking the counters on the ACLs.
    >
    >
    > ip access-l 100 permit tcp 1.1.1.1 2.2.2.2 established
    > ip access-l 100 permit tcp 1.1.1.1 2.2.2.2
    > ip access-l 100 permit ip any any
    >
    > ip access-group .....
    >
    > sh ip access-l 100
    >
    > ACL lets ALL tarffic through but the first line lets you
    > see if packets match it.


    What's weird is that when I had previously enabled ip inspect and and
    outside ACL on the remote router and saw that I was receiving hits on
    this ACL. NB I had the entry in there as the client was trying to telnet
    on port 80 and ip inspect was dropping the session (ip inspect had
    entries for tcp, udp and icmp only not HTTP). I therefore reckoned
    application inspection was dropping the traffic as it didn't recongise
    this as true HTTP traffic hence my outside ACL entry to allow it in.

    Point being that I saw the counter increment but no data would flow.
    Client then tried a wget and same result.
    >
    > I think this should be reliable on non hardware
    > switched platforms (not 4500 6500)
    >
    >
    > Good luck.
    >


    Again thanks. I will see what the ACL's / debugs turn up.

    Regards

    Darren
     
    Darren Green, Feb 9, 2008
    #4
  5. Darren Green

    Thrill5 Guest

    "Darren Green" <> wrote in message
    news:...
    >I have an instance where a remote ADSL site (PPPOA) wants to talk to a
    >server in a Co-Lo but the 3 x way handshake doesn't fully complete. PC at
    >remote site is Linux as is the Server in the Co-Lo.
    >
    > When the traffic traverses the ADSL it comes into the clients main office
    > and jumps across a couple of Ethernet circuits to the Co-Lo.
    >
    > I've taken off the remote acl and firewall from the site router. The Hosts
    > gets all the way to Established but the server stays in the SYN_REC state
    > indicating that:
    >
    > The originating PC sent the ACK to the server. The server sent the SYN Ack
    > back to to client. The client moves to Established but the last ACK never
    > hists the server to open up the communication.
    >
    > I don't think it's a fragmentation issue. We can ping with DF bit set to
    > 1500 bytes (1472 + headers) between source and destination (haven't tested
    > the other way though). There are no other filtering devices in the way.
    > The path (traceroute) from site to site isn't exactly the same. Could out
    > of sequence traffic somehow be the issue, or maybe fragmentation could be
    > the issue on the way back as I haven't tested the return path.
    >
    > We have tried a HTTP request, FTP and SSH and none work, so it isn't an
    > issue restricted to HTTP.
    >
    > Any helpful debugs - I have seen 'debug tcp transactions' to test the 3 x
    > way handshake. I know this will help but the Co-Lo router crunches a lot
    > of data and I am concerned that the debug may have a negative effect.
    > Anyone got any other good debugs or traps they can suggest so that I can
    > determine where the 3 x way communication is breaking down.
    >
    > Regards
    >
    > Darren


    You have a firewall or ACL issue. Fragmentation is not an issue because all
    of the three-way handshake packets are only 40 bytes.
    You NEVER run packet debugs on production routers or switches, that is what
    sniffers are for. Do a traceroute and check the configs on every single
    device to verify that ACL's are not your problem, then check them again, and
    then a third time. Sometimes you can't see the forest because of the trees,
    so yes check them three times. Check the logs on any firewall that also may
    be in the path.for packets to the destination. If they don't have logging
    of dropped packets, have them check the ruleset. If all else fails, setup
    captures at points along the path to track down where the packets are being
    dropped. Tedious, but probably the only way to find the offending device.
     
    Thrill5, Feb 11, 2008
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. neelaka

    WPA-PSK handshake

    neelaka, Jan 23, 2005, in forum: Wireless Networking
    Replies:
    1
    Views:
    9,737
    Niklas
    Jan 27, 2005
  2. Alexandr Mishagin

    Windows XP SP2 Supplicant ==> 4-Way Handshake in IBSS

    Alexandr Mishagin, Apr 27, 2005, in forum: Wireless Networking
    Replies:
    3
    Views:
    3,376
    Niklas
    Apr 28, 2005
  3. Alexandr Mishagin

    4-Way Handshake

    Alexandr Mishagin, May 5, 2005, in forum: Wireless Networking
    Replies:
    1
    Views:
    1,295
    Anusha Dandapani[MSFT]
    May 10, 2005
  4. Bay Area Dave
    Replies:
    20
    Views:
    707
    Ray Fischer
    Sep 19, 2003
  5. brickwalls19
    Replies:
    2
    Views:
    913
    brickwalls19
    Jul 30, 2009
Loading...

Share This Page