PIX 501 - SYN Timeout Duration 0:02:01

Discussion in 'Cisco' started by Rik Bain, Oct 20, 2003.

  1. Rik Bain

    Rik Bain Guest

    On Mon, 20 Oct 2003 22:59:10 +0600, Lars Purschke wrote:

    > Hi!
    >
    > We have two PIX 501 6.2.2 site to site connected via VPN with the main
    > PIX 6.3.1. Several times a day all connections are closed by a SYN
    > Timeout at the same time and always with a duration of 0:02:01.
    >
    > %PIX-6-302014: Teardown TCP connection 5873 for outside:xxx to
    > inside:yyy duration 0:02:01 bytes 0 SYN Timeout
    >
    > The IPSec SA's are still valid and the TCP connection timeout is set to
    > 9:00:00. I've checked the Internet connection but it's working all the
    > time.
    >
    > timeout xlate 5:00:00
    > timeout conn 9:00:00 half-closed 0:10:00 udp 2:30:00 rpc 2:30:00 h225
    > 1:00:00
    > timeout h323 0:05:00 mgcp 0:05:00 sip 0:30:00 sip_media 0:02:00 timeout
    > uauth 0:05:00 absolute
    >
    > How can this happen? Does the PIX cancel the connections or could it be
    > the AS/400 Server? What can I do to avoid this?
    >
    > Thanks
    > Lars



    Check for assymetric routing. ICMP can still "work" under said
    conditions.

    Pix also had a problem (cant remember which code, but it surfaced on 6.1)
    with tcp connections when window size approaching 65K, so might want to
    check that too. Under this condition, the pix would pass the SYN-ACK
    back, but silently (i.e. syslog shows nothing) drop the ACK. Syslog
    subsequently report SYN timeout.
     
    Rik Bain, Oct 20, 2003
    #1
    1. Advertising

  2. Hi!

    We have two PIX 501 6.2.2 site to site connected via VPN with the main
    PIX 6.3.1. Several times a day all connections are closed by a SYN
    Timeout at the same time and always with a duration of 0:02:01.

    %PIX-6-302014: Teardown TCP connection 5873 for outside:xxx to
    inside:yyy duration 0:02:01 bytes 0 SYN Timeout

    The IPSec SA's are still valid and the TCP connection timeout is set to
    9:00:00. I've checked the Internet connection but it's working all the time.

    timeout xlate 5:00:00
    timeout conn 9:00:00 half-closed 0:10:00 udp 2:30:00 rpc 2:30:00 h225
    1:00:00
    timeout h323 0:05:00 mgcp 0:05:00 sip 0:30:00 sip_media 0:02:00
    timeout uauth 0:05:00 absolute

    How can this happen? Does the PIX cancel the connections or could it be
    the AS/400 Server? What can I do to avoid this?

    Thanks
    Lars
     
    Lars Purschke, Oct 20, 2003
    #2
    1. Advertising

  3. In article <>,
    Lars Purschke <> wrote:
    :We have two PIX 501 6.2.2 site to site connected via VPN with the main
    :pIX 6.3.1. Several times a day all connections are closed by a SYN
    :Timeout at the same time and always with a duration of 0:02:01.

    :%PIX-6-302014: Teardown TCP connection 5873 for outside:xxx to
    :inside:yyy duration 0:02:01 bytes 0 SYN Timeout

    :The IPSec SA's are still valid and the TCP connection timeout is set to
    :9:00:00. I've checked the Internet connection but it's working all the time.

    Are you saying that active connections that had been up for some
    time are being reported closed with SYN timeout duration 2:01, 0 bytes?
    Even though they weren't even in waiting-for-syn status? If so, that
    would definitely be a *big* bug! Turn your log level up to debugging,
    capture one of these, and open a case with the TAC if this is what
    is happening!!


    Or are you saying that at some time during the day, old connections
    drop, and *new* connections stop working, and those new connection
    attempts show up with SYN timeout duration 2:01, 0 bytes? If that is
    what is happening, then it is possible that the PIX is fine but
    that outside network is dropping out. When the problem is happening,
    can you go on to the PIX and ping the next hop? During the time
    that the connections are timing out, does 'debug packet' show traffic
    on the outside interface?


    What model is your central PIX that is running 6.3(1) ? If it
    is a PIX 515 or PIX 525, have you checked the Field Service Advisories?
    Some 515 and 525 have had hardware problems.
    --
    "No one has the right to destroy another person's belief by
    demanding empirical evidence." -- Ann Landers
     
    Walter Roberson, Oct 20, 2003
    #3
  4. It's your second suggestion. I've already pinged the next hop all day
    long from an outside computer and there were no failed packets. The main
    PIX is a 501 and has OS ver. 6.1.1 - sorry I have mistyped it before.
    After rebooting the Remote PIX everything works fine again for a couple
    of hours.

    Here some more logfiles from the MAIN Pix:

    2003-10-20 17:30:08 Local4.Info MAIN %PIX-6-602301: sa created, (sa) sa_dest= REMOTE, sa_prot= 50,
    sa_spi= 0xe0b8a9a6(3770198438), sa_trans= esp-3des esp-md5-hmac ,
    sa_conn_id= 5

    2003-10-20 17:30:08 Local4.Info MAIN %PIX-6-302013: Built inbound TCP connection 43 for
    outside:192.168.2.106/4156 (192.168.2.106/4156) to
    inside:192.168.0.100/449 (192.168.0.100/449)

    2003-10-20 17:31:11 Local4.Info MAIN %PIX-6-302013: Built inbound TCP connection 44 for
    outside:192.168.2.106/1896 (192.168.2.106/1896) to
    inside:192.168.0.100/449 (192.168.0.100/449)

    2003-10-20 17:32:09 Local4.Info MAIN %PIX-6-302014: Teardown TCP connection 43 for
    outside:192.168.2.106/4156 to inside:192.168.0.100/449 duration 0:02:01
    bytes 0 SYN Timeout

    2003-10-20 17:33:13 Local4.Info MAIN %PIX-6-302014: Teardown TCP connection 44 for
    outside:192.168.2.106/1896 to inside:192.168.0.100/449 duration 0:02:01
    bytes 0 SYN Timeout

    --> now only rebooting the Remote PIX helps to get it working again.

    What exactly means "SYN timeout"? Is the remote PIX not answering any more?

    Thanks,
    Lars

    Walter Roberson wrote:

    > In article <>,
    > Lars Purschke <> wrote:
    > :We have two PIX 501 6.2.2 site to site connected via VPN with the main
    > :pIX 6.3.1. Several times a day all connections are closed by a SYN
    > :Timeout at the same time and always with a duration of 0:02:01.
    >
    > :%PIX-6-302014: Teardown TCP connection 5873 for outside:xxx to
    > :inside:yyy duration 0:02:01 bytes 0 SYN Timeout
    >
    > :The IPSec SA's are still valid and the TCP connection timeout is set to
    > :9:00:00. I've checked the Internet connection but it's working all the time.
    >
    > Are you saying that active connections that had been up for some
    > time are being reported closed with SYN timeout duration 2:01, 0 bytes?
    > Even though they weren't even in waiting-for-syn status? If so, that
    > would definitely be a *big* bug! Turn your log level up to debugging,
    > capture one of these, and open a case with the TAC if this is what
    > is happening!!
    >
    >
    > Or are you saying that at some time during the day, old connections
    > drop, and *new* connections stop working, and those new connection
    > attempts show up with SYN timeout duration 2:01, 0 bytes? If that is
    > what is happening, then it is possible that the PIX is fine but
    > that outside network is dropping out. When the problem is happening,
    > can you go on to the PIX and ping the next hop? During the time
    > that the connections are timing out, does 'debug packet' show traffic
    > on the outside interface?
    >
    >
    > What model is your central PIX that is running 6.3(1) ? If it
    > is a PIX 515 or PIX 525, have you checked the Field Service Advisories?
    > Some 515 and 525 have had hardware problems.
    >
     
    Lars Purschke, Oct 20, 2003
    #4
  5. In article <>,
    Lars Purschke <> wrote:
    :The main
    :pIX is a 501 and has OS ver. 6.1.1 - sorry I have mistyped it before.

    Is ssh enabled on the 501? If so, then you should upgrade it. 6.1(1)
    has a vulnerability:

    http://www.cisco.com/warp/public/707/SSH-scanning.shtml


    Generally speaking, the 501's have weak power connectors, and people
    have reported overheating with them. Make sure the power connector is
    fastened securely to the unit, and make sure the unit is not hot --
    make sure there is good air flow around it, and that it isn't stacked
    with hot devices.


    :Here some more logfiles from the MAIN Pix:

    :2003-10-20 17:30:08 Local4.Info MAIN %PIX-6-302013: Built inbound TCP connection 43 for
    :eek:utside:192.168.2.106/4156 (192.168.2.106/4156) to
    :inside:192.168.0.100/449 (192.168.0.100/449)

    :2003-10-20 17:32:09 Local4.Info MAIN %PIX-6-302014: Teardown TCP connection 43 for
    :eek:utside:192.168.2.106/4156 to inside:192.168.0.100/449 duration 0:02:01
    :bytes 0 SYN Timeout

    Interesting -- that is telling us that the PIX thinks that *inside*
    device 192.168.0.100 has stopped responding. Can you ping that device
    from the PIX while the problem is happening? Can you ping it from
    other inside devices? Does the PIX arp table have an entry for it at that
    time?

    :What exactly means "SYN timeout"? Is the remote PIX not answering any more?

    Notice that the logs say "Built inbound". Inbound means that the outside
    interface received a packet destined for the inside interface -- so
    the remote PIX is alive and sending traffic to you, and something is
    happening at your side.

    "SYN timeout" means that the PIX has sent a TCP SYN packet to the
    inside IP address, but the inside IP address did not respond within
    the timeout (2 minutes.) A SYN timeout would be a normal response
    if the inside IP address did not exist, or if the inside IP address
    was not responding. SYN is the TCP flag indicating that TCP sequence
    numbers need to be SYNchronized, which is the very first thing that
    happens in a TCP connection. The normal response from the
    target system would be a packet with the SYN ACK flags set.
    --
    Beware of bugs in the above code; I have only proved it correct,
    not tried it. -- Donald Knuth
     
    Walter Roberson, Oct 20, 2003
    #5
  6. By the time the pix stops working I can't ping the pix from the inside
    network any more. The PDM is also not accessible at that time from the
    inside network and the arp tables on the internal clients have no entry
    of the pix mac address.

    any suggestions?

    Walter Roberson wrote:

    > In article <>,
    > Lars Purschke <> wrote:
    > :The main
    > :pIX is a 501 and has OS ver. 6.1.1 - sorry I have mistyped it before.
    >
    > Is ssh enabled on the 501? If so, then you should upgrade it. 6.1(1)
    > has a vulnerability:
    >
    > http://www.cisco.com/warp/public/707/SSH-scanning.shtml
    >
    >
    > Generally speaking, the 501's have weak power connectors, and people
    > have reported overheating with them. Make sure the power connector is
    > fastened securely to the unit, and make sure the unit is not hot --
    > make sure there is good air flow around it, and that it isn't stacked
    > with hot devices.
    >
    >
    > :Here some more logfiles from the MAIN Pix:
    >
    > :2003-10-20 17:30:08 Local4.Info MAIN %PIX-6-302013: Built inbound TCP connection 43 for
    > :eek:utside:192.168.2.106/4156 (192.168.2.106/4156) to
    > :inside:192.168.0.100/449 (192.168.0.100/449)
    >
    > :2003-10-20 17:32:09 Local4.Info MAIN %PIX-6-302014: Teardown TCP connection 43 for
    > :eek:utside:192.168.2.106/4156 to inside:192.168.0.100/449 duration 0:02:01
    > :bytes 0 SYN Timeout
    >
    > Interesting -- that is telling us that the PIX thinks that *inside*
    > device 192.168.0.100 has stopped responding. Can you ping that device
    > from the PIX while the problem is happening? Can you ping it from
    > other inside devices? Does the PIX arp table have an entry for it at that
    > time?
    >
    > :What exactly means "SYN timeout"? Is the remote PIX not answering any more?
    >
    > Notice that the logs say "Built inbound". Inbound means that the outside
    > interface received a packet destined for the inside interface -- so
    > the remote PIX is alive and sending traffic to you, and something is
    > happening at your side.
    >
    > "SYN timeout" means that the PIX has sent a TCP SYN packet to the
    > inside IP address, but the inside IP address did not respond within
    > the timeout (2 minutes.) A SYN timeout would be a normal response
    > if the inside IP address did not exist, or if the inside IP address
    > was not responding. SYN is the TCP flag indicating that TCP sequence
    > numbers need to be SYNchronized, which is the very first thing that
    > happens in a TCP connection. The normal response from the
    > target system would be a packet with the SYN ACK flags set.
    >
     
    Lars Purschke, Nov 2, 2003
    #6
  7. In article <>,
    Lars Purschke <> wrote:
    :By the time the pix stops working I can't ping the pix from the inside
    :network any more. The PDM is also not accessible at that time from the
    :inside network and the arp tables on the internal clients have no entry
    :eek:f the pix mac address.

    Earlier, you wrote,

    |> :Here some more logfiles from the MAIN Pix:

    |> :2003-10-20 17:30:08 Local4.Info MAIN %PIX-6-302013: Built inbound TCP connection 43 for
    |> :eek:utside:192.168.2.106/4156 (192.168.2.106/4156) to
    |> :inside:192.168.0.100/449 (192.168.0.100/449)

    |> :2003-10-20 17:32:09 Local4.Info MAIN %PIX-6-302014: Teardown TCP connection 43 for
    |> :eek:utside:192.168.2.106/4156 to inside:192.168.0.100/449 duration 0:02:01
    |> :bytes 0 SYN Timeout


    I notice that those log entries are timestamped and show the priority
    level (Local4.Info). That's a syslog format, not the format that
    PIX uses to log messages to its internal buffer. How are you logging
    those messages if you cannot reach internal hosts? The PIX 501
    only has 2 interfaces, so you cannot be logging to something on
    a third interface. Are you sending your PIX logs to a remote host?
    --
    *We* are now the times. -- Wim Wenders (WoD)
     
    Walter Roberson, Nov 2, 2003
    #7
  8. I'm logging to an external host. I know it's a security issue, but I
    need to do it this way.

    Walter Roberson wrote:

    > In article <>,
    > Lars Purschke <> wrote:
    > :By the time the pix stops working I can't ping the pix from the inside
    > :network any more. The PDM is also not accessible at that time from the
    > :inside network and the arp tables on the internal clients have no entry
    > :eek:f the pix mac address.
    >
    > Earlier, you wrote,
    >
    > |> :Here some more logfiles from the MAIN Pix:
    >
    > |> :2003-10-20 17:30:08 Local4.Info MAIN %PIX-6-302013: Built inbound TCP connection 43 for
    > |> :eek:utside:192.168.2.106/4156 (192.168.2.106/4156) to
    > |> :inside:192.168.0.100/449 (192.168.0.100/449)
    >
    > |> :2003-10-20 17:32:09 Local4.Info MAIN %PIX-6-302014: Teardown TCP connection 43 for
    > |> :eek:utside:192.168.2.106/4156 to inside:192.168.0.100/449 duration 0:02:01
    > |> :bytes 0 SYN Timeout
    >
    >
    > I notice that those log entries are timestamped and show the priority
    > level (Local4.Info). That's a syslog format, not the format that
    > PIX uses to log messages to its internal buffer. How are you logging
    > those messages if you cannot reach internal hosts? The PIX 501
    > only has 2 interfaces, so you cannot be logging to something on
    > a third interface. Are you sending your PIX logs to a remote host?
    >
     
    Lars Purschke, Nov 2, 2003
    #8
  9. In article <>,
    Lars Purschke <> wrote:
    :By the time the pix stops working I can't ping the pix from the inside
    :network any more. The PDM is also not accessible at that time from the
    :inside network and the arp tables on the internal clients have no entry
    :eek:f the pix mac address.

    You mentioned before an AS/400 internally, and you mention now
    "internal clients". It seems likely that you have more than 3 internal
    clients, so I deduce that you probably have at least one internal
    switch (since the PIX 501 only has four switched ports.)

    Can you tell us more about your internal network? Do you have more than
    one switch? If so, exactly what models? Are you using VLANs on those
    switches?

    The situation you describe could be caused by "vlan flapping". If an
    802.11Q switch learns a MAC address on a second port and decides to
    stops sending packets to that MAC on the first port, you can run into
    strange communication problems. If you are not using VLANs then this
    kind of problem would normally be resolved by Spanning Tree negotiations.
    If, though, you are using VLANs then might need per-VLAN spanning trees.

    A few months ago, someone brought this problem to our collective
    attention here in the context of asking about a Layer 3 Switch that
    happened to assign the same MAC address to every VLAN. It turns out
    that some of the Cisco multilayer switches and router do the same thing
    (e.g., the C5500+RSM). The poster at the time had different VLANs
    fanning out of their router, and was connecting more than one port-based
    VLAN to their switch, but the switch was seeing the same MAC in
    both cases and was not aware that the same MAC can appear in
    different VLANs, so it was assuming that the device was moving
    between the two ports every couple of seconds. Made for pretty
    inconsistant data reception...

    The same kind of situation has cropped up in this newsgroup a couple of
    times since then as well.
    --
    Before responding, take into account the possibility that the Universe
    was created just an instant ago, and that you have not actually read
    anything, but were instead created intact with a memory of having read it.
     
    Walter Roberson, Nov 3, 2003
    #9
  10. > :By the time the pix stops working I can't ping the pix from the inside
    > :network any more. The PDM is also not accessible at that time from the
    > :inside network and the arp tables on the internal clients have no entry
    > :eek:f the pix mac address.
    >
    > You mentioned before an AS/400 internally, and you mention now
    > "internal clients". It seems likely that you have more than 3 internal
    > clients, so I deduce that you probably have at least one internal
    > switch (since the PIX 501 only has four switched ports.)



    We have an AS/400 Server in our Main Office, where are also a lot of
    local Clients working on it. It's a 192.168.0.0/24 network where the
    AS/400 has the IP 192.168.0.100 and the PIX 192.168.0.200. Furthermore
    we have two smaller Offices where we build VPN Connections to the Main
    Office. There we have the networks 192.168.1.0/24 and 192.168.2.0/24.
    All Clients are working on the AS/400. The Internal Network looks like
    this, so every packet must go over the switch:


    Client A <--> <--> PIX 501 <--> VPN
    Client B <--> Switch
    Client C <--> <--> AS/400
    .....

    The Switch is a "no-name" device without VLAN and we have the PIX
    connected to the switch. The other internal ports on the PIX are not
    used. The AS/400 is also directly connected on the switch.

    On the PIX we have the default ARP timeout of 14400 sec.

    May I set the "noproxy-arp" Sysoption on the pix internal NIC? May
    putting a second switch between the first one, the AS/400 and the PIX
    could be a solution?

    Internal Network <-- Switch --> <-- Switch 2 --> AS/400
    --> PIX


    >
    > Can you tell us more about your internal network? Do you have more than
    > one switch? If so, exactly what models? Are you using VLANs on those
    > switches?
    >
    > The situation you describe could be caused by "vlan flapping". If an
    > 802.11Q switch learns a MAC address on a second port and decides to
    > stops sending packets to that MAC on the first port, you can run into
    > strange communication problems. If you are not using VLANs then this
    > kind of problem would normally be resolved by Spanning Tree negotiations.
    > If, though, you are using VLANs then might need per-VLAN spanning trees.
    >
    > A few months ago, someone brought this problem to our collective
    > attention here in the context of asking about a Layer 3 Switch that
    > happened to assign the same MAC address to every VLAN. It turns out
    > that some of the Cisco multilayer switches and router do the same thing
    > (e.g., the C5500+RSM). The poster at the time had different VLANs
    > fanning out of their router, and was connecting more than one port-based
    > VLAN to their switch, but the switch was seeing the same MAC in
    > both cases and was not aware that the same MAC can appear in
    > different VLANs, so it was assuming that the device was moving
    > between the two ports every couple of seconds. Made for pretty
    > inconsistant data reception...
    >
    > The same kind of situation has cropped up in this newsgroup a couple of
    > times since then as well.
    >
     
    Lars Purschke, Nov 3, 2003
    #10
  11. In article <>,
    Lars Purschke <> wrote:
    |> :By the time the pix stops working I can't ping the pix from the inside
    |> :network any more. The PDM is also not accessible at that time from the
    |> :inside network and the arp tables on the internal clients have no entry
    |> :eek:f the pix mac address.

    |The Switch is a "no-name" device without VLAN and we have the PIX
    |connected to the switch. The other internal ports on the PIX are not
    |used. The AS/400 is also directly connected on the switch.

    I would suspect a switch problem of some sort.

    :May I set the "noproxy-arp" Sysoption on the pix internal NIC?

    You could, but I do not think that would help you at all. The inside
    NIC is not going to be proxy-arp'ing on behalf of any IP address
    (not unless you are using "outside nat".)

    :May
    :putting a second switch between the first one, the AS/400 and the PIX
    :could be a solution?

    Possibly, but I would suggest changing the switch to something
    reliable from Cisco or Nortel. Perhaps you can borrow a switch long
    enough to test?
    --
    "The human genome is powerless in the face of chocolate."
    -- Dr. Adam Drewnowski
     
    Walter Roberson, Nov 3, 2003
    #11
  12. > I would suspect a switch problem of some sort.


    > Possibly, but I would suggest changing the switch to something
    > reliable from Cisco or Nortel. Perhaps you can borrow a switch long
    > enough to test?
    >



    Thanks! I'll give it a try and change the switch.
     
    Lars Purschke, Nov 3, 2003
    #12
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Jimmy
    Replies:
    4
    Views:
    6,189
    Jimmy
    Nov 10, 2003
  2. Dave
    Replies:
    4
    Views:
    5,320
  3. The J-Man
    Replies:
    0
    Views:
    650
    The J-Man
    Jan 8, 2006
  4. Scott Townsend
    Replies:
    0
    Views:
    6,491
    Scott Townsend
    May 24, 2006
  5. John Caruso
    Replies:
    1
    Views:
    784
    John Caruso
    Nov 20, 2006
Loading...

Share This Page