PIX 501 - SYN Timeout Duration 0:02:01

Check for assymetric routing. ICMP can still "work" under said
conditions.

Pix also had a problem (cant remember which code, but it surfaced on 6.1)
with tcp connections when window size approaching 65K, so might want to
check that too. Under this condition, the pix would pass the SYN-ACK
back, but silently (i.e. syslog shows nothing) drop the ACK. Syslog
subsequently report SYN timeout.

 

Hi!

We have two PIX 501 6.2.2 site to site connected via VPN with the main
PIX 6.3.1. Several times a day all connections are closed by a SYN
Timeout at the same time and always with a duration of 0:02:01.

%PIX-6-302014: Teardown TCP connection 5873 for outside:xxx to
inside:yyy duration 0:02:01 bytes 0 SYN Timeout

The IPSec SA's are still valid and the TCP connection timeout is set to
9:00:00. I've checked the Internet connection but it's working all the time.

timeout xlate 5:00:00
timeout conn 9:00:00 half-closed 0:10:00 udp 2:30:00 rpc 2:30:00 h225
1:00:00
timeout h323 0:05:00 mgcp 0:05:00 sip 0:30:00 sip_media 0:02:00
timeout uauth 0:05:00 absolute

How can this happen? Does the PIX cancel the connections or could it be
the AS/400 Server? What can I do to avoid this?

Thanks
Lars

 

:We have two PIX 501 6.2.2 site to site connected via VPN with the main
IX 6.3.1. Several times a day all connections are closed by a SYN
:Timeout at the same time and always with a duration of 0:02:01.

:%PIX-6-302014: Teardown TCP connection 5873 for outside:xxx to
:inside:yyy duration 0:02:01 bytes 0 SYN Timeout

:The IPSec SA's are still valid and the TCP connection timeout is set to
:9:00:00. I've checked the Internet connection but it's working all the time.

Are you saying that active connections that had been up for some
time are being reported closed with SYN timeout duration 2:01, 0 bytes?
Even though they weren't even in waiting-for-syn status? If so, that
would definitely be a *big* bug! Turn your log level up to debugging,
capture one of these, and open a case with the TAC if this is what
is happening!!


Or are you saying that at some time during the day, old connections
drop, and *new* connections stop working, and those new connection
attempts show up with SYN timeout duration 2:01, 0 bytes? If that is
what is happening, then it is possible that the PIX is fine but
that outside network is dropping out. When the problem is happening,
can you go on to the PIX and ping the next hop? During the time
that the connections are timing out, does 'debug packet' show traffic
on the outside interface?


What model is your central PIX that is running 6.3(1) ? If it
is a PIX 515 or PIX 525, have you checked the Field Service Advisories?
Some 515 and 525 have had hardware problems.

 

It's your second suggestion. I've already pinged the next hop all day
long from an outside computer and there were no failed packets. The main
PIX is a 501 and has OS ver. 6.1.1 - sorry I have mistyped it before.
After rebooting the Remote PIX everything works fine again for a couple
of hours.

Here some more logfiles from the MAIN Pix:

2003-10-20 17:30:08 Local4.Info MAIN %PIX-6-602301: sa created, (sa) sa_dest= REMOTE, sa_prot= 50,
sa_spi= 0xe0b8a9a6(3770198438), sa_trans= esp-3des esp-md5-hmac ,
sa_conn_id= 5

2003-10-20 17:30:08 Local4.Info MAIN %PIX-6-302013: Built inbound TCP connection 43 for
outside:192.168.2.106/4156 (192.168.2.106/4156) to
inside:192.168.0.100/449 (192.168.0.100/449)

2003-10-20 17:31:11 Local4.Info MAIN %PIX-6-302013: Built inbound TCP connection 44 for
outside:192.168.2.106/1896 (192.168.2.106/1896) to
inside:192.168.0.100/449 (192.168.0.100/449)

2003-10-20 17:32:09 Local4.Info MAIN %PIX-6-302014: Teardown TCP connection 43 for
outside:192.168.2.106/4156 to inside:192.168.0.100/449 duration 0:02:01
bytes 0 SYN Timeout

2003-10-20 17:33:13 Local4.Info MAIN %PIX-6-302014: Teardown TCP connection 44 for
outside:192.168.2.106/1896 to inside:192.168.0.100/449 duration 0:02:01
bytes 0 SYN Timeout

--> now only rebooting the Remote PIX helps to get it working again.

What exactly means "SYN timeout"? Is the remote PIX not answering any more?

Thanks,
Lars

 

:The main
IX is a 501 and has OS ver. 6.1.1 - sorry I have mistyped it before.

Is ssh enabled on the 501? If so, then you should upgrade it. 6.1(1)
has a vulnerability:

http://www.cisco.com/warp/public/707/SSH-scanning.shtml


Generally speaking, the 501's have weak power connectors, and people
have reported overheating with them. Make sure the power connector is
fastened securely to the unit, and make sure the unit is not hot --
make sure there is good air flow around it, and that it isn't stacked
with hot devices.


:Here some more logfiles from the MAIN Pix:

:2003-10-20 17:30:08 Local4.Info MAIN %PIX-6-302013: Built inbound TCP connection 43 for
utside:192.168.2.106/4156 (192.168.2.106/4156) to
:inside:192.168.0.100/449 (192.168.0.100/449)

:2003-10-20 17:32:09 Local4.Info MAIN %PIX-6-302014: Teardown TCP connection 43 for
utside:192.168.2.106/4156 to inside:192.168.0.100/449 duration 0:02:01
:bytes 0 SYN Timeout

Interesting -- that is telling us that the PIX thinks that *inside*
device 192.168.0.100 has stopped responding. Can you ping that device
from the PIX while the problem is happening? Can you ping it from
other inside devices? Does the PIX arp table have an entry for it at that
time?

:What exactly means "SYN timeout"? Is the remote PIX not answering any more?

Notice that the logs say "Built inbound". Inbound means that the outside
interface received a packet destined for the inside interface -- so
the remote PIX is alive and sending traffic to you, and something is
happening at your side.

"SYN timeout" means that the PIX has sent a TCP SYN packet to the
inside IP address, but the inside IP address did not respond within
the timeout (2 minutes.) A SYN timeout would be a normal response
if the inside IP address did not exist, or if the inside IP address
was not responding. SYN is the TCP flag indicating that TCP sequence
numbers need to be SYNchronized, which is the very first thing that
happens in a TCP connection. The normal response from the
target system would be a packet with the SYN ACK flags set.

 

By the time the pix stops working I can't ping the pix from the inside
network any more. The PDM is also not accessible at that time from the
inside network and the arp tables on the internal clients have no entry
of the pix mac address.

any suggestions?

 

:By the time the pix stops working I can't ping the pix from the inside
:network any more. The PDM is also not accessible at that time from the
:inside network and the arp tables on the internal clients have no entry
f the pix mac address.

Earlier, you wrote,

|> :Here some more logfiles from the MAIN Pix:

|> :2003-10-20 17:30:08 Local4.Info MAIN %PIX-6-302013: Built inbound TCP connection 43 for
|> utside:192.168.2.106/4156 (192.168.2.106/4156) to
|> :inside:192.168.0.100/449 (192.168.0.100/449)

|> :2003-10-20 17:32:09 Local4.Info MAIN %PIX-6-302014: Teardown TCP connection 43 for
|> utside:192.168.2.106/4156 to inside:192.168.0.100/449 duration 0:02:01
|> :bytes 0 SYN Timeout


I notice that those log entries are timestamped and show the priority
level (Local4.Info). That's a syslog format, not the format that
PIX uses to log messages to its internal buffer. How are you logging
those messages if you cannot reach internal hosts? The PIX 501
only has 2 interfaces, so you cannot be logging to something on
a third interface. Are you sending your PIX logs to a remote host?

 

I'm logging to an external host. I know it's a security issue, but I
need to do it this way.

 

:By the time the pix stops working I can't ping the pix from the inside
:network any more. The PDM is also not accessible at that time from the
:inside network and the arp tables on the internal clients have no entry
f the pix mac address.

You mentioned before an AS/400 internally, and you mention now
"internal clients". It seems likely that you have more than 3 internal
clients, so I deduce that you probably have at least one internal
switch (since the PIX 501 only has four switched ports.)

Can you tell us more about your internal network? Do you have more than
one switch? If so, exactly what models? Are you using VLANs on those
switches?

The situation you describe could be caused by "vlan flapping". If an
802.11Q switch learns a MAC address on a second port and decides to
stops sending packets to that MAC on the first port, you can run into
strange communication problems. If you are not using VLANs then this
kind of problem would normally be resolved by Spanning Tree negotiations.
If, though, you are using VLANs then might need per-VLAN spanning trees.

A few months ago, someone brought this problem to our collective
attention here in the context of asking about a Layer 3 Switch that
happened to assign the same MAC address to every VLAN. It turns out
that some of the Cisco multilayer switches and router do the same thing
(e.g., the C5500+RSM). The poster at the time had different VLANs
fanning out of their router, and was connecting more than one port-based
VLAN to their switch, but the switch was seeing the same MAC in
both cases and was not aware that the same MAC can appear in
different VLANs, so it was assuming that the device was moving
between the two ports every couple of seconds. Made for pretty
inconsistant data reception...

The same kind of situation has cropped up in this newsgroup a couple of
times since then as well.

 

:By the time the pix stops working I can't ping the pix from the inside

We have an AS/400 Server in our Main Office, where are also a lot of
local Clients working on it. It's a 192.168.0.0/24 network where the
AS/400 has the IP 192.168.0.100 and the PIX 192.168.0.200. Furthermore
we have two smaller Offices where we build VPN Connections to the Main
Office. There we have the networks 192.168.1.0/24 and 192.168.2.0/24.
All Clients are working on the AS/400. The Internal Network looks like
this, so every packet must go over the switch:


Client A <--> <--> PIX 501 <--> VPN
Client B <--> Switch
Client C <--> <--> AS/400
.....

The Switch is a "no-name" device without VLAN and we have the PIX
connected to the switch. The other internal ports on the PIX are not
used. The AS/400 is also directly connected on the switch.

On the PIX we have the default ARP timeout of 14400 sec.

May I set the "noproxy-arp" Sysoption on the pix internal NIC? May
putting a second switch between the first one, the AS/400 and the PIX
could be a solution?

Internal Network <-- Switch --> <-- Switch 2 --> AS/400
--> PIX

 

|> :By the time the pix stops working I can't ping the pix from the inside
|> :network any more. The PDM is also not accessible at that time from the
|> :inside network and the arp tables on the internal clients have no entry
|> f the pix mac address.

|The Switch is a "no-name" device without VLAN and we have the PIX
|connected to the switch. The other internal ports on the PIX are not
|used. The AS/400 is also directly connected on the switch.

I would suspect a switch problem of some sort.

:May I set the "noproxy-arp" Sysoption on the pix internal NIC?

You could, but I do not think that would help you at all. The inside
NIC is not going to be proxy-arp'ing on behalf of any IP address
(not unless you are using "outside nat".)

:May
utting a second switch between the first one, the AS/400 and the PIX
:could be a solution?

Possibly, but I would suggest changing the switch to something
reliable from Cisco or Nortel. Perhaps you can borrow a switch long
enough to test?

 

I would suspect a switch problem of some sort.

Thanks! I'll give it a try and change the switch.