Troubleshooting site-to-site VPN with two PIX 501s

I have a central office PIX 501 that currently runs independent
site-to-site VPNs for around 5 remote offices, and I'm trying to add a
6th VPN. I'm using pre-shared keys for authentication.

Central office LAN is 172.16.1.0/16, remote office LAN is
172.20.6.0/24. When I try and pass interesting traffic over the VPN,
the VPN LED on the PIX illuminates so it seems like a connection is
actually made. However, the packets never get through -- pings and
telnet attempts time out. I have even added a static route on a target
machine in the 172.16.1.0 network.

What information could I please provide to help y'all debug this with
me? Here is the important output from config term:

(central office)

ip address outside 64.sss.198.6 255.255.255.240
ip address inside 172.16.1.192 255.255.0.0
isakmp key ****** address 63.xxx.214.138 netmask 255.255.255.255
access-list nonat permit ip 172.16.1.0 255.255.255.0 172.20.6.0
255.255.255.0
access-list nonat permit ip 172.16.10.0 255.255.255.0 172.20.6.0
255.255.255.0
access-list nonat permit ip 172.16.11.0 255.255.255.0 172.20.6.0
255.255.255.0
access-list SPRINGVALE permit ip 172.16.1.0 255.255.255.0 172.20.6.0
255.255.255.0
access-list SPRINGVALE permit ip 172.16.10.0 255.255.255.0 172.20.6.0
255.255.255.0
access-list SPRINGVALE permit ip 172.16.11.0 255.255.255.0 172.20.6.0
255.255.255.0
crypto ipsec transform-set springvaleset esp-3des esp-md5-hmac
crypto map site-vpn 6 ipsec-isakmp
crypto map site-vpn 6 set peer 63.xxx.214.138
crypto map site-vpn 6 set transform-set springvaleset
crypto map site-vpn 6 match address SPRINGVALE
sysopt connection permit-ipsec
global (outside) 1 interface
nat (inside) 0 access-list nonat
nat (inside) 1 0.0.0.0 0.0.0.0 0 5000
route outside 0.0.0.0 0.0.0.0 64.sss.198.1 1

(remote office)

ip address outside 63.xxx.214.138 255.255.255.0
ip address inside 172.20.6.191 255.255.255.0
access-list rcd_pph permit ip 172.20.6.0 255.255.255.0 172.16.0.0
255.255.0.0
global (outside) 1 interface
nat (inside) 0 access-list rcd_pph
nat (inside) 1 0.0.0.0 0.0.0.0 0 0
route outside 0.0.0.0 0.0.0.0 63.xxx.214.129 1
sysopt connection permit-ipsec
crypto ipsec transform-set transrcd esp-3des esp-md5-hmac
crypto map rcdmap 1 ipsec-isakmp
crypto map rcdmap 1 match address rcd_pph
crypto map rcdmap 1 set peer 64.sss.198.6
crypto map rcdmap 1 set transform-set transrcd
crypto map rcdmap interface outside
isakmp enable outside
isakmp key q1w2e3r4 address 64.sss.198.6 netmask 255.255.255.255
isakmp identity address
isakmp policy 1 authentication pre-share
isakmp policy 1 encryption 3des
isakmp policy 1 hash md5
isakmp policy 1 group 2
isakmp policy 1 lifetime 86400

Given that I have existing remote networks (e.g. 172.20.1.0/24,
172.20.2.0/24, etc) connecting to the central office I am quite baffled
by what the problem is. When I try and ping across the VPN, here is the
output from a combined "debug crypto {ipsec,isakmp,engine}" running on
the remote office PIX:

ISAKMP (0): beginning Quick Mode exchange, M-ID of
1243172245:4a194d95IPSEC(key_engine): got a queue event...
IPSEC(spi_response): getting spi 0x596bc730(1500235568) for SA
from 64.sss.198.6 to 63.xxx.214.138 for prot 3

crypto_isakmp_process_block:src:64.sss.198.6, dest:63.xxx.214.138
spt:500 dpt:500
ISAKMP (0): processing NOTIFY payload 14 protocol 3
spi 1500235568, message ID = 1860390180
ISAKMP (0): deleting spi 818375513 message ID = 1243172245
return status is IKMP_NO_ERR_NO_TRANS6: ICMP echo-request from
inside:172.20.6.1 to 172.16.1.30 ID=768 seq=22784 length=40
7: ICMP echo-request from inside:172.20.6.1 to 172.16.1.30 ID=768
seq=23040 length=40
8: ICMP echo-request from inside:172.20.6.1 to 172.16.1.30 ID=768
seq=23296 length=40
IPSEC(key_engine): request timer fired: count = 1,
(identity) local= 63.xxx.214.138, remote= 64.sss.198.6,
local_proxy= 172.20.6.0/255.255.255.0/0/0 (type=4),
remote_proxy= 172.16.0.0/255.255.0.0/0/0 (type=4)

ISAKMP (0): beginning Quick Mode exchange, M-ID of
-31348343:fe21a989IPSEC(key_engine): got a queue event...
IPSEC(spi_response): getting spi 0xa180df8c(2709577612) for SA
from 64.sss.198.6 to 63.xxx.214.138 for prot 3

crypto_isakmp_process_block:src:64.sss.198.6, dest:63.xxx.214.138
spt:500 dpt:500
ISAKMP (0): processing NOTIFY payload 14 protocol 3
spi 2709577612, message ID = 1859416444
ISAKMP (0): deleting spi 2363457697 message ID = 4263618953
return status is IKMP_NO_ERR_NO_TRANS9: ICMP echo-request from
inside:172.20.6.1 to 172.16.1.30 ID=768 seq=23552 length=40
10: ICMP echo-request from inside:172.20.6.1 to 172.16.1.30 ID=768
seq=23808 length=40
11: ICMP echo-request from inside:172.20.6.1 to 172.16.1.30 ID=768
seq=24064 length=40
12: ICMP echo-request from inside:172.20.6.1 to 172.16.1.30 ID=768
seq=24320 length=40
13: ICMP echo-request from inside:172.20.6.1 to 172.16.1.30 ID=768
seq=24576 length=40
14: ICMP echo-request from inside:172.20.6.1 to 172.16.1.30 ID=768
seq=24832 length=40
IPSEC(key_engine): request timer fired: count = 2,
(identity) local= 63.xxx.214.138, remote= 64.sss.198.6,
local_proxy= 172.20.6.0/255.255.255.0/0/0 (type=4),
remote_proxy= 172.16.0.0/255.255.0.0/0/0 (type=4)
15: ICMP echo-request from inside:172.20.6.1 to 172.16.1.30 ID=768
seq=25088 length=40

And so on and so on. It's interesting that I never see what is a
"fatal" error; or perhaps I'm not interpreting the output correctly?
TIA for any & all help!


Chris

 

Hi Chris,

Possibly your Cisco PIX 501 license may only allow 10 concurrent users.


If you require more than 10 users to have access through the PIX 501
Firewall at one time, perform these steps:

1. Purchase a 50-user license upgrade.

The part number is: PIX-501-SW-10-50

For a Price Quote call Reggie Grant Toll Free 877-549-2680 or
828-277-7272:

http://www.bradreese.com/contact-us.htm

Additionally, you may wish to investigate Cisco PIX Security Appliance
Licensing:

http://www.cisco.com/en/US/products/hw/vpndevc/ps2030/products_data_sheet09186a00800b0d85.html

2. If you have already purchased a license, send an email to:

licensing *at* cisco.com

Include serial number, purchase information, PIX Firewall Software
version and model and what needs to be added on the activation key.

To get the PIX serial number, PIX software version and PIX model
number, issue the show version command.

A show version command also tells you what type of license the PIX
Firewall is running, either R ( restricted ) or UR ( unrestricted ).

Hope this helps.

Brad Reese
BradReese.Com - Refurbished Cisco PIX Firewall Guide
http://www.bradreese.com/refurbished-cisco-pix-firewalls.htm
1293 Hendersonville Road, Suite 17
Asheville, North Carolina USA 28803
USA & Canada: 877-549-2680
International: 828-277-7272
Fax: 775-254-3558
AIM: R2MGrant
BradReese.Com - Cisco Power Supply Headquarters
http://www.bradreese.com/cisco-power-supply-inventory.htm

 

Trolling through a support groups to drum up business is not generally
accepted, that is what advertising is for.


http://www.cisco.com/en/US/products/hw/vpndevc/ps2030/products_data_sheet09186a0080091b18.html
Simultaneous VPN peers: 10*
* Maximum number of simultaneous site-to-site or remote access IKE
Security Association (SAs) supported

Since this is his 6th IPSec connection there is no license issue.

When you have the remote unit up, what does "show crypto ipsec sa" show?
It is kind of late where I am so I am going to look at this tomorrow,
but in the meantime, I know this is a stupid question, but do you have
ICMP allowed through the interfaces? For example, if you do not have
ICMP allowed for the LAN address at the remote it will of course fail in
pinging. The reason I say this is this just came up about 2 days ago.
Someone freeked out because they thought their VPN did not work, when in
fact it was only that pinging was disable from their remote network.

When setting up the 6th site, did you recreate from scratch or copy an
existing config editing the specifics like IP address?

 

Does the "10-user limit" work off of 10 separate VPN connections with N
users connected to each (i.e. the actual amount of *users* is
irrelevant), or can you have N connections so longer as the number of
users < 10?

I didn't explicitly allow (or deny) ICMP... I am only using ACLs to
differentiate between traffic that should go across the VPN, and
traffic that shouldn't. Perhaps I have done this incorrectly? Either
way, neither ping nor telnet across the VPN worked; hence I didn't
think it was just an ICMP issue (though I could be wrong!)

The latter, I just changed what I thought the connection would need.
I'm most confused that the VPN LED on the PIX illuminates, indicating
that there *is* a connection; just that I can't get the connection to
do what I want it to.


Chris

 

I forgot to look something up... the central office PIX actually has an
unlimited license, thus it obviously doesn't care how many PIXes are
connecting to it. The remote PIX has a 10-user license, but that's 9
more users than I have.

CO PIX:

Licensed Features:
Failover: Disabled
VPN-DES: Enabled
VPN-3DES-AES: Enabled
Maximum Physical Interfaces: 2
Maximum Interfaces: 2
Cut-through Proxy: Enabled
Guards: Enabled
URL-filtering: Enabled
Inside Hosts: 50
Throughput: Unlimited
IKE peers: 10


Chris

 

He hasn't happened to mention which PIX OS version he is running,
and the config extracts did not happen to include anything
characteristic of one PIX version but not another.

On the 501, the peer limit did not become 10 until PIX 6.3(1);
it is 5 in PIX 6.1 and PIX 6.2.

 

One of the PIXes is running 6.3(4), the other is running 6.3(3).


Chris

 

Number of comp.dcom.sys.cisco postings presenting useful information
attempting to help people:

Brad Reese (, ,
): more than 1100

John Doe (, , ,
, , , ,
AKA Bryan Martin , et al.): less than 40


I've never seen Brad troll *this* newsgroup (alt.marketing.online.ebay
perhaps, but not comp.dcom.sys.cisco.) I have seen him attempting to
help people many times. He perhaps does not always read the problem
as deeply as might be desirable, and tends to post documentation
references instead of his own explanations -- but he does *try* to help
people.

 

Helping is one thing all together different than trying to make a
unnecessary sale. At the very least lets work out Chris's problem first
and foremost. If there was a need to upgrade the license then maybe
options could be discussed. I just take this from the perspective of
being a reseller myself (I am here to learn and help out if I can).
Maybe I was a little harsh, and for that I apologize, but first and
foremost let us deal with the problem at hand and worry about upgrades
and such later.

 

I would also add for convenience in testing
"icmp permit any inside"

Also, does "show crypto sa" clearly show that the SA's are established?
I believe though the problem lies more with NAT, I just do not really
see where the problem lies, all the access lists look fine.

I just had issues the other day where using FreeBSD to Cisco will create
a tunnel, but SAs are not established. So the PIX would show a
connection, yet no traffic can traverse the VPN.

Try the command on both central and remote to see what happens. Also,
have you tried connection in reverse, meaning from central to remote?


Here is an example of what you should see with "show crypto sa":

interface: outside
Crypto map tag: dyn-map, local addr. 6x.xxx.xxx.xx
local ident (addr/mask/prot/port): (192.168.1.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (192.168.2.0/255.255.255.0/0/0)
current_peer: 72.135.40.162:500
PERMIT, flags={}
#pkts encaps: 40276, #pkts encrypt: 40276, #pkts digest 40276
#pkts decaps: 33478, #pkts decrypt: 33478, #pkts verify 33478
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0, #pkts decompress
failed: 0
#send errors 0, #recv errors 0
local crypto endpt.: 6x.xxx.xxx.xx, remote crypto endpt.:
7x.xxx.xxx.xx
path mtu 1500, ipsec overhead 72, media mtu 1500
current outbound spi: 5f8992c2
inbound esp sas:
spi: 0x62de9891(1658755217)
transform: esp-aes-256 esp-md5-hmac ,
in use settings ={Tunnel, }
slot: 0, conn id: 4, crypto map: dyn-map
sa timing: remaining key lifetime (k/sec): (4600201/26885)
IV size: 16 bytes
replay detection support: Y
inbound ah sas:
inbound pcp sas:
outbound esp sas:
spi: 0x5f8992c2(1602851522)
transform: esp-aes-256 esp-md5-hmac ,
in use settings ={Tunnel, }
slot: 0, conn id: 3, crypto map: dyn-map
sa timing: remaining key lifetime (k/sec): (4599036/26885)
IV size: 16 bytes
replay detection support: Y
outbound ah sas:
outbound pcp sas:

Also, have you brought this to the discussion boards at Cisco.com?
Sometimes a little slow to get a response, but usually there is a really
high level of expertise there.