PIX/FWSM: allow inbound connections to dynamic NAT address?

Discussion in 'Cisco' started by marc.luethi, Nov 15, 2007.

  1. marc.luethi

    marc.luethi Guest

    A set of client PCs on the inside interface of a Firewall Services
    Module (FWSM) 2.3(1) is connecting outbound to a partner company. In
    general, all outbound connections to this destination network
    ( /24) are PATted to one address ( This is
    well-established and has been working well for a long time.

    A new application provided by the partner company is offered on a set
    of four servers with tcp/6500, and it has the inevitable requirement
    that the servers connect back to the clients on tcp/6500 within
    seconds (i.e. well within an xlate's lifetime) after the client
    successfully started the outbound connection.

    We would like to avoid configuring static NATs for the client systems
    by all possible means, because of administrative overhead and tedious
    work generated by PCs changing locations and IP addresses, so we want
    to stick with some form of dynamic NAT. The new application is not
    well-known and can't be inspected by the available "fixup" features of
    the FWSM.

    Solution Attempt: dynamic policy NAT

    We tried with dynamic policy NAT; outbound connections to NEW-APP-
    SERVERS on tcp/6500 should be NATted dynamically to a pool of 10
    source addresses, as we don't expect more than 10 simultaneous
    connections. Then, incoming connections to tcp/6500 on the NAT pool's
    IP addresses are allowed by OUTSIDE-ACL.

    The dynamic policy NAT part of the this setup works well. Outbound
    connections to NEW-APP-SERVERS:6500 are successfully source-NATted to
    the pool.

    The reverse connection does not work; we see OUTSIDE-ACL counting hits
    of incoming connections from the NEW-APP-SERVERS to the source NAT the
    client had been assiged before, but the SYNs don't make it to the
    clients and nothing is logged on the FWSM, neither with "warning" nor
    "informational" logging levels. It seems that a nat&global translation
    slot is a "one way street" from "nat" to "global"

    Excerpt from documentation
    The document "Catalyst 6500 Series Switch and Cisco 7600 Series Router
    Firewall Services Module Configuration Guide, Release 2.3" states on
    Page 9-4 in a "Note" box, in the dynamic NAT chapter:

    "For the duration of the translation, a global host can initiate a
    connection to the local host if an ACL allows it. Because the address
    is unpredictable, a connection to the host is unlikely. However in
    this case, you can rely on the security of the ACL"

    That's where we had the idea from: The first outbound connection
    creates the xlate, and we only need to open the ACL to allow incoming
    session initiations. However: no luck.

    This is how we configured the FWSM according to our idea:

    object-group NEW-APP-SERVERS
    network-object host
    network-object host
    network-object host
    network-object host
    object-group NEW-APP-CLIENTS-NAT
    network-object host
    network-object host
    network-object host
    network-object host
    network-object host
    network-object host
    network-object host
    network-object host
    network-object host
    network-object host
    access-list POLICYNAT remark ------------- Policy NAT access List
    access-list POLICYNAT extended permit tcp any object-group NEW-APP-
    SERVERS eq 6500 ...
    nat (inside) 31 access-list POLICYNAT
    global (outside) 31 netmask ...
    access-list OUTSIDE-ACL remark ------------- allow reverse incoming
    connections for NEW-APP
    access-list OUTSIDE-ACL permit tcp object-group NEW-APP-SERVERS object-
    group NEW-APP-CLIENTS-NAT eq 6500
    access-group OUTSIDE-ACL in interface outside ...

    Is there another configuration scheme that would allow to fix our
    problem? I tried sketching this up with policy static NAT or an
    outsite NAT, but so far I haven't been able to think of a setup that
    looks like it could work.

    Thanks for your comments on this

    marc.luethi, Nov 15, 2007
    1. Advertisements

  2. marc.luethi

    Sam Wilson Guest

    I think you will need an 'out' ACL on the inside interface to allow
    connections to be made in that direction. In fact I *think* (don't hold
    me to it) that you just need to move the ACL above to the inside

    access-group OUTSIDE-ACL out interface inside

    Sam Wilson, Nov 15, 2007
    1. Advertisements

  3. marc.luethi

    marc.luethi Guest

    Thank you for your comment.

    Tthat wouldn't actually change a lot, since the inside interface
    already has a "permit ip any any" ACL bound to it. I considered it so
    "normal" that I even forgot to mention it.

    While brooding over this problem this afternoon, I discovered that on
    the same firewall, we had been facing a very similar problem three
    years back. It's quite close, only that the reverse incoming
    connections are udp in this case, and that there's no PAT or no policy
    NAT going on, jus regular NAT. The application is an information
    terminal from one of the major financial data vendors that requires
    some UDP datagrams to flow back to the client software.

    We tried at first with PAT, and the APP would work, but we'd see drops
    of the incoming UDP messages to the PAT address (of course). So we
    switched over to a NAT pool and the problem disappeared. In the
    configuration, it kind of looks like this (bare with me if some
    details are wrong with the syntax, I am writing this from memory and
    it's getting late; I just want to show the general way we did it):

    object-group INFO-APP-SERVERS
    network-object host
    network-object host
    network-object host
    network-object host

    nat (inside) 1
    global (info-app-dmz) 1 netmask
    access-list INFO-APP-ACL remark ------------- allow reverse incoming
    connections for INFO-APP
    access-list INFO-APP-ACL permit udp object-group INFO-APP-SERVERS range 49030 49050
    access-group INFO-APP-ACL in interface info-app-dmz

    (the inside client for NEW-APP just happens to be a client for INFO-
    APP as well)

    On the other hand.. coming to think of it - while we were still
    running INFO-APP with PAT, we never had any compaints from the users
    that anything was missing or didn't work, we just saw the numerous
    drops of incoming UDP to the PAT address. After switching over to a
    NAT pool and opening the ACL, we didn't see any drops anymore. To be
    honest, we never verified that the udp messages actually _were_
    reaching the PCs on the inside and never got any reports that some
    feature now was working that didn't before... :-/

    That matches the current situation with NEW-APP: The NAT part is now
    working, and we see hits on the outside ACL of the incoming
    connections - It's just now that NEW-APP does not run if the reverse
    connection fails.

    Ah. this is going to be one great nut to crack!


    marc.luethi, Nov 15, 2007
  4. Marc,

    Ask yourself the following question:

    If a SYN packet from the app-server arrives on the outside interface,
    after the ACL passes it, what is the firewall going to do with it?
    Since it's not a well-known app, there is no way the firewall can
    associate it with an existing connection. With no static defined, the
    firewall doesn't know where to send it. The existing nat table entry for
    the outbound connection looks like

    PC/<random-source-port> --> Pool-ip/<random-source-port>

    This entry doesn't match what you need, which is

    PC/6500 --> Pool-ip/6500

    The firewall should drop the packet with Denied (no xlate)' message.
    This may be a debug-level message, which is why you aren't seeing it.

    AFAIK, the only ways to allow an outside-initiated connection are static
    nat, or nat 0 with an access list (nat exemption). If the partner can
    take your native addresses, the latter may be an option for you. If
    not, I'm out of ideas, but I think you can see you're asking the
    firewall to 'read minds' in this setup.


    Mike Flanigan
    Michael Flanigan, Nov 15, 2007
  5. marc.luethi

    Marc Luethi Guest

    I was hoping for the next obvious thing: match it to the existing xlate
    that was generated and used when the first outgoing connection was made.
    And if that doesn't work: at least say so. ;-)
    No need to associate to an existing _connection_: It is a _new_
    connection I want to allow. It should however associate it to an existing

    As it says in my quoted excerpt from the configuration guide: "For the
    duration of the translation, a global host can initiate a connection to
    the local host [...]".

    The fact that this global host also uses tcp/6500 to connect back to the
    local one is pure coincidence (see below for an example with different
    port numbers).
    Well, there is no static definition, since I wanted to avoid them in the
    first place. But there _is_ an xlate entry matching a local IP to a
    global one. According to "sh xlate int outside debug", this existing
    entry does not know of any port numbers.
    In the meantime, I increased the logging level to "debug", and the FWSM
    is still silent about what happens after the packet has positively
    matched the permit entry of the OUTSIDE-ACL.

    I rebuilt the whole setup with another interface and two hosts: A DNS
    server on the inside, from which I do an ssh session to a host in a
    (lower security level) DMZ. While connecting "downhill", the DNS server's
    connection is (policy-)source-NATted with addresses from a pool of DMZ
    addresses. The DMZ ACL allows udp/53 and tcp/22 to the addresses of the
    NAT pool.

    The result is the same. From the DNS server, I can login to the machine
    in the DMZ, it sees the connection coming from one of the pool addresses.
    So both policy-NAT-with-pool and the implicit security rules to allow
    return traffic work as intended.

    But I can neither get a DNS request nor a SSH session back to the DNS
    server. While FWSM does count hits on the ACL, it doesn't log anything
    about what happens to the packets after passing the ACL.
    I don't think I want it to read minds - I just want to be able to
    _initate_ a connection from the outside using an xlate that was defined
    by a pair of nat&global statements (and a previous outgoing connection
    that triggered it's creation) instead of a "static(..)" , that's
    basically all.

    If that isn't possible, because a "nat&global"-generated xlate entry is
    unidirectional from the "nat side" to the "global side" (which I presume
    is the cause for the problem, but would mean that the Configuration Guide
    has an error or is at least quite unclear), then I guess I'll have to
    bite the bullet and do static configurations.

    thanks for your comments in any case!

    best regards

    Marc Luethi, Nov 18, 2007
  6. Marc,

    Thank you for pointing out that the existing connection entry is not
    required, it all hinges on the xlates.

    I'm not sure why you aren't seeing port information in your show xlate,
    but in my experience, which is mostly on version 3, all FWSM xlates have
    been port-specific. If you read between the lines (often necessary with
    Cisco doc), I think you can see that the note you quote is really
    intended as a caveat that the firewall has been opened a little by the
    xlate, but only to the source port originally used by the inside host.
    It's not intended as a means of allowing outside connections, as I read

    From the same section of the Guide you quoted:


    Dynamic NAT translates a group of local addresses to a pool of global
    addresses that are routable on the destination network. The global pool
    can include fewer addresses than the local group. When a local host
    accesses the destination network, the FWSM assigns it an IP address from
    the global pool. Because the translation is only in place for the
    duration of the connection, a given user does not keep the same IP
    address after the translation times out (see the timeout xlate command
    in the Catalyst 6500 Series Switch and Cisco 7600 Series Router Firewall
    Services Module Command Reference). Users on the destination network,
    therefore, cannot reliably initiate a connection to a host that uses
    dynamic NAT (even if the connection is allowed by an access control list
    (ACL)). Not only can you not predict the global IP address of the host,
    but the FWSM does not create a translation at all unless the local host
    is the initiator. See "Static NAT" or "Static PAT" below for reliable
    access to hosts.

    Note For the duration of the translation, a global host can initiate a
    connection to the local host if an ACL allows it. Because the address is
    unpredictable, a connection to the host is unlikely. However in this
    case, you can rely on the security of the ACL.


    I think the key is:

    "but the FWSM does not create a translation at all unless the local
    host is the initiator. See "Static NAT" or "Static PAT" below for
    reliable access to hosts."

    Your policy acl specifies port 6500 on the inside, but the host hasn't
    connected out on that port, so no xlate exists for the SYNN packet from
    the outside.

    You may have a bug issue with the lack of logging, and the lack of port
    info in your displays, I don't know.

    Also in the same chapter, it is made fairly clear (for Cisco doc), that
    static and nat exemption are the only 'reliable' means of initiating
    connections from the outside. Sounds like policy nat was not intended
    for this purpose.

    Not to beat this to death, but I think static is your only option, as
    you indicate. Good luck.


    Mike Flanigan
    Michael Flanigan, Nov 19, 2007
  7. Marc,

    Hope it's not too late for you, but I think you can use the
    'established' command to do what you want.


    Mike Flanigan

    Michael Flanigan, Nov 22, 2007
  8. marc.luethi

    Marc Luethi Guest

    Thanks Michael for the input

    That sounded too good - I went and tried it with the test setup I did
    with the DNS server on the inside ssh'ing to a DMZ system (while being
    policy-Source-NATted to an address of the DMZ).

    So to allow the DMZ system to send DNS lookups back to the inside DNS
    server, I configured:

    established tcp 22 0 permitto udp 53 permitfrom udp 1024-65536

    The ACL still allows udp/53 from the DMZ host to the SouceNAT-Addresses,
    and "logging monitor debug" shows the positive match on the ACL, but
    still: no joy :-(

    In the meantime, we've been supporting NEW-APP with "static
    (inside,outside)" configurations to make it work; it went productive over
    the last weekend.

    On Friday last week, our partner company announced that they found a way
    to configure the NEW-APP-SERVERS so that they would no longer require the
    reverse connection to the client; it remains to be tested and to see if
    it actually works.

    That's just as well :-/ , we've been ranting about NEW-APP's pre-historic
    and firewall unfriendly network behaviour from the beginning - and now
    they come up with it.

    All the same, Thank you (all) very much for your help and support.

    Marc Luethi, Nov 26, 2007
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.