Pix fail-over questions

Discussion in 'Cisco' started by J, Jun 16, 2006.

  1. J

    J Guest

    I have two questions regarding Pix fail-over.

    Are there any special considerations for upgrading the code on a Pix in
    fail-over mode? I have 2 515Es running 6.3(4) that I'd like to upgrade
    ideally to 7.0.4 at 6.3(5) at the least. I inherited these Pixs in
    their current fail-over state so I don't have the benefit of setting
    them up. I'm short on Pix fail-over experience. I'm assuming that
    I'll have to take the standby Pix offline manually (fail-over cable and
    inside interface), upgrade it, and then reconnect it while
    disconnecting the first Pix. I need to accomplish this with minimal
    downtime, ideally 0 downtime. A phone system resides behind these
    Pixs. Can anyone point me to any docs that they've found useful on Pix
    fail-over configurations?

    My second question deals with the details of how a pair of fail-over
    Pixs operate when one of their inside or outside interfaces goes down.
    Current the outside interface of each Pix is connected to a stack of
    3750s at one site (diffent 3750 for each Pix) and 2 independant 2950s
    at another site (one Pix in each switch, same vlan on both). Their
    inside interfaces wrap back around to those same switches and are in a
    different vlan. Horrible design and I'm working on fixing that. I
    need to do code upgrades on the switches at both sites. If I take down
    the switch connected to the standby Pix I expect the active Pix to
    continue operating like normal. What happens if I take down the switch
    connected to the active Pix? I would expect fail-over to immediately
    promote the standby Pix and to continue passing traffic. Exactly how
    long would this take? If it's more than a couple seconds and the
    telephone switches will throw out errors. I've been reading up on
    fail-over but I'd like some real-world opinions on times and
    procedures.

    Thanks
    J
     
    J, Jun 16, 2006
    #1
    1. Advertising

  2. You may wish to investigate the Cisco Press Article.

    Cisco PIX: Failover Demystified

    http://www.ciscopress.com/articles/article.asp?p=24686&aid=dcb9cea5-50c2-44d3-af02-9ab5cc199d74

    A pair of identical PIX Firewall devices must be used for failover to
    function.

    To determine the system requirements and the steps needed to configure
    failover with a failover cable or LAN-based failover, refer to:

    Using PIX Firewall Failover for PIX operating on code 6.x and earlier

    http://www.cisco.com/en/US/products...figuration_guide_chapter09186a00800eb72f.html

    For PIX operating on code 7.x and later, refer to:

    Configuring Failover

    http://www.cisco.com/en/US/products/ps6120/products_configuration_guide_chapter09186a008045247e.html

    For general information on failover, refer to:

    How Failover Works on the Cisco Secure PIX Firewall

    http://www.cisco.com/en/US/products/hw/vpndevc/ps2030/products_tech_note09186a0080094ea7.shtml

    -----------------------------------------------

    How to replace the primary PIX Firewall in a failover environment PIX
    version 6.3 and earlier in Models PIX 515 - 535.

    It is important that the secondary unit is running a current version of
    the configuration that exists on the primary PIX Firewall. Issue the
    following command on the primary, if possible, to copy the current
    configuration to the standby unit:

    pixfirewall# configure terminal
    pixfirewall (config t)# copy standby

    The configuration will be copied over to the standby unit. Once this is
    completed, remove the primary PIX from the setup by shutting down the
    primary unit. Make sure that the secondary PIX has taken over before
    physically removing the primary PIX. This can done by issuing the
    following command on the secondary unit:

    secondarypix# configure terminal

    secondarypix (config t)# show failover

    The output should look like the following:

    pixfirewall (config)# show failover

    Failover On

    Cable status:Normal

    Reconnect timeout 0:00:00

    Poll frequency 15 seconds

    failover replication http

    This host:Secondary - Active

    Active time:30 (sec)

    Interface FailLink (172.16.31.2):Normal

    Interface 4th (172.16.16.1):Normal

    Interface int5 (192.168.168.1):Normal

    Interface intf2 (192.168.1.1):Normal

    Interface outside (209.165.200.225):Normal

    Interface inside (10.1.1.4):Normal

    At this point, the secondary PIX should show up as active in the above
    command output. Now it is possible to safely remove the primary PIX
    physically.

    Note: If the secondary unit is not active at this point,
    troubleshooting will be necessary.

    Once the old primary unit has been removed, you will need to physically
    install the new primary unit. The hardware and software must match the
    secondary unit. Be sure that the new primary PIX is running the same
    PIX operating system version and has the same hardware configuration as
    the secondary. If these requirement are not met, the new primary PIX
    will not perform properly in a failover environment.

    Once the new primary PIX has been attached properly and is running, you
    will need the existing configuration on the new unit. This can be done
    through TFTP or entered manually. The entire configuration is not
    necessary at this time, but the interfaces must be set up properly and
    failover commands must be in place.

    Issue the show failover command again on the secondary unit.

    The above information from the previous show failover command will
    appear along with the following:

    Other host: Secondary - Standby

    Active time:0 (sec)

    Interface FailLink (172.16.31.2) : Normal

    This will verify that the primary PIX is properly attached using
    failover.

    Issue the copy standby command on the secondary unit. This will copy
    the full configuration from the secondary PIX to the new primary PIX.

    Issue the failover active command on the primary PIX so that it will
    resume the active role for the failover connection.

    Issue the show active command on the new primary to make sure that it
    is identified as the primary failover device.

    On the new primary PIX, issue the write memory command to save the
    configuration to memory.

    -----------------------------------------------

    How to replace the secondary PIX Firewall in a failover environment PIX
    version 6.3 and earlier in Models PIX 515 - 535.

    Verify that the primary PIX Firewall is actually the active PIX for
    failover, by issuing the following command on the primary:

    pixfirewall# configure terminal
    pixfirewall (config t)# show failover

    The output should look like the following:

    Failover On

    Cable status:Normal

    Reconnect timeout 0:00:00

    Poll frequency 15 seconds

    failover replication http

    This host:primary - Active

    Active time:30 (sec)

    Interface FailLink (172.16.31.2):Normal

    Interface 4th (172.16.16.1):Normal

    Interface int5 (192.168.168.1):Normal

    Interface intf2 (192.168.1.1):Normal

    Interface outside (209.165.200.225):Normal

    Interface inside (10.1.1.4):Normal

    Other host: Secondary - Standby

    Active time:0 (sec)

    Interface FailLink (172.16.31.1) : Normal

    After you have confirmed that the primary PIX is actually the active
    PIX, remove the secondary PIX from the setup by shutting down the
    secondary unit.

    Make sure that the primary PIX is still active before physically
    removing the secondary PIX by issuing the show failover command again.

    Once the old secondary unit has been removed, you will need to
    physically install the new secondary unit. The hardware and software
    for the new secondary PIX must match the primary unit. If these
    requirement are not met, the new primary PIX will not perform properly
    in a failover environment.

    Once the new secondary PIX has been attached properly and is running,
    you will need the existing configuration on the new unit. This can be
    done through TFTP or entered manually.

    The entire configuration is not necessary at this time, but the
    interfaces must be set up properly and failover commands must be in
    place.

    Issue the show failover command again on the secondary unit.

    The above information from the previous show failover command will
    appear along with the following:

    Other host: Secondary - Standby

    Active time:0 (sec)

    Interface FailLink (172.16.31.1) : Normal

    This will verify that the secondary PIX is properly attached using
    failover.

    Issue the copy standby command on the primary unit.

    This will copy the full configuration from the primary PIX to the new
    secondary PIX.

    Sincerely,

    Brad Reese
    BradReese.Com - Cisco Network Engineer Directory
    http://www.bradreese.com/network-engineer-directory.htm
    1293 Hendersonville Road, Suite 17
    Asheville, North Carolina USA 28803
    USA & Canada: 877-549-2680
    International: 828-277-7272
    Fax: 775-254-3558
    AIM: R2MGrant
    Website: http://www.bradreese.com/contact-us.htm
     
    www.BradReese.Com, Jun 16, 2006
    #2
    1. Advertising

  3. J

    J Guest

    www.BradReese.Com wrote:
    > You may wish to investigate the Cisco Press Article.
    >
    > Cisco PIX: Failover Demystified


    Thanks for the info, Brad. I found a few of the Cisco.com articles
    after posting my message. However that doesn't answer my first
    question. In fact it make it more confusing. The docs state that the
    hardware that you insert into a failover situation (after removing it
    due to failure or whatever) has to exactly match the current active
    Pix. If that's the case then how do you ever upgrade the code or RAM
    on the cluster of Pixs? If I remove a Pix 515e w/ a VAC+, 128MB of RAM
    and 6.3(4) I'll have to replace it with an exact model according to the
    docs. I can't even upgrade it to 6.3(5).

    The way I see it, the only way to upgrade the code is to first break
    failover on the standy. Then upgrade it accordingly. Once it's ready
    to go I'll have to schedule downtime, first to physically disconnect
    the active Pix and then to physically connect the interfaces on the
    upgraded Pix (or shut and unshut interfaces appropriately). Then I can
    upgrade the former active Pix and put it back online. Finally I'll
    have to re-enable failover on everything.

    This would definitely cause downtime due to the state table being lost
    and for when I physically move cables. Isn't there a better to
    accomplish this? The docs that I've read make no mention of upgrading
    a pair of Pixs in failover mode. I do see that v7 adds a plethora of
    useful failover features though.

    Thanks
    J
     
    J, Jun 16, 2006
    #3
  4. J wrote:

    > www.BradReese.Com wrote:
    >> You may wish to investigate the Cisco Press Article.
    >>
    >> Cisco PIX: Failover Demystified

    >
    > Thanks for the info, Brad. I found a few of the Cisco.com articles
    > after posting my message. However that doesn't answer my first
    > question. In fact it make it more confusing. The docs state that the
    > hardware that you insert into a failover situation (after removing it
    > due to failure or whatever) has to exactly match the current active
    > Pix. If that's the case then how do you ever upgrade the code or RAM
    > on the cluster of Pixs? If I remove a Pix 515e w/ a VAC+, 128MB of RAM
    > and 6.3(4) I'll have to replace it with an exact model according to the
    > docs. I can't even upgrade it to 6.3(5).
    >
    > The way I see it, the only way to upgrade the code is to first break
    > failover on the standy. Then upgrade it accordingly. Once it's ready
    > to go I'll have to schedule downtime, first to physically disconnect
    > the active Pix and then to physically connect the interfaces on the
    > upgraded Pix (or shut and unshut interfaces appropriately). Then I can
    > upgrade the former active Pix and put it back online. Finally I'll
    > have to re-enable failover on everything.
    >
    > This would definitely cause downtime due to the state table being lost
    > and for when I physically move cables. Isn't there a better to
    > accomplish this? The docs that I've read make no mention of upgrading
    > a pair of Pixs in failover mode. I do see that v7 adds a plethora of
    > useful failover features though.
    >
    > Thanks
    > J


    That is more or less the case. Do a search of this group for articles on
    failover pix upgrade. You will find a variety of opinions and approaches,
    ranging from "just do it" (and pray) to "paranoid conservative" (from me).

    Bottom line is that you WILL suffer some down time, but with adequate
    planning you can reduce the downtime to seconds although you will drop all
    translations and VPNs at least once. You will also be running for a period
    of time with only one PIX, so you will be exposed to the potential for
    significant downtime if it chooses that instant to fail.

    Good luck and have fun!
    --
    Vincent C Jones, Consultant Expert advice and a helping hand
    Networking Unlimited, Inc. for those who want to manage and
    Tenafly, NJ Phone: 201 568-7810 control their networking destiny
    http://www.networkingunlimited.com
     
    Vincent C Jones, Jun 17, 2006
    #4
  5. J

    J Guest

    www.BradReese.Com wrote:
    > You may wish to investigate the Cisco Press Article.
    >
    > Cisco PIX: Failover Demystified


    Thanks for the info, Brad. I found a few of the Cisco.com articles
    after posting my message. However that doesn't answer my first
    question. In fact it make it more confusing. The docs state that the
    hardware that you insert into a failover situation (after removing it
    due to failure or whatever) has to exactly match the current active
    Pix. If that's the case then how do you ever upgrade the code or RAM
    on the cluster of Pixs? If I remove a Pix 515e w/ a VAC , 128MB of RAM
    and 6.3(4) I'll have to replace it with an exact model according to the
    docs. I can't even upgrade it to 6.3(5).

    The way I see it, the only way to upgrade the code is to first break
    failover on the standy. Then upgrade it accordingly. Once it's ready
    to go I'll have to schedule downtime, first to physically disconnect
    the active Pix and then to physically connect the interfaces on the
    upgraded Pix (or shut and unshut interfaces appropriately). Then I can
    upgrade the former active Pix and put it back online. Finally I'll
    have to re-enable failover on everything.

    This would definitely cause downtime due to the state table being lost
    and for when I physically move cables. Isn't there a better to
    accomplish this? The docs that I've read make no mention of upgrading
    a pair of Pixs in failover mode. I do see that v7 adds a plethora of
    useful failover features though.

    Thanks
    J
     
    J, Jun 17, 2006
    #5
  6. Excellent that Dr. Vincent C. Jones, PE

    http://www.bradreese.com/vincent-c-jones.htm

    Is contributing to this thread as Vince Jones is the author of the
    revered Addison Wesley Book:

    High Availability Networking with Cisco

    http://www.awprofessional.com/books...aid=dcb9cea5-50c2-44d3-af02-9ab5cc199d74&rl=1

    and the Addison Wesley article:

    Configuration for Transparently Redundant Firewalls

    http://www.awprofessional.com/articles/article.asp?p=23407&aid=dcb9cea5-50c2-44d3-af02-9ab5cc199d74

    Which was adapted from Vince's book, High Availability Networking with
    Cisco.

    --------------------------------------------------------

    How to upgrade the PIX Firewall software in a failover scenario.

    The PIXes must be running version 5.1(x) or later to use this procedure
    because it uses the copy tftp flash command.

    http://www.cisco.com/en/US/products...erence_chapter09186a00801727a6.html#wp1026589

    It was introduced in PIX 5.1(x).

    --------------------------------------------------------

    To upgrade PIXes in a failover set by establishing a console connection
    to the PIXes, perform these steps:

    Step 1: Force a failover to the secondary PIX by issuing the

    no failover active

    http://www.cisco.com/en/US/products...erence_chapter09186a00801727a8.html#wp1029143

    command on the primary PIX, or power off the primary PIX.

    This causes the secondary PIX to become active.

    Step 2: Disconnect all network cables from the primary PIX ( including
    the failover cable ).

    Step 3: Power on the primary PIX and attach a PC with a TFTP server on
    it. Connect the inside interface of the primary PIX to the TFTP server
    with a crossover cable.

    Step 4: Issue the

    copy tftp flash

    http://www.cisco.com/en/US/products...erence_chapter09186a00801727a6.html#wp1026589

    to upgrade the primary.

    Perform the upgrade procedures for the primary PIX as given in:

    Upgrading Software for the Cisco Secure PIX Firewall

    http://www.cisco.com/en/US/products/hw/vpndevc/ps2030/products_tech_note09186a0080094a5d.shtml

    Step 5: Reload the primary PIX and verify the new version, license keys
    and features, configuration and so on.

    Step 6: Power off the primary PIX.

    Step 7: Re-connect all cables to the primary PIX.

    Step 8: Quickly power off the secondary PIX, then immediately power on
    primary PIX.

    Verify that the primary PIX is now passing traffic running the new
    version of code.

    Note: Your downtime occurs while the primary is booting.

    Step 9: Once the primary PIX is up, it becomes active and passes
    traffic.

    Step 10: Repeat steps two through seven for the secondary PIX.

    Step 11: Power on the secondary PIX. It becomes the standby unit.

    Wait two minutes and verify that the secondary PIX is in standby mode
    and that all interfaces have a status of Normal.

    Both PIXes are now running the upgraded version and back to normal
    operation.

    --------------------------------------------------------

    To upgrade PIXes in a failover set through Telnet and/or Secure Shell (
    SSH ) session, perform these steps.

    Step 1: Open separate remote Telnet sessions to both the primary and
    seconday PIXes.

    Step 2: Perform a show failover to ensure that the primary PIX is the
    active PIX.

    Note: If it is not, issue the failover active command from the show
    primary PIX to make it so.

    Step 3: Enter config t mode on both PIXes.

    Step 4: For recovery purposes, in case something goes wrong, copy the
    running config command from the primary PIX to a text file, or issue
    the

    write net

    http://www.cisco.com/en/US/products...erence_chapter09186a00801727ae.html#wp1027782

    command from the primary PIX to save the config to the TFTP server.

    Step 5: Issue the write standby command, followed by the write memory
    command on the primary PIX to insure that the secondary PIX is
    up-to-date, in case something happens to the primary PIX .

    Step 6: Issue the copy tftp flash command on the primary PIX and wait
    until it downloads the new PIX image from the TFTP server successfully.

    Step 7: If the step 6 download is successful on the primary PIX, repeat
    step six on the secondary PIX.

    Note: At this point, a new PIX image has been downloaded to both the
    primary and secondary PIX.

    Step 8: On the primary PIX, press the Reset button on the front of the
    PIX.

    An alternative is turn off the power and turn it back on.

    Step 9: Wait five seconds, then press the Reset button on the front of
    the seconday PIX, or turn off the power and turn it back on.

    Step 10: Wait about one minute for both PIXes to complete their resets,
    then perform a separate Telnet to both primary and secondary PIXs.

    Step 11: Verify through the

    show version

    http://www.cisco.com/en/US/products...erence_chapter09186a00805fd881.html#wp1536220

    command output on both PIXes that the new PIX image has been installed
    successfully.

    Step 12: Verify through the show failover command output on the primary
    PIX that it is active, and verify that the secondary is in standby
    mode.

    Both PIXes are now running the upgraded version and back to normal
    operation.

    --------------------------------------------------------

    To upgrade PIXes in a failover set remotely, initiate two Telnet ( or
    SSH ) sessions to the PIXes.

    Initiate one session to the primary and one session to the secondary.

    Replace steps eight and nine ( see above ) with these steps:

    Step 1: On the secondary ( standby ) PIX, type reload to reboot the
    PIX.

    Step 2: On the primary ( active ) PIX, type reload to reboot the PIX.

    Note: The primary must be reloaded before the secondary comes back on
    line.

    This gives you only a few seconds to perform these steps.

    Step 3: Wait about one minute for both PIXes to complete their resets,
    then initiate another Telnet ( or SSH ) session to both PIXes.

    Step 4: Verify through the show version command output that both PIXes
    are running the new image.

    Step 5: Verify through the show failover command output that the
    secondary is active.

    Step 6: On the primary PIX, issue the failover active command to force
    it to become the active PIX.

    Step 7: Both PIXes are now running the upgraded image and back to
    normal operation.

    When upgrading remotely, the secondary comes up first, so it is active
    when this process is complete.

    You issue the failover active command from the primary to force it to
    be active.

    Note: Downtime occurs when both PIXes are powered off and as the
    primary PIX boots up.

    This downtime is necessary because the PIXes cannot communicate to one
    another on different versions of code.

    --------------------------------------------------------

    PIX version 7.0 introduced the ability to perform software upgrades of
    failover pairs without impacting network uptime or connections flowing
    through the units.

    Version 7.0 introduced the ability to do inter-version state sharing
    between security appliance failover pairs, allowing you to perform
    software upgrades to maintenance releases for example, Version 7.0(1)
    upgrading to 7.0(2)) without impacting traffic flowing through the
    pair.

    In active or standby failover environments or active/active
    environments where the pair is not oversubscribed, more that 50 percent
    load on each pair member.

    --------------------------------------------------------

    To perform the upgrade without impacting connections, back up current
    configurations and perform these steps:

    Step 1: Copy the image files to the device, and select the image to
    boot by issuing the

    boot system

    http://www.cisco.com/en/US/products...erence_chapter09186a008017d030.html#wp1030929

    command.

    Step 2: Reload the standby unit, which causes it to boot the new image.

    Step 3: When the standby unit is in standby_ready state, force a
    failover so it becomes active.

    Re-load the new standby unit. Now both units are running the new image.

    For more information refer to:

    Configuring Failover

    http://www.cisco.com/en/US/products/ps6120/products_configuration_guide_chapter09186a008045247e.html

    For the process to work smoothly, it is better to reload the standby
    unit in the procedure instead of the active unit.

    In a situation where the units are Active/Active, and you do not want
    to impact connections on the secondary active unit, you must change the
    failover groups on the unit to achieve the same active/standby state
    before performing the steps.

    If it is necessary to upgrade to a later version of software, you must
    upgrade to the next version for the upgrade to avoid impacting network
    uptime.

    For example, if you need to upgrade from version 7.0(1) to version
    7.0(3), you must upgrade from version 7.0(1) to version 7.0(2), then to
    version 7.0(3).

    For detailed information visit:

    Performing Zero Downtime Upgrades for Failover Pairs

    http://www.cisco.com/en/US/products..._guide_chapter09186a0080450b92.html#wp1053398

    Hope this helps.

    Brad Reese
    Cisco Repair
    http://www.bradreese.com/cisco-big-iron-repair.htm
     
    www.BradReese.Com, Jun 17, 2006
    #6
  7. J

    J Guest

    www.BradReese.Com wrote:
    > Excellent that Dr. Vincent C. Jones, PE
    >
    > http://www.bradreese.com/vincent-c-jones.htm
    >
    > Is contributing to this thread as Vince Jones is the author of the
    > revered Addison Wesley Book:
    >
    > High Availability Networking with Cisco
    >
    > http://www.awprofessional.com/books...aid=dcb9cea5-50c2-44d3-af02-9ab5cc199d74&rl=1


    Brad,

    Excellent information. The first of Dr. Jones' books that you cited
    has been on my Amazon wishlist for a while now. I think it's time to
    go ahead and pick it up. I think I have what I need to formulate a
    solid MOP for the upgrade. I also think I have a strong case for
    justifying an upgrade to v7. That will prompt a separate question to
    the group though. This particular application demands 5 9s or better.
    If v7 offers a better way of meeting that requirement then I can
    justify the upgrade.

    Thanks again for the wealth of useful information.
    J
     
    J, Jun 18, 2006
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Guest

    PIX 515e Fail-over

    Guest, Apr 5, 2004, in forum: Cisco
    Replies:
    2
    Views:
    817
    Guest
    Apr 14, 2004
  2. Jens Bader

    fail-over ethernet uplink?!

    Jens Bader, Jul 21, 2004, in forum: Cisco
    Replies:
    4
    Views:
    1,905
    Hansang Bae
    Jul 22, 2004
  3. Replies:
    1
    Views:
    2,012
  4. Lee
    Replies:
    5
    Views:
    485
    www.networking-forum.com
    Sep 28, 2005
  5. Theo Markettos

    VOIP over VPN over TCP over WAP over 3G

    Theo Markettos, Feb 3, 2008, in forum: UK VOIP
    Replies:
    2
    Views:
    1,087
    Theo Markettos
    Feb 14, 2008
Loading...

Share This Page