PIX 515E CPU util at 98% with a compromised system on inside network

Discussion in 'Cisco' started by shahidsheikh....com, Dec 30, 2005.

  1. Hi all,

    I am looking for recommendation on how to prevent a repeat of a recent
    episode on one of the networks I support.

    Pix 515E running 6.3(3) unrestricted. 3 inside networks. DMZ1, DMZ2,
    and Inside. All inside networks use a 172.16.x.x ip address space. The
    physical medium for all inside networks is switched 100 Mbps ethernet.
    Outside connects to a dual bonded T1 via a cisco 2600.

    Inside has the highest security level. Then DMZ1 and DMZ2.

    An old linux web server on DMZ2 not under our direct control (and
    probably never will be because of political reasons) gets compromised
    thru a php hack and causes a syn flood. That results in the Pix with
    98% CPU util and it stops responding to requests from other inside
    networks resulting in a DoS to the other inside networks. Packets were
    being dumped at the DMZ2 interface at 70~80 Mbps.

    The embryonic connection limit feature doesn't help here because it
    cannot be applied for outside NAT. And even if it did work it wouldn't
    help in this case because the embryonic connection limit does nothing
    to protect the Pix against high CPU util.

    What are my options if all I have is this one Pix to work with. Force
    the DMZ2 interface to 10baseT. That may help with the CPU utilization
    but what about my 3 Mbps WAN pipe? Any other way to do rate limiting?
    Is PIX os 7 the answer?

    Thanks,

    Shahid
    shahidsheikh....com, Dec 30, 2005
    #1
    1. Advertising

  2. shahidsheikh....com

    Mark Lar Guest

    Re: PIX 515E CPU util at 98% with a compromised system on insidenetwork

    shahidsheikh....com wrote:

    > I am looking for recommendation on how to prevent a repeat of a recent
    > episode on one of the networks I support.

    <snip tale of woe>
    Simple... turn the Linux box off.

    Seriously, if you've got a known vulnerable web server that nobody has
    the guts to fix, it's time to either turn it off or pack up and piss
    off, because you aren't being allowed to do your job.
    Mark Lar, Dec 30, 2005
    #2
    1. Advertising

  3. "shahidsheikh....com" <> wrote in message
    news:...
    > Hi all,
    >
    > I am looking for recommendation on how to prevent a repeat of a recent
    > episode on one of the networks I support.
    >
    > Pix 515E running 6.3(3) unrestricted. 3 inside networks. DMZ1, DMZ2,
    > and Inside. All inside networks use a 172.16.x.x ip address space. The
    > physical medium for all inside networks is switched 100 Mbps ethernet.
    > Outside connects to a dual bonded T1 via a cisco 2600.
    >
    > Inside has the highest security level. Then DMZ1 and DMZ2.
    >
    > An old linux web server on DMZ2 not under our direct control (and
    > probably never will be because of political reasons) gets compromised
    > thru a php hack and causes a syn flood. That results in the Pix with
    > 98% CPU util and it stops responding to requests from other inside
    > networks resulting in a DoS to the other inside networks. Packets were
    > being dumped at the DMZ2 interface at 70~80 Mbps.
    >
    > The embryonic connection limit feature doesn't help here because it
    > cannot be applied for outside NAT. And even if it did work it wouldn't
    > help in this case because the embryonic connection limit does nothing
    > to protect the Pix against high CPU util.
    >
    > What are my options if all I have is this one Pix to work with. Force
    > the DMZ2 interface to 10baseT. That may help with the CPU utilization
    > but what about my 3 Mbps WAN pipe? Any other way to do rate limiting?
    > Is PIX os 7 the answer?



    could it be that log-rates are killing you ?
    If so apply log rate limit

    And in general terms: Kill the server - it's compromised - You can not risk
    getting your own IP's getting abuse repported on the outside world, just
    because you have a shitty sys-admin
    What would you think your Management will say, if they got word that the
    company name was reportted together with hacking abuses ??

    HTH'
    Martin



    >
    > Thanks,
    >
    > Shahid
    >
    Martin Bilgrav, Dec 30, 2005
    #3
  4. shahidsheikh....com

    slim Guest

    Re: PIX 515E CPU util at 98% with a compromised system on insidenetwork

    Martin Bilgrav wrote:
    > "shahidsheikh....com" <> wrote in message
    > news:...
    >
    >>Hi all,
    >>
    >>I am looking for recommendation on how to prevent a repeat of a recent
    >>episode on one of the networks I support.
    >>
    >>Pix 515E running 6.3(3) unrestricted. 3 inside networks. DMZ1, DMZ2,
    >>and Inside. All inside networks use a 172.16.x.x ip address space. The
    >>physical medium for all inside networks is switched 100 Mbps ethernet.
    >>Outside connects to a dual bonded T1 via a cisco 2600.
    >>
    >>Inside has the highest security level. Then DMZ1 and DMZ2.
    >>
    >>An old linux web server on DMZ2 not under our direct control (and
    >>probably never will be because of political reasons) gets compromised
    >>thru a php hack and causes a syn flood. That results in the Pix with
    >>98% CPU util and it stops responding to requests from other inside
    >>networks resulting in a DoS to the other inside networks. Packets were
    >>being dumped at the DMZ2 interface at 70~80 Mbps.
    >>
    >>The embryonic connection limit feature doesn't help here because it
    >>cannot be applied for outside NAT. And even if it did work it wouldn't
    >>help in this case because the embryonic connection limit does nothing
    >>to protect the Pix against high CPU util.
    >>
    >>What are my options if all I have is this one Pix to work with. Force
    >>the DMZ2 interface to 10baseT. That may help with the CPU utilization
    >>but what about my 3 Mbps WAN pipe? Any other way to do rate limiting?
    >>Is PIX os 7 the answer?

    >
    >
    >
    > could it be that log-rates are killing you ?
    > If so apply log rate limit
    >
    > And in general terms: Kill the server - it's compromised - You can not risk
    > getting your own IP's getting abuse repported on the outside world, just
    > because you have a shitty sys-admin
    > What would you think your Management will say, if they got word that the
    > company name was reportted together with hacking abuses ??
    >
    > HTH'
    > Martin
    >
    >
    >
    >
    >>Thanks,
    >>
    >>Shahid
    >>

    >
    >
    >


    Another thought might be that you're exhausting your NAT translation
    entry limit. I had a host get infected with a worm and because it was
    spawning so many connections so fast, the logs indicated that additional
    NAT flows couldn't be built. This killed all legitimate use.

    If Walter Roberson chimes in, I'd be interested to hear his take on your
    and my experience. I don't know if there's much that can be done in this
    scenario. I would have thought that the PIX OS would provide some sort
    of SYN limiting that would prevent this from happening, so I'm wondering
    if I missed a nerd-knob to tweak.
    slim, Dec 30, 2005
    #4
  5. I thought about Logging. Disabled it completely and still get the same
    results when I blast one of the inside interfaces with a syn flood.

    As for the NAT limit being exhasted, in my case the three way handshake
    is never complete. The IRC server out on the internet that the exploit
    is trying to get to doesn't exist. So each connection is still in
    embryonic state. I don't think defining max connections on the static
    for these servers (I don't have control over) will help but I haven't
    tried it.

    I'm on the same page as you are looking for a way to limit/drop syn
    packets if they come at a pace faster than a defined threshold.

    Thanks,

    Shahid
    shahidsheikh....com, Dec 30, 2005
    #5
  6. shahidsheikh....com

    farmerc

    Joined:
    Nov 20, 2006
    Messages:
    2
    You might also want to check that there are no Debugs running on the box...

    I know its not what your looking at currently but ive seen worse.
    farmerc, Sep 21, 2007
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    4
    Views:
    3,754
  2. Speed3ple
    Replies:
    0
    Views:
    2,988
    Speed3ple
    Apr 4, 2006
  3. Jene Keller
    Replies:
    4
    Views:
    527
  4. =?Utf-8?B?VHJpbGl0aG9u?=

    2nd cpu on 100% Util

    =?Utf-8?B?VHJpbGl0aG9u?=, Jul 18, 2005, in forum: Windows 64bit
    Replies:
    3
    Views:
    343
    Joseph Conway [MSFT]
    Jul 22, 2005
  5. Sanal Kisi

    High CPU util on 3825

    Sanal Kisi, Nov 26, 2007, in forum: Cisco
    Replies:
    9
    Views:
    944
    £ukasz Bromirski
    Nov 29, 2007
Loading...

Share This Page