apache swapping the system to death

Discussion in 'Linux Networking' started by unruh, May 22, 2012.

  1. unruh

    unruh Guest

    I am running a web server on a Mandriva 2010.2 system. Today the system
    suddenly became increadibly non-responsive. When I finally logged on as
    root, the swap file was up around 3GB, (no wonder response was horrible)
    and there were 160 instances of httpd running. After I finally managed
    to shut them down, (killall -9 httpd) response was restored. I looked in
    the /var/log/httpd/access_log there did not seem to be much unusual
    there. There were some google.com with weird addreses, and some internal
    connections which were the only things that looked out of the oridinary.

    Eg
    66.249.68.198 - - [21/May/2012:18:19:23 -0700] "GET /aggregator/www.umsl.edu/~keelr/010/www.twitter.com/www.iaea.org/Publications/Documents/Board/2008/bit.ly/www.guardian.co.uk/business/2012/may/04/pay-vince-cable HTTP/1.1" 200 72658 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.68.73 - - [21/May/2012:18:19:26 -0700] "GET /aggregator/www.nytimes.com/2012/04/05/opinion/node/node/www.bbc.co.uk/news/uk-17769717?page=226 HTTP/1.1" 200 38984 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

    and

    ::1 - - [21/May/2012:17:32:03 -0700] "OPTIONS * HTTP/1.0" 200 - "-"
    "Apache/2.2.15 (Mandriva Linux/PREFORK-3.2mdv2010.2) (internal dummy
    connection)"
    ::1 - - [21/May/2012:17:32:03 -0700] "OPTIONS * HTTP/1.0" 200 - "-"
    "Apache/2.2.15 (Mandriva Linux/PREFORK-3.2mdv2010.2) (internal dummy
    connection)"
    ::1 - - [21/May/2012:17:32:05 -0700] "OPTIONS * HTTP/1.0" 200 - "-"
    "Apache/2.2.15 (Mandriva Linux/PREFORK-3.2mdv2010.2) (internal dummy
    connection)"
    ::1 - - [21/May/2012:17:32:07 -0700] "OPTIONS * HTTP/1.0" 200 - "-"
    "Apache/2.2.15 (Mandriva Linux/PREFORK-3.2mdv2010.2) (internal dummy
    connection)"

    .....


    But could any of these be respoinsible for 160 connections?

    The other suspicious thing is that there was 6 hour gap in
    the logs

    112.111.174.175 - - [21/May/2012:11:34:11 -0700] "GET /user/register
    HTTP/1.0" 200 29860 "http://emergentgravity.org/user/register"
    "Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) Opera 6.01 [en]"
    ::1 - - [21/May/2012:17:32:03 -0700] "OPTIONS * HTTP/1.0" 200 - "-"
    "Apache/2.2.15 (Mandriva Linux/PREFORK-3.2mdv2010.2) (internal dummy
    connection)"
     
    unruh, May 22, 2012
    #1
    1. Advertisements

  2. unruh

    J.O. Aho Guest

    Tweak your apache settings, so that no 160 instances will be started,
    with some google you should be able to find some setting recommendation
    formulas.

    I think I had some issue with an earlier wordpress version (pre 3.3),
    which got really strange, specially when it tried to access a page which
    didn't exist and it came into a horrible loop (yes, wordpress may
    request pages itself).
    This is all okey, nothing to worry about, if you don't want to see them,
    there is an instruction how to filter those at apache.org.
    Has nothing to do with the increased number of services loaded.

    That don't sound good at all, could have been caused by the swapped out
    syslog, but I would think it's wise to take a look at your system with
    rkhunter and/or chkrootkit. Also check your other logs and see if you
    miss 6 hours.
     
    J.O. Aho, May 22, 2012
    #2
    1. Advertisements

  3. If you are running apache versions from 2010, this sounds suspiciously
    like you have been a victim of

    http://httpd.apache.org/security/CVE-2011-3192.txt

    It would be best to upgrade to newer binaries then.

    Regards,
    Wolfram.
     
    Wolfram Gloger, May 22, 2012
    #3
  4. unruh

    unruh Guest

    Thanks. I had partially upgraded to 2.2.22, but not all of the installed
    apache stuff had been upgraded. I have now done so and will watch to see
    if I get further problems. So far so good.

    It seems that the attack vector is via those weird google searches. Ie,
    they also seem to be spoofing google bot requests as part of the attack.
     
    unruh, May 22, 2012
    #4
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.