Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Ruby > (noob) need help parsing Apache log file

Reply
Thread Tools

(noob) need help parsing Apache log file

 
 
Koncept
Guest
Posts: n/a
 
      02-28-2004

Being very new to Ruby, I want to be able to write a small script to
open my Apache access_log file and remove any lines that contain (\.exe
| \.ida) from the source file. I managed to figure out how to get this
information and log it to a new file, but I would really appreciate an
example of how to read and remove from one file. I am the kind of
person who learns best watching others.


This is what I wrote to parse the file. Any revisions, hints, hacks,
etc would also be very much appreciated.

src = '/var/log/httpd/access_log'
arr = IO.readlines(src)
len = arr.length
lw = 73
msEvil = Array.new
div = '--'

arr.each do |line|
msEvil.push line if line =~ /(\.exe| \.ida)/
end

# Generate report
puts ('Report for: ' + Time.new.to_s).center(lw)
puts (div * 20).center(lw)
puts
puts 'Total lines:'.ljust(lw/2) + len.to_s.rjust(lw/2)
puts 'Total Window exploits:'.ljust(lw/2) +
msEvil.length.to_s.rjust(lw/2)

# log to text file
f = File.new("codeRed.txt", "w+")

msEvil.each do |line|
f.puts line
end

f.close

--
Koncept <<
"Contrary to popular belief, the most dangerous animal is not the lion or
tiger or even the elephant. The most dangerous animal is a shark riding
on an elephant, just trampling and eating everything they see." - Jack Handey
 
Reply With Quote
 
 
 
 
Mark Hubbart
Guest
Posts: n/a
 
      02-28-2004
On Feb 27, 2004, at 10:19 PM, Koncept wrote:

> Being very new to Ruby, I want to be able to write a small script to
> open my Apache access_log file and remove any lines that contain (\.exe
> | \.ida) from the source file. I managed to figure out how to get this
> information and log it to a new file, but I would really appreciate an
> example of how to read and remove from one file. I am the kind of
> person who learns best watching others.
>
>
> This is what I wrote to parse the file. Any revisions, hints, hacks,
> etc would also be very much appreciated.


One small change will write out the file. Instead of:

> arr.each do |line|
> msEvil.push line if line =~ /(\.exe| \.ida)/
> end


try:

# reject! deletes matched lines
arr.reject! do |line|
msEvil.push line if line =~ /(\.exe| \.ida)/
end
# truncate the file and write out
File.open(src, "w+"){|f| f.write arr.join }


Array#reject! does the same thing as Array#sort, except it pays
attention to the return value of the passed block. If the block returns
a true value, it deletes the item from the array. There is also the
non-destructive version, Array#reject (no bang), but reject! makes more
sense here.

File.open opens the file (surprise!). In this case, it truncates it and
opens it write only. If you give it a block, it passes the opened file
handle to the block, and automagically closes the file when it's done
You might want to use the same trick at the bottom:

> # log to text file
> f = File.new("codeRed.txt", "w+")
>
> msEvil.each do |line|
> f.puts line
> end
>
> f.close


could become:

# log to text file
File.open("codeRed.txt", "w+"){|f| f.write msEvil.join }


That's a handy Ruby idiom. Shortens code considerably

And finally, here's the results of running this script on my logs:

mark@imac% sudo mslogstripminer
Password:
Report for: Sat Feb 28 00:06:42 PST 2004
----------------------------------------

Total lines: 1670
Total Window exploits: 104
mark@imac%

And that's my box at home!!! absolutely disgusting.

--Mark



 
Reply With Quote
 
 
 
 
Koncept
Guest
Posts: n/a
 
      02-28-2004
In article <(E-Mail Removed)>, Mark Hubbart
<(E-Mail Removed)> wrote:

> mark@imac% sudo mslogstripminer
> Password:
> Report for: Sat Feb 28 00:06:42 PST 2004
> ----------------------------------------
>
> Total lines: 1670
> Total Window exploits: 104
> mark@imac%
>
> And that's my box at home!!! absolutely disgusting.


You think you have it bad... This is MY home box.

Report for: Sat Feb 28 03:53:05 EST 2004

----------------------------------------


Total lines: 31068
Total Window exploits: 637

Thanks for you help and tips. Everything made total sense and you gave
me a few pointers which will save tons of time. Appreciate your help.

Cheers

--
Koncept <<
"Contrary to popular belief, the most dangerous animal is not the lion or
tiger or even the elephant. The most dangerous animal is a shark riding
on an elephant, just trampling and eating everything they see." - Jack Handey
 
Reply With Quote
 
Koncept
Guest
Posts: n/a
 
      02-28-2004
Mark. One other thing....

If the script is run without *sudo* permissions, this is the error
thrown. How do I catch something like this and present a nicer
message to the user?

logReader.rb:29:in `initialize': Permission denied -
/var/log/httpd/access_log (Errno::EACCES)
from logReader.rb:29:in `open'
from logReader.rb:29

--
Koncept <<
"Contrary to popular belief, the most dangerous animal is not the lion or
tiger or even the elephant. The most dangerous animal is a shark riding
on an elephant, just trampling and eating everything they see." - Jack Handey
 
Reply With Quote
 
Mark Hubbart
Guest
Posts: n/a
 
      02-28-2004
On Feb 28, 2004, at 10:24 AM, Koncept wrote:

> Mark. One other thing....
>
> If the script is run without *sudo* permissions, this is the error
> thrown. How do I catch something like this and present a nicer
> message to the user?


You need to catch the errors. Enclose the error-generating code in a
begin..rescue..end statement:

begin
# error-prone code here
rescue
warn "You don't have permission to access the log. Try using sudo."
exit
end

--

Just replace the comment with the line (or lines) of code. You'll
probably want to do this with the code where you read in the file, and
also the two places where you write the data out. For much more
detailed information on error catching, refer to the PickAxe Guide[1],
"Exceptions, Catch, and Throw".

--Mark

[1]: http://phrogz.net/ProgrammingRuby/



 
Reply With Quote
 
Koncept
Guest
Posts: n/a
 
      02-28-2004
In article <(E-Mail Removed)>, Mark Hubbart
<(E-Mail Removed)> wrote:

> Just replace the comment with the line (or lines) of code. You'll
> probably want to do this with the code where you read in the file, and
> also the two places where you write the data out. For much more
> detailed information on error catching, refer to the PickAxe Guide[1],
> "Exceptions, Catch, and Throw".


Mark. Thanks for all you help.

It is so nice to be a part of a group ( unlike those Perl folks ) where
people are actually nice to each other.

you = "the man"

--
Koncept <<
"Contrary to popular belief, the most dangerous animal is not the lion or
tiger or even the elephant. The most dangerous animal is a shark riding
on an elephant, just trampling and eating everything they see." - Jack Handey
 
Reply With Quote
 
Mark Hubbart
Guest
Posts: n/a
 
      02-28-2004

On Feb 28, 2004, at 1:19 PM, Koncept wrote:

> It is so nice to be a part of a group ( unlike those Perl folks ) where
> people are actually nice to each other.


We're all nice here because programming in ruby is fun as for
Perl... I'd better not say anything.

--Mark



 
Reply With Quote
 
Eric Schwartz
Guest
Posts: n/a
 
      03-03-2004
Mark Hubbart <(E-Mail Removed)> writes:
> On Feb 28, 2004, at 1:19 PM, Koncept wrote:
>> It is so nice to be a part of a group ( unlike those Perl folks ) where
>> people are actually nice to each other.

>
> We're all nice here because programming in ruby is fun as for
> Perl... I'd better not say anything.


To be fair, unlike Ruby, Perl has *tons* of documentation distributed
with it that a poster could easily consult before posting a question;
where it's not fair to tell someone to just "Read the Pickaxe", since
that's not definitive or complete, it is fairly reasonable for a Perl
programmer to be told to read the standard documentation on her hard
drive before asking a question it could easily answer.

Hopefully, the hard work being put into the RDoc<->ri integration
effort will result in similarly high-quality documentation being
included with a default distribution of Ruby.

-=Eric
--
Come to think of it, there are already a million monkeys on a million
typewriters, and Usenet is NOTHING like Shakespeare.
-- Blair Houghton.
 
Reply With Quote
 
Mac
Guest
Posts: n/a
 
      03-03-2004
Eric Schwartz wrote:
> Mark Hubbart <(E-Mail Removed)> writes:
>
>>On Feb 28, 2004, at 1:19 PM, Koncept wrote:
>>
>>>It is so nice to be a part of a group ( unlike those Perl folks ) where
>>>people are actually nice to each other.

>>
>>We're all nice here because programming in ruby is fun as for
>>Perl... I'd better not say anything.

>
>
> To be fair, unlike Ruby, Perl has *tons* of documentation distributed
> with it that a poster could easily consult before posting a question;
> where it's not fair to tell someone to just "Read the Pickaxe", since
> that's not definitive or complete, it is fairly reasonable for a Perl
> programmer to be told to read the standard documentation on her hard
> drive before asking a question it could easily answer.
>
> Hopefully, the hard work being put into the RDoc<->ri integration
> effort will result in similarly high-quality documentation being
> included with a default distribution of Ruby.
>
> -=Eric


To be realy fair, as a perl newbie (and even now) I found/find the Perl
docs to be unstructured, disjointed, scattered within the bundle. It
reflects the history of the language, not a concise view of how the
language works and how to get things done.

It also reflects (im very ho) the sometimes hard choices that were made
to keep the language alive in the face of the requirements of it's users
without completely breaking with it's history.

As to reponding to us newbies/nubys, Elwood P. Dowd (the movie "Harvey")
said he recommended " pleasant". Even in the face of documentation.
Hopefully you're not suffering fools, you're helping people on an
intellectual journey.

Mac

 
Reply With Quote
 
Mark Hubbart
Guest
Posts: n/a
 
      03-03-2004

On Mar 2, 2004, at 4:49 PM, Eric Schwartz wrote:

> Mark Hubbart <(E-Mail Removed)> writes:
>> On Feb 28, 2004, at 1:19 PM, Koncept wrote:
>>> It is so nice to be a part of a group ( unlike those Perl folks )
>>> where
>>> people are actually nice to each other.

>>
>> We're all nice here because programming in ruby is fun as for
>> Perl... I'd better not say anything.

>
> To be fair, unlike Ruby, Perl has *tons* of documentation distributed
> with it that a poster could easily consult before posting a question;
> where it's not fair to tell someone to just "Read the Pickaxe", since
> that's not definitive or complete, it is fairly reasonable for a Perl
> programmer to be told to read the standard documentation on her hard
> drive before asking a question it could easily answer.
>
> Hopefully, the hard work being put into the RDoc<->ri integration
> effort will result in similarly high-quality documentation being
> included with a default distribution of Ruby.


what I meant was:
We're all nice here because programming in ruby is fun. As for
Perl... I'd better not say anything.

I think that's clearer

Regardless, even if Ruby WAS superbly documented, I still think newbies
should be responded to in the same way: a quick example, and a
reference to more information. When you are new to a programming
language, sometimes you only know what the result of your code *should*
be, and have no way of knowing how to go about actually *getting* it.
So even if you did look in the docs, you might not know what to look
for.

Simple questions from these "extreme newbies", IMHO, should be left for
the more advanced newbies. If they were treated the same way, they will
probably be happy to answer them, sort of giving back to the group.

So everyone should feel it's okay to jump in; ask a question, give a
response if you think it's correct, correct someone else if they are
wrong, show another way of doing it. Politely. With this kind of open
communication, people can say things without fear of getting
backhanded. Good temperaments in a community can be a big draw, and it
can be easily evidenced by interactions in the newsgroup

heh. that sounded really cheezy, but you know what I mean.

Just my 2

--Mark



 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
ruby + apache. need to export path=/opt/ruby/bin every time beforestarting apache ad Ruby 3 12-29-2009 06:40 AM
On my web site need the capability to log in as administrator and for others to log in anonymously AAaron123 ASP .Net 1 10-14-2008 08:14 PM
Parsing an apache access log line Joe Nciri Ruby 5 07-16-2007 02:27 PM
Parsing apache log files Jim Richardson Python 9 02-22-2004 06:54 AM
Need help on the Permissions needed to log to Event Log from ASP.NET? Henrik_the_boss ASP .Net 0 11-05-2003 10:14 AM



Advertisments