Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Java > analysis of java application logs

Reply
Thread Tools

analysis of java application logs

 
 
Robert Klemme
Guest
Posts: n/a
 
      05-23-2011
On 23.05.2011 15:17, Patricia Shanahan wrote:
> On 5/23/2011 12:50 AM, Ulrich Scholz wrote:
>> I'm looking for an approach to the problem of analyzing application
>> log files.
>>
>> I need to analyse Java log files from applications (i.e., not logs of
>> web servers). These logs contain Java exceptions, thread dumps, and
>> free-form log4j messages issued by log statements inserted by
>> programmers during development. Right now, these man-made log entries
>> do not have any specific format.
>>
>> What I'm looking for is a tool and/or strategy that supports in lexing/
>> parsing, tagging, and analysing the log entries. Because there is only
>> little defined syntax and grammar - and because you might not know
>> what you are looking for - the task requires the quick issuing of
>> queries against the log data base. Some sort of visualization would be
>> nice, too.
>>
>> Pointers to existing tools and approaches as well as appropriate tools/
>> algorithms to develop the required system would be welcome.

>
> I would use Perl, and begin by recognizing some of the more important
> formats, such as thread dumps. I agree with the desirability of
> introducing some organized formatting into the log messages, but an
> ad-hoc Perl program can often get useful data out of a disorganized log.


Only that Perl is so awful - YMMV of course. But for these kinds of
tasks (more correctly: for *any* task) I very much prefer to use Ruby
because of its cleaner OO and cleaner syntax. In these cases where the
basic format is fixed I place general parsing code in a library (a
single file really) and then I can write ad hoc scripts which do
arbitrary processing of the data. That's very productive.

Kind regards

robert

--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/
 
Reply With Quote
 
 
 
 
Daniele Futtorovic
Guest
Posts: n/a
 
      05-23-2011
On 23/05/2011 20:27, Robert Klemme allegedly wrote:
> On 23.05.2011 19:06, Daniele Futtorovic wrote:
>> On 23/05/2011 18:20, Lew allegedly wrote:
>>> jlp wrote:
>>>> I agree with you, Lee, it is what i [sic] did with my own tool. Native
>>>> logs are
>>>
>>> Who's Lee?
>>>

>>
>> You're Lee now, Lee.

>
> Did you mean to say "Bruce"?
>


Wait. I thought Lee was a Yank, not an Aussie?
 
Reply With Quote
 
 
 
 
Lew
Guest
Posts: n/a
 
      05-23-2011
On 05/23/2011 01:16 PM, Daniele Futtorovic wrote:
> On 23/05/2011 15:11, Lew allegedly wrote:
>> Ulrich Scholz wrote:
>>> I'm looking for an approach to the problem of analyzing application
>>> log files.
>>>
>>> I need to analyse Java log files from applications (i.e., not logs of
>>> web servers). These logs contain Java exceptions, thread dumps, and
>>> free-form log4j messages issued by log statements inserted by
>>> programmers during development. Right now, these man-made log entries
>>> do not have any specific format.
>>>
>>> What I'm looking for is a tool and/or strategy that supports in lexing/
>>> parsing, tagging, and analysing the log entries. Because there is only
>>> little defined syntax and grammar - and because you might not know
>>> what you are looking for - the task requires the quick issuing of
>>> queries against the log data base. Some sort of visualization would be
>>> nice, too.
>>>
>>> Pointers to existing tools and approaches as well as appropriate tools/
>>> algorithms to develop the required system would be welcome.

>>
>> It helps if you have a logging strategy that mandates a consistent
>> logging format, specific information in particular positions or marked
>> by particular markup, logging levels and other such so that your
>> analysis tool isn't faced with a completely open-ended input. What you
>> describe requires a general text-analysis approach, as you indicate that
>> you can make no guarantees about the format. Based on that, your best
>> tool is "less" or equivalent text-file reader.
>>
>> What is a tool supposed to do, read your mind?
>>
>> It's really hard to extract information from a garbage can where people
>> just randomly dumped whatever they individually felt like dumping
>> without regard for operational needs. You can't build a skyscraper on a
>> bad foundation, and you can't build a good log analysis off a crappy log.
>>
>> Fix the logging system, then the analysis problem will be tractable.
>>

>
> I would argue around the same lines.
>
> I've been faced a while ago with a situation where some orthogonal
> organisational unit wanted to exploit my logs. I told them to GTFO.
>
> My logs are my logs. I put in it what I consider necessary. I often
> improve them as I step through the code. I might change the message, fix
> the level, &c. I don't want to have them set in stone. Neither do I
> generally have enough confidence in them to allow them to be used for
> analysis.
>
> "The solution, then, is simple", I told them, "spec out the exact
> messages and arguments you want, and the exact situations you want them
> logged in, and I'll add them for you. But leave me my precious debugging
> logs."
>
> Let me emphasize: IMHO debugging logs and logs for analysis are two
> different things and should be kept strictly separated -- possibly
> logged to a different target respectively.


That last is rather a brilliant idea, to use different targets. Heretofore
I've espoused that logs are primarily an operations tool, not a debugging
tool, although in service of the former they inevitably and inherently must
support the former. The problem I've always seen is that logging statements
are left up to the programmer, and not specified for the project.

--
Lew
Honi soit qui mal y pense.
http://upload.wikimedia.org/wikipedi.../c/cf/Friz.jpg
 
Reply With Quote
 
Lew
Guest
Posts: n/a
 
      05-23-2011
On 05/23/2011 02:27 PM, Robert Klemme wrote:
> On 23.05.2011 19:06, Daniele Futtorovic wrote:
>> On 23/05/2011 18:20, Lew allegedly wrote:
>>> jlp wrote:
>>>> I agree with you, Lee, it is what i [sic] did with my own tool. Native
>>>> logs are
>>>
>>> Who's Lee?
>>>

>>
>> You're Lee now, Lee.

>
> Did you mean to say "Bruce"?
>


Just call me Lew Lee. Then you can sing, "Lew Lee. Lew, lay, thou little
tiny child. Bye-bye, Lew Lee. Lew, lay!"

--
Lew
Honi soit qui mal y pense.
http://upload.wikimedia.org/wikipedi.../c/cf/Friz.jpg
 
Reply With Quote
 
Martin Gregorie
Guest
Posts: n/a
 
      05-23-2011
On Mon, 23 May 2011 20:33:07 +0200, Robert Klemme wrote:

> On 23.05.2011 15:17, Patricia Shanahan wrote:
>> On 5/23/2011 12:50 AM, Ulrich Scholz wrote:
>>> I'm looking for an approach to the problem of analyzing application
>>> log files.
>>>
>>> I need to analyse Java log files from applications (i.e., not logs of
>>> web servers). These logs contain Java exceptions, thread dumps, and
>>> free-form log4j messages issued by log statements inserted by
>>> programmers during development. Right now, these man-made log entries
>>> do not have any specific format.
>>>
>>> What I'm looking for is a tool and/or strategy that supports in
>>> lexing/ parsing, tagging, and analysing the log entries. Because there
>>> is only little defined syntax and grammar - and because you might not
>>> know what you are looking for - the task requires the quick issuing of
>>> queries against the log data base. Some sort of visualization would be
>>> nice, too.
>>>
>>> Pointers to existing tools and approaches as well as appropriate
>>> tools/ algorithms to develop the required system would be welcome.

>>
>> I would use Perl, and begin by recognizing some of the more important
>> formats, such as thread dumps. I agree with the desirability of
>> introducing some organized formatting into the log messages, but an
>> ad-hoc Perl program can often get useful data out of a disorganized
>> log.

>
> Only that Perl is so awful - YMMV of course. But for these kinds of
> tasks (more correctly: for *any* task) I very much prefer to use Ruby
> because of its cleaner OO and cleaner syntax.
>

I do the same, but use gawk rather than Perl: I have the same objections
to Perl as you, while gawk is pretty straight forward if you understand
regexes and can write C.

So far, using gawk to extract the information I've needed from Linux
system logs has been rather straight forward. Besides, I generally find
gawk to be more concise and readable than Perl, for this type of job,
anyway.


--
martin@ | Martin Gregorie
gregorie. | Essex, UK
org |
 
Reply With Quote
 
Daniele Futtorovic
Guest
Posts: n/a
 
      05-23-2011
On 23/05/2011 21:02, Lew allegedly wrote:
> On 05/23/2011 01:16 PM, Daniele Futtorovic wrote:
>> I've been faced a while ago with a situation where some orthogonal
>> organisational unit wanted to exploit my logs. I told them to GTFO.
>>
>> My logs are my logs. I put in it what I consider necessary. I often
>> improve them as I step through the code. I might change the message, fix
>> the level, &c. I don't want to have them set in stone. Neither do I
>> generally have enough confidence in them to allow them to be used for
>> analysis.
>>
>> "The solution, then, is simple", I told them, "spec out the exact
>> messages and arguments you want, and the exact situations you want them
>> logged in, and I'll add them for you. But leave me my precious debugging
>> logs."
>>
>> Let me emphasize: IMHO debugging logs and logs for analysis are two
>> different things and should be kept strictly separated -- possibly
>> logged to a different target respectively.

>
> That last is rather a brilliant idea, to use different targets.
> Heretofore I've espoused that logs are primarily an operations tool, not
> a debugging tool, although in service of the former they inevitably and
> inherently must support the former. The problem I've always seen is that
> logging statements are left up to the programmer, and not specified for
> the project.
>


I'd call it (what I described): audit logging. I don't know if the
meaning of that term normally extends beyond databases, but I don't see
why it shouldn't.

--
DF.
An escaped convict once said to me:
"Alcatraz is the place to be"
 
Reply With Quote
 
Robert Klemme
Guest
Posts: n/a
 
      05-23-2011
On 23.05.2011 21:06, Lew wrote:
> On 05/23/2011 02:27 PM, Robert Klemme wrote:
>> On 23.05.2011 19:06, Daniele Futtorovic wrote:
>>> On 23/05/2011 18:20, Lew allegedly wrote:
>>>> jlp wrote:
>>>>> I agree with you, Lee, it is what i [sic] did with my own tool. Native
>>>>> logs are
>>>>
>>>> Who's Lee?
>>>>
>>>
>>> You're Lee now, Lee.

>>
>> Did you mean to say "Bruce"?
>>

>
> Just call me Lew Lee. Then you can sing, "Lew Lee. Lew, lay, thou little
> tiny child. Bye-bye, Lew Lee. Lew, lay!"


Aye!

Which reminds me of a guy who called himself "LL Cool J".

Associative memory...

robert

--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/
 
Reply With Quote
 
Lew
Guest
Posts: n/a
 
      05-23-2011
On 05/23/2011 04:10 PM, Robert Klemme wrote:
> On 23.05.2011 21:06, Lew wrote:
>> On 05/23/2011 02:27 PM, Robert Klemme wrote:
>>> On 23.05.2011 19:06, Daniele Futtorovic wrote:
>>>> On 23/05/2011 18:20, Lew allegedly wrote:
>>>>> jlp wrote:
>>>>>> I agree with you, Lee, it is what i [sic] did with my own tool. Native
>>>>>> logs are
>>>>>
>>>>> Who's Lee?
>>>>>
>>>>
>>>> You're Lee now, Lee.
>>>
>>> Did you mean to say "Bruce"?
>>>

>>
>> Just call me Lew Lee. Then you can sing, "Lew Lee. Lew, lay, thou little
>> tiny child. Bye-bye, Lew Lee. Lew, lay!"

>
> Aye!
>
> Which reminds me of a guy who called himself "LL Cool J".
>
> Associative memory...


Wow, I feel lucky. I was so afraid I'd be put in Coventry for that pun.

--
Lew
Honi soit qui mal y pense.
http://upload.wikimedia.org/wikipedi.../c/cf/Friz.jpg
 
Reply With Quote
 
Arved Sandstrom
Guest
Posts: n/a
 
      05-23-2011
On 11-05-23 04:02 PM, Lew wrote:
> On 05/23/2011 01:16 PM, Daniele Futtorovic wrote:
>> On 23/05/2011 15:11, Lew allegedly wrote:
>>> Ulrich Scholz wrote:
>>>> I'm looking for an approach to the problem of analyzing application
>>>> log files.
>>>>
>>>> I need to analyse Java log files from applications (i.e., not logs of
>>>> web servers). These logs contain Java exceptions, thread dumps, and
>>>> free-form log4j messages issued by log statements inserted by
>>>> programmers during development. Right now, these man-made log entries
>>>> do not have any specific format.
>>>>
>>>> What I'm looking for is a tool and/or strategy that supports in lexing/
>>>> parsing, tagging, and analysing the log entries. Because there is only
>>>> little defined syntax and grammar - and because you might not know
>>>> what you are looking for - the task requires the quick issuing of
>>>> queries against the log data base. Some sort of visualization would be
>>>> nice, too.
>>>>
>>>> Pointers to existing tools and approaches as well as appropriate tools/
>>>> algorithms to develop the required system would be welcome.
>>>
>>> It helps if you have a logging strategy that mandates a consistent
>>> logging format, specific information in particular positions or marked
>>> by particular markup, logging levels and other such so that your
>>> analysis tool isn't faced with a completely open-ended input. What you
>>> describe requires a general text-analysis approach, as you indicate that
>>> you can make no guarantees about the format. Based on that, your best
>>> tool is "less" or equivalent text-file reader.
>>>
>>> What is a tool supposed to do, read your mind?
>>>
>>> It's really hard to extract information from a garbage can where people
>>> just randomly dumped whatever they individually felt like dumping
>>> without regard for operational needs. You can't build a skyscraper on a
>>> bad foundation, and you can't build a good log analysis off a crappy
>>> log.
>>>
>>> Fix the logging system, then the analysis problem will be tractable.

>>
>> I would argue around the same lines.
>>
>> I've been faced a while ago with a situation where some orthogonal
>> organisational unit wanted to exploit my logs. I told them to GTFO.
>>
>> My logs are my logs. I put in it what I consider necessary. I often
>> improve them as I step through the code. I might change the message, fix
>> the level, &c. I don't want to have them set in stone. Neither do I
>> generally have enough confidence in them to allow them to be used for
>> analysis.
>>
>> "The solution, then, is simple", I told them, "spec out the exact
>> messages and arguments you want, and the exact situations you want them
>> logged in, and I'll add them for you. But leave me my precious debugging
>> logs."
>>
>> Let me emphasize: IMHO debugging logs and logs for analysis are two
>> different things and should be kept strictly separated -- possibly
>> logged to a different target respectively.

>
> That last is rather a brilliant idea, to use different targets.
> Heretofore I've espoused that logs are primarily an operations tool, not
> a debugging tool, although in service of the former they inevitably and
> inherently must support the former. The problem I've always seen is
> that logging statements are left up to the programmer, and not specified
> for the project.
>

General agreement with all. I also am coming off one particular project
where part of the work - not a major part, but an important part - was
to improve logging. One of the first things we did was officially
recognize that we had many different clients of logging output. They
wanted different things at different levels at different times with
different storage stipulations.

The solution was pretty simple, and it's dynamic. I don't propose to get
into a logging framework war, but in this case we saw that JUL wouldn't
cut it, but log4j would do the trick. We had to do some arcane app
server-related stuff for JMX and log4j.xml, also integrate exception
handling with various "global" handlers that could also log, and wrap
log4j calls with a plethora of methods that would result in messages
formatted to our liking, but after that the heavy lifting was and is
done: it's now up to the clients - *not* to the developers - to request
what gets logged and in what manner.

Developers of course are clients themselves.

Again, not to get into a logging framework war, but for these purposes
log4j brings a lot to the table. It's common to need logging on specific
Java packages to be at a certain level, for output of that specific
logging to go to a specific target (like its own file) and have its own
storage policy, and for that logging to not be (or be, as the case
demands) to be additive to parent logging. Being able to do this is a
minimum for supporting different clients.

We also added, as part of our log4j method wrappers, an extra field for
all log messages that characterizes a "functional category". This allows
decorating all messages with information as to the identity of a
functional subsystem, and is helpful to post-processing tools like Splunk.

This system has been in production now for about 4 months, and
operational support staff and other clients are very pleased with it.
It's not perfect, because not all the log statements exist in the code
to support every informational requirement (known or unknown), but the
framework is not a problem.

One sidenote: despite doing everything I describe above, you can still
end up with logs that are difficult to interpret, and more log
statements aren't necessarily the answer. This typically happens when
your code itself is a spaghetti tangle. Sometimes to fix a logging
problem you really do need to refactor your logged code.

AHS
 
Reply With Quote
 
Martin Gregorie
Guest
Posts: n/a
 
      05-23-2011
On Mon, 23 May 2011 15:02:23 -0400, Lew wrote:

> On 05/23/2011 01:16 PM, Daniele Futtorovic wrote:
>> Let me emphasize: IMHO debugging logs and logs for analysis are two
>> different things and should be kept strictly separated -- possibly
>> logged to a different target respectively.

>
> That last is rather a brilliant idea, to use different targets.
> Heretofore I've espoused that logs are primarily an operations tool, not
> a debugging tool, although in service of the former they inevitably and
> inherently must support the former. The problem I've always seen is
> that logging statements are left up to the programmer, and not specified
> for the project.
>

I tend to use at least two logging streams: debugging and operational. I
leave debugging statements in production code: its normally off (of
course) but can be turned on if needed. Operational debugging includes
informational and error messages to be used by sysadmins which are always
enabled and should be fairly infrequent as well as performance
measurement messages. The latter can be configured on or off. As others
have said, the messages need to be designed with both log stream
selection and ease of parsing for later analysis in mind.

In a C application for a *NIX OS its easiest to send all these messages
to the system logger and let it deal with creating separate logs for the
various message streams: its then trivial to use 'tail' to present the
operational stream to sysadmins.

If the application is written in a language that doesn't provide easy
access to the system logger or is run on an OS that doesn't have one, I'd
include a custom logging process as part of the application.


--
martin@ | Martin Gregorie
gregorie. | Essex, UK
org |
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Logs button not opening Logs GUI Lester Lane Cisco 6 08-28-2009 10:02 AM
WinXP Home SP2 logs in then right away logs off Andrew Computer Support 15 10-19-2004 09:45 AM
Win XP SP2 Logs in then Logs out awallwork at sign gmail dot com Computer Support 2 10-16-2004 08:19 PM
Win XP SP2 Logs in then Logs out Andrew Computer Support 2 10-16-2004 04:27 PM
WinXP Home SP2 Logs on then Logs off awallwork at sign gmail dot com Computer Support 2 10-16-2004 02:28 AM



Advertisments