Structure of PIX syslog messages

Hi,
Is there a definiton of keywords used by PIX Syslog messages?

I want to extract important information from these messages, like
source-port, destination-port, ...

Best regards,
tobi

 

:Is there a definiton of keywords used by PIX Syslog messages?

Not really, no.


:I want to extract important information from these messages, like
:source-port, destination-port, ...

That was reasonably easy up to 6.1. From 6.2 onward, the messages
changed in such a way as to lose important contextual information,
making it necessary to actively keep state on each connection in
order to figure out which is the source and which is the destination
[at least if one wants to do accounting by incoming vs outgoing.]

Also, if one has multiple interfaces, the relationship between
the interfaces can only be infered, with the inference being
potentially incorrect if one is using "reverse nat".

There is also the issue that if one has created a 'name' for
an IP address, then the name will be used in Deny messages,
but the IP address will be used in Translation and Build messages.

VPN SA messages use a mix of the two, and unless one has
crypto isakmp debugs turned on, it is not possible to figure out
which device the peer is for most VPN messages.

There is a 6.3 bug/oddity in which translation information is torn
down too quickly after a TCP connection is reset, resulting in
Deny reports for the trailing packets from the destination towards
the source -- making it appear as if the destination was attempting
to contact the source.


If you don't need accounting, and you don't need VPNs, and you
are prepared to hard-code knowledge of interface security levels
for each monitored device, then you can get by with monitoring
a fairly small number of messages.

Unless, that is, you want to be able to track -successful- connections
other than UDP or TCP (e.g., ICMP or GRE) -- if you need to track
those, then it gets messy again.

 

Yes. reading PIX logs is quite simple. For ever tcp or udp connection here
are two detailed records for buiilding and summarising connection.
Cisco website has a detailed page about an every single instance in PIX log
and how to get any info you want out of it

 

:Yes. reading PIX logs is quite simple. For ever tcp or udp connection here
:are two detailed records for buiilding and summarising connection.
:Cisco website has a detailed page about an every single instance in PIX log
:and how to get any info you want out of it

Sorry, but -every- part of what you say is incorrect.

1) denied connections have no Build or Teardown message.

2) UDP and TCP use -different- Build and Teardown messages.

3) The only way to be sure that one is associating the correct Build
with the correct Teardown is by tracking the connection number.

4) Connection numbers are independant for udp and tcp.

5) Unless you are using the TCP version of PIX's syslog, one cannot
be sure that the Build or Teardown message made it into the log.

6) Connections must be tracked independantly for each device monitored,
because the interface names, IP address spaces, and connection numbers
can all overlap.

7) It is not possible to tell from the Teardown message which side
initiated the connection. Therefore one must track that information
from the Build message.

8) The Build message always puts parameters in the same order, so
one must look for the keyword 'inbound' or 'outbound' in order to
determine the direction and hence to know which of the IP addresses
one should record in the "source" field of one's accounting structure.

9) The Teardown message always puts parameters in the same order, so
one must carry around the directional flag in the accounting structure
in order to get the accounting right.

10) There are at least 7777777rent formats of Deny messages which
must be processed differently -- some of which do not even mention
the word 'Deny' (nor 'Denied'). PIX-4-406002 for example.

11) Several of the variant deny messages give no interface information.
PIX-2-106007 for example.

12) The PIX SYSLOG reference manuals have numerous places in which the
manual indicates simply that at a particular point in the message,
there is a "string". The possible content and semantic meaning of the
string is not described.

13) In order to properly break out the interface information from
the IP information, one must be aware of the fact that interface names
may contain arbitrary number of colons. One must also be aware that
named hosts may contain underscore and dash, and one must know all
the details of when names are used and when IPs are used.

14) Every time the PIX is reset, the counters start all over, so one
must know how to detect a restart and flush out one's accounting buffers.

15) The PIX simply never sends Teardown messages in response to some
Build conditions. One must therefore watch out for re-use of the
connection number, and for restarts, and at the earliest time that one
can deduce that the connection died, one must emit a conditional
accounting record that indicates that the connection lasted "up to"
the deduced time. This requires, naturally, that carry around
the connection times.

16) Considering (5) and (15), when one finds a Teardown message, one
must cross-check the IPs involved to be sure that the Teardown is
referring to the same event.

17) Connection numbers can wrap while connections are still alive,
so for the purposes of (14) one cannot simply watch to see that a
new connection has a lower connection number than a previous one.

18) Syslog being UDP, connection information can arrive out-of-order:
the teardown may arrive before the build.

19) Some of the Deny messages are shared between various protocols.
One must thus examine the protocol field in order to know how to
parse the parameters -- including, for example, that some protocols
have ports after the IPs and others don't.

20) One cannot use fixed field offsets for finding the message
content in the syslog, as there are three different time formats in
use, plus one other field if a particular sysop is chosen.

21) One cannot use fixed relative field offsets for extracting information
from some of the messages, as they may use (e.g.) "protocol 47"
instead of a simple protocol name such as "udp" or "ESP".

22) In 7.0, there are sysops that allow the device identification
to be changed in ways that permit spaces, thus throwing the field counts
off again.

23) Some icmp messages put the icmp type and subcode information into
positions normally occupied by port numbers, rather than by the
[e.g.] (type 11, code 0) format.

24) Any actual attempt to use port number 0 in a udp or tcp packet
will generate a message with a completely different format.

25) None the less, in some instances, 0 can show up in the port number
slots.

26) In some instances, the IP address emitted by the PIX to identify
itself in syslog is 0.0.0.0 .

27) "Deny inbound (No xlate)" has two completely different causes,
which can only be [approximately] distinguished by keeping connection
histories and examining timestamps.

28) Standard deny-by-acl messages can have two different causes,
again requiring connection histories and timestamp tracking. In
PIX 6.3, the alternative interpretation occurs fairly often, and
must not be misinterpeted as an attack or ACL failure or IP forgery.

29) In both Build and Deny messages, it is possible for the same
interface name to occur in both the left and right hand sides. One
must know how to interpret this.

30) One must understand how translations are noted in the PIX messages.
This is NOT documented in the SYSLOG reference.

31) In order to know about connections other than TCP or UDP, one must
track Translation messages -- no other messages are produced for
successful connections for other protocols (unless one puts in explicit
'log' keywords.)

32) There are various circumstances under which Translation messages
are not emitted and so NO trace is made of successful non- TCP/UDP
connections.

33) The exact set of messages possible depends on which subrelease
one is using. There were major transitions at 6.2 and again at 7.0,
and extensions in pretty much every subrelease that introduced new
features.

34) Now... do you know exactly how the messages differ when Reverse
Nat is being used?


And there are other tricks one has to know as well.

I don't know about you, but I would not call this "simple". I know for
sure that two records are not produced for each TCP and UDP connection.
There is clearly a lot about the messages that the SYSLOG reference
doesn't tell you that you need in order to extract information
properly.

 

Hi,
thank you very much for all these answers.

But my problem haven't been solved yet.
The problem is that Cisco uses a different syntax for the same
semantic.
Example: In log messages (type: IDS) the source IP address followed the
keyword "from". This isn't the case if you have a look at log messages
of the SEC type - now there isn't any keyword for the source ip
address.

How should I parse these different information?
I want to extract - from all Cisco Syslog Messages - the source/target
ip address, ports, ....

The question in matter is, how can I do this quiet efficient???

Best regards,
Tobi

 

:But my problem haven't been solved yet.

Yup.

:The problem is that Cisco uses a different syntax for the same
:semantic.

Yup.

:How should I parse these different information?
:I want to extract - from all Cisco Syslog Messages - the source/target
:ip address, ports, ....

What do you want -done- with the information? How accurate do you need
the results to be? How quickly do you need to be able to process the
messages?

For example, if the message is the one complaining that the wrong port
number was used in an FTP command, then considering that the message
itself does not contain information about interfaces, control port, is
always missing one of the two IP addresses, and lists the untranslated
IP instead of the NAT'd IP, should the analysis program be keeping around
enough context and do a sophisticated enough parsing to be able to
emit a log entry that contains the source and destination IPs and ports,
by matching the entry back through active transactions?


:The question in matter is, how can I do this quiet efficient???

If you need per-source and per-destination accounting, and you need
the outputs to be in a consistant format such as "source first
then destination", and if you want the IP adddresses to be looked
up into names, and if you want features like ability to select
a range of times or to quickly to a file block and start searching
from there, and if you need the code to be -efficient- and to
handle a series of devices simultaneously...

Then you can expect to spend 3+ months writing the program and
fine-tuning it and featurizing it... and that's if you already know
Perl decently well.


Network Intelligence used to have a commercial program that did
some of this work... but it made a number of mistakes and it was
slow, and it was not exactly inexpensive. And it's now discontinued
anyhow.

I know of two non-commercial programs that do more than a superficial
analysis. As far as I know, neither of them is available to the public.

I don't know what the other one is like; it sounds to be faster
than mine, but I do not have information about it's features or accuracy.
I had the -impression- that it was considered Proprietary Software.

Mine would need probably a month or more additional work before a public
release could be realistic. I'm not likely to have that month or more
available this year. My management doesn't perceive the program as
having any value beyond keeping me amused, and it doesn't have anything
to do with our Mandate...


For more information on this topic, and to find the reference to the
author of the other program I mentioned, see the subthread
starting from:

http://groups.google.ca/group/comp.dcom.sys.cisco/msg/18d97bca9a5fd4a5

 

No Walter, seems like you don't know anything at all about logs!
and making a big deal out of it.
1. First of all, Teardown messages can be tracked down using ConnecitonID,
2. second of all, you can find out who initiated connection, if you pay
attention and use your brain a little, you can see destinctive tags such as
Built outbound or Built Inbound
3. thirdly, i did not mention Deny statements to you you faggot
4. fourthly: Connection numbers are independant for udp and tcp <- what the
hell does that have to do with anything??
also, statement 5 does not make any sense at all
5. "10) There are at least 7777777rent formats of Deny messages which must
be processed differently" <-- no it doesn't, there are only a few
variations, such as access-list denied or no connection
and i woul'd worryabout the rest of your silly comments down ..

 

:1. First of all, Teardown messages can be tracked down using ConnecitonID,

Yes -- but you have to maintain that state. We're only considered
SMB in size, but we typically have more than 75000 simultaneously
active connections at our daily peak.

In PIX up thru 6.1, the Teardown syslog message was sufficient by itself
to figure out all the necessary state information.

:2. second of all, you can find out who initiated connection, if you pay
:attention and use your brain a little, you can see destinctive tags such as
:Built outbound or Built Inbound

There are no Built message for any protocols other than TCP and UDP,
and if you want your tool to be compatable with a mix of PIXen, you
need to be able to analyze 7 different Build messages.

This isn't about tracking one -individual- connection by looking through
a few kilobytes of system logs: this is about tracking every one of
more than a million messages per day (consistantly); our logs peak at
1 1/4 Gb per day, and some of our connections have to be tracked over
a period of 3 weeks.

1 million messages per day. How long would it take you just to read one
day's worth -once- through? Unless you are a certified Speed Reader, my
calculation is that it would take two months of (literally) non-stop reading.

You just don't have the luxary of going back and checking every
connection by hand: your program has to process all the messages
automatically, and it has to get *all* the details right.

:3. thirdly, i did not mention Deny statements to you you faggot

"On the Internet, nobody knows you are a dog" -- but I think one
can safely assume that I am neither a bundle of wood tied together,
nor a cigarette. I don't think a reference to sexual orientation has
any place in a technical discussion.

I agree completely that you did not mention Deny statements. You did,
though, say that parsing PIX syslog messages was simple, and the OP had
indicated that s/he wanted to be able to process -all- PIX messages.
One could be generous and suppose that you didn't read the question
properly, but the fact remains that you are incorrect that parsing PIX
syslog messages is simple: you ignored all the difficult cases.

:4. fourthly: Connection numbers are independant for udp and tcp <- what the
:hell does that have to do with anything??

It means that you have to keep *two* data structures, not just one.
A layer of complexity.

:also, statement 5 does not make any sense at all

Statement 5 was,
:> 5) Unless you are using the TCP version of PIX's syslog, one cannot
:> be sure that the Build or Teardown message made it into the log.

If you did not understand this, then it becomes less surprising that
you thought parsing the messages was simple.

PIX normally uses SYSLOG to send its log messages (though it can
send the messages via SNMP Traps instead.) The SYSLOG protocol is
normally UDP port 514. Being UDP, the protocol is fundamentally
unreliable -- the message is pushed out on to the wire, and if the
message disappears along the line, nothing notices. One must thus assume
that not *all* of the messages made it from PIX to the logging
device. This is particularily true if one is using the 'log' keyword
on an ACL entry, as:

http://www.cisco.com/univercd/cc/td/doc/product/iaabu/pix/pix_sw/v_63/config/sysmgmt.htm#wp1097293

There may exist a large number of flows concurrently at any point
of time. To prevent unlimited consumption of memory and CPU
resources, a limit is placed on the number of concurrent
deny-flows. When the limit is reached, no new deny-flow will be
created until the existing deny-flows expire.


PIX has an alternate SYSLOG delivery mechanism, via TCP instead of UDP.
When the alternate TCP mechanism is used, the PIX will stop processing
activities that would generate log messages, until the ACK comes back
from the TCP logging server indicating that the message was recorded
successfully. If one uses this mechanism, then one can be sure that
one receives all of the messages, but the flip side is that if the
logging host goes down or the logging host disk fills up, then the PIX
will stop passing traffic [which is a mandatory security requirement
in some locations, but just a veritable nuisance in others.]


When a message does not make it into the log, one has the difficulty
of what to do with the partial data found -- how to synthesize the
missing information. For example, if it is a Build message that goes
missing, then one might look back through the connection information,
find the next connection number (of the same type) that -did- show up,
and use that as a bound as to the time the connection was created.
This does, though, require that one keep connection information even
for connections that one has fully accounted for [though one can
optimize this by only recording the adjacent connection information
in cases where there are gaps in the sequence numbers.... keeping in
mind that it isn't uncommon for the packets to arrive out-of-order...]

:5. "10) There are at least 7777777rent formats of Deny messages which must
:be processed differently"

Sorry about that, somehow my '7 different' got transformed. I must
have accidently typed 7r7 in vi instead of just r7 .

: <-- no it doesn't, there are only a few
:variations, such as access-list denied or no connection

There are several different messages which imply that the connection
was denied, and for which no explicit Deny message will be generated.
Even ignoring the ones that do not say "Deny" or "denied", there are:

%PIX-2-106001 (deny inbound TCP, no ACL mentioned)
%PIX-2-106006 (deny inbound UDP, no ACL mentioned)
%PIX-2-106007 (deny due to DNS)
%PIX-3-106010 (deny inbound, no ACL mentioned)
%PIX-3-106011 (deny because of no xlate)
%PIX-3-106014 (deny inbound icmp, no ACL given)
%PIX-3-106015 (deny TCP no connection)
%PIX-3-106016 (deny IP spoof)
%PIX-2-106017 (deny IP Land Attack)
%PIX-2-106018 (deny outbound icmp, ACL given)
%PIX-2-106020 (deny teardrop fragment)
%PIX-1-106021 (deny reverse path check)
%PIX-1-106022 (deny connection spoof / asymmetric routing)
%PIX-4-106023 (deny icmp or udp or tcp, ACL given)
%PIX-n-106100 (ACL log entry that might contain Deny)
%PIX-3-302302 ("ACL = deny" -- an IPSec related message)
%PIX-5-304002 (URL access denied)
%PIX-3-313001 (deny icmp addressed to a PIX interface)
%PIX-2-316001 (denied new tunnel)
%PIX-4-407001 (deny traffic for local-host)
%PIX-6-605004 (login denied)
%PIX-3-610001 (NTP packet denied)
%PIX-3-710003 (access denied by ACL -- for http, ssh, telnet commands)

That's 23 variations, and doesn't even include the messages that use
synonyms for deny such as "dropped", let alone the implicit denials.

Exactly how big does "a few" encompass for you?


:and i woul'd worryabout the rest of your silly comments down ..

Well, perhaps -you- would not worry about the details, but that
doesn't mean that parsing the logs is simple: it just means that
your analysis would be neither complete nor accurate. One hopes that
you have a good Intrusion Detection System in place...