Velocity Reviews

Velocity Reviews (http://www.velocityreviews.com/forums/index.php)
-   C Programming (http://www.velocityreviews.com/forums/f42-c-programming.html)
-   -   What happens when your program crashes? (http://www.velocityreviews.com/forums/t599909-what-happens-when-your-program-crashes.html)

jacob navia 03-18-2008 11:24 PM

What happens when your program crashes?
 
CB Falconer boasts:

>> My point is that those units have never seen a debugger.



Well, he should not read this article, since we are going to see what
happens when in the middle of the execution of your program, you see

"Segmentation fault"

And nothing. Your program doesn't exist any more, and it is up
to you to find *where* in those thousands of lines lies the fault.

First, you have to compile your program for debug, i.e. to switch on the
debug information generation. Personally I always leave that switch ON,
and only tell the linker to not include that information in the final
executable if I am going to ship the program to the customer. Otherwise,
it is always in.

Second you have to start the debugger, and run the program in the same
environment as the one where it crashed. This is important. A slight
difference in environment could mean that the crash disappears, or
that you get another crash, and not the one you were looking
for...

:-)

Let's suppose for now, that you reproduce the crash.

A "crash" is actually a signal from the CPU to the operating system
(an interrupt) that alters the flow of your program, since an interrupt
routine takes over.

This interrupt can be triggered by a wrong address, for instance, and in
the old times when I started learning C, the processor would signal that
the system bus did not accept some address. The program would stop with

"bus error".


Nowadays it is still the same. Only the appearances have changed. An
interrupt routine informs the OS that your program is trying to access
some bad address, the OS notices that the process is running under a
debugger, and then it notifies the debugger of the fact that the
program under debug crashed. The details of how this is done change
from OS to OS, but there is ONE information that MUST arrive to the
debugger (at least): the program counter.

The debugger then, looks its database of debug information in the line
numbers associative tables, and looks up where this address appears.

In the easy cases, it will find the address in question and will then be
able to display the file where the crash happened.

In many cases however, the debugger will NOT find any correspondence
between the address of the crash, and the program. In those cases, the
debugger can either
1) give up, and just display the address where the program crashed,
as gdb does some times,
2) Try to use the st*** (Oh excuse me I was just going to use that
5 letter word)... Try again
Try to use the space allocated for local variables and follow the
linked list of function activations.

One way of trying to find the current point is the brutal approach of
the lcc-win debugger. I take all locals space, and go reading all of it
trying to find out if any of the numbers in the stack correspond to any
address I have in my line number information.

If I find (as usually is the case) an integer that correspond to a
program address, I assume that this address has been left by a CALL
instruction. To verify this, I read the contents around that address
and verify that a CALL instruction is in there. If the verification
passes I am certain that a CALL was done, and that this is where
the call in the program source code is located where things went wrong.

Because, obviously, you can pass a wrong pointer to a system routine
that calls another system routine with that pointer, and the chain can
be quite deep until the pointer is actually used.

Once the debugger has the point where the last user routine was active,
it can display the source code to the user. Local variables can be
retrieved only if the debugger is able to figure out the value of
the frame pointer when this routine was called. In general this is
enormously difficult, since the system routines are highly optimized
code, without stack frames, and maybe using the frame pointer as
a normal register to hold values that have nothing to do with stack
frames, values that if followed at face value could make the debugger
crash.

---------------------------------------------------------------------

The above is a description of debugging in a workstation type
environment. In other environments, this whole stuff is much more
difficult.

Take the lcc-win debugger when used with a 16 bit DSP of Analog devices
(80-90K of RAM + 512K EPROM).

For starters, there is no possibilities of breakpoints. Eproms can be
written in 4K chunks, and setting/unsetting breakpoints like that
is out of the question.

The DSP is running a version of a FORTH virtual machine, a very
small OS/controller. The generated C then, is just instructions for
this monitor that allows for a single STOPWM instruction, that
allows to stop the program and enter the monitor.

This approach has been described by Dave Hanson et al, see reference
[1]. In the general approach of Dave Hanson, the monitor is not
FORTH based but just a small C routine.

In this case, the FORTH interpreter is used as monitor.

In this environment, there is no MMU, and writing to a wrong address
will just destroy some data elsewhere, but not provoke any
visible problem immediately. There are no crashes, at least not
of the kind we saw above.

When the program stops because of a breakpoint, the first thing the
debug monitor does, is to see if the address is in the table of 16
active breakpoints. If it is not in there, the program continues execution.

If we have reached an active breakpoint, the monitor sends a character
through the serial line to lcc-win, patiently waiting for news from the
program in the PC hooked up to the circuit board. Then, the debugger
asks for addresses of the key variables, stack contents, etc. The user
sees his/her source code, and the whole feels like visual studio :-)

lcc-win can also send a break (holding the serial line to zero for 8 or
nine bits, I do not remember exactly). The break provokes a monitor
interrupt, and we start a debugging session like if there was
an active breakpoint.

As you can see both environments are completely different. It would be
interesting to hear from people that use debuggers in other embedded
systems to share their experiences here.


-----------------------------------
[1] D. R. Hanson and M. Raghavachari, A Machine-Independent Debugger,
Software—Practice and Experience 26 (11), 1277-1299, Nov. 1996.

--
jacob navia
jacob at jacob point remcomp point fr
logiciels/informatique
http://www.cs.virginia.edu/~lcc-win32

Richard Heathfield 03-19-2008 12:41 AM

Re: What happens when your program crashes?
 
jacob navia said:

> [...] we are going to see what
> happens when in the middle of the execution of your program, you see
>
> "Segmentation fault"
>
> And nothing. Your program doesn't exist any more, and it is up
> to you to find *where* in those thousands of lines lies the fault.


Easy.
gdb ./foo core
bt

That gives you a backtrace, which will allow you to identify the point in
your program where the bug manifested itself sufficiently seriously to
cause a segfault. Then you simply work back from there.

Example:

me@here> ./foo Users "Test base 1 - users" users.sql tb1
usersstyle.css.partial > users.php
Processing table [users]
Segmentation fault (core dumped)

Hmmm, a core dump. Oh dear, how sad, never mind.

me@here> gdb ./foo core
[Version and copyright info snipped]
This GDB was configured as "i386-suse-linux"...
Core was generated by `./foo Users Test base 1 - users users.sql tb1
users'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
#0 0x40149ea3 in _IO_2_1_stdout_ () from /lib/libc.so.6

Hmmm. stdout? Weird. Okay...

(gdb) bt
#0 0x40149ea3 in _IO_2_1_stdout_ () from /lib/libc.so.6
#1 0x0804ff8c in pcl_DllistIterate (list=0x80660b8, extra=0xbffff414) at
pcldll.c:365
#2 0x0804cc69 in GenerateDisplay (fieldlist=0x80660b8,
prefixname=0xbffff729 "Users", fp=0x40149ea0, mdata=0xbffff4cc)
at foo.c:2026
#3 0x0804d319 in GenerateForms (fieldlist=0x80660b8, PageTitle=0xbffff72f
"Test base 1 - users",
prefixname=0xbffff729 "Users", fp=0x40149ea0, mdata=0xbffff4cc,
dbname=0xbffff75d "brc") at foo.c:2198
#4 0x0804d92b in GenSQLTableEditorCode (prefixname=0xbffff729 "Users",
PageTitle=0xbffff72f "Test base 1 - users", infilename=0xbffff753
"users.sql",
dbname=0xbffff75d "tb1", stylename=0xbffff761 "usersstyle.css.partial")
at foo.c:2331
#5 0x0804d9ce in main (argc=6, argv=0xbffff594) at foo.c:2356
#6 0x400479ed in __libc_start_main () from /lib/libc.so.6

Hmmm - pcl_DllistIterate has a good track record, so it's a fair bet that
it's been given bad data. So let's look at the caller, GenerateDisplay, at
line 2026 of my code (and yes, I know, by now I should have refactored it
into smaller files):

pcl_DllistIterate(fieldlist, &cdata);

Well, that line looks fine, so let's trace back from there a few lines.

pcl_DllistSetExecutorFunction(fieldlist, stdout);
cdata.fp = fp;
cdata.prefixname = prefixname;
cdata.mdata = *mdata;
cdata.mdata.cur_field = 0;
pcl_DllistIterate(fieldlist, &cdata);

pcl_DllistIterate iterates through the whole of a double-linked list (deep
shock there), the list being specified by its first parameter. The node
data and the second argument are both passed to a function that has been
specified in a previous call to pcl_DllistSetExecutorFunction. So there
are four likely-looking possibilities - a bad list, bad data at a
particular node, a bad 'extra' param, or a bad function.

The simplest to check is the function. (Always check the easiest option
first.) Looking a few lines up the code, we find the statement that sets
up the function, and... stdout isn't a function. Whoops. I'm so used to
passing stdout as an 'extra' that I seem to have passed it to the wrong
function by mistake. (If I'd read my diagnostics more carefully, in fact,
I'd have seen this - but I was blinded by the silly 'warning: missing
initializer' messages, of which I get dozens, and which I haven't yet
found out how to disable without also disabling lots of useful warnings.)

I change stdout to sqlfield_write_select, a real function rather than a
FILE *(!); in fact, the function I /intended/ to use for walking the list.
I re-compile. This time, I check the diagnostics more carefully. All is
well, I re-test, and the segfault is gone.

This all happened about 12 hours ago. The above is pretty much a verbatim
account of my debugging session.

Total number of lines in program (including library): 9987.
Number of lines stepped through: 0.
Number of breakpoints set: 0.
Number of watches set: 0.
Number of lines examined: 6.
Total time spent in debugger: 10-15 seconds.
Total time from segfault to bugfix: about a minute and a half.

--
Richard Heathfield <http://www.cpax.org.uk>
Email: -http://www. +rjh@
Google users: <http://www.cpax.org.uk/prg/writings/googly.php>
"Usenet is a strange place" - dmr 29 July 1999

robertwessel2@yahoo.com 03-19-2008 01:14 AM

Re: What happens when your program crashes?
 
On Mar 18, 6:24*pm, jacob navia <ja...@nospam.com> wrote:
> CB Falconer boasts:
>
> *>> My point is that those units have never seen a debugger.
>
> Well, he should not read this article, since we are going to see what
> happens when in the middle of the execution of your program, you see
>
> "Segmentation fault"
>
> And nothing. Your program doesn't exist any more, and it is up
> to you to find *where* in those thousands of lines lies the fault.
>
> First, you have to compile your program for debug, i.e. to switch on the
> debug information generation. Personally I always leave that switch ON,
> and only tell the linker to not include that information in the final
> executable if I am going to ship the program to the customer. Otherwise,
> it is always in.



I spent a fair number of my formative years in smaller mainframe
shops. While debuggers were available, they cost money, and tended
*not* to be the sorts of things smaller mainframe shops bought.

Anyway, almost all debugging was postmortem from either inspecting the
incorrect output, or looking at a core dump (assuming the program did
the equivalent of seg faulting), and then studying the source code.
The compilers usually were set to print a map of addresses and
statements, and we'd essentially always get linkage maps as well. If
you did have a dump, you'd be able to hand disassemble the
instructions leading up to the crash, figure out what they were point
at from the compile maps, and inspect the data in question for
interesting clues. Complex issues might take considerable spelunking
through the dump.

On occasion we would, indeed, and various trace statements ("printfs")
and rerun, and some compilers had various diagnostic aids (one Cobol
compiler could collect a trace of the statements executed, which was
sometimes helpful) - although those tended to be turned off for the
production builds. But recompiling a program tending to be something
of a chore, so inserting printfs was something of a last resort. Not
to mention something of an admission of failure.

And many of these were quite substantial programs and systems.

And I still spend a good chunk of my life in places where good
debuggers are less than handy (device drivers, embedded stuff, code in
the field, etc.) - although in almost all cases something, either a
formal debugger, or something ad-hoc, can be added if necessary, in
most cases it's rather faster to not bother.

A debugger is certainly handy at times, but not essential. While work
styles vary, I see a lot of people who, IMO, have an overreliance on
debuggers (there are a couple of programmers here who seem to fire up
a debugger when their coffee gets cold). The point is to make the
best use of the available tools - which, IMO, *doesn't* include firing
up the debugger at the first sign of trouble.

Doug Miller 03-19-2008 02:53 AM

Re: What happens when your program crashes?
 
In article <frpiv6$i4a$1@aioe.org>, jacob@nospam.org wrote:
>CB Falconer boasts:
>
> >> My point is that those units have never seen a debugger.

>
>
>Well, he should not read this article, since we are going to see what
>happens when in the middle of the execution of your program, you see


Why do you keep splitting threads and retitling them? Is it out of ignorance,
inconsideration, or stupidity?

Mark Bluemel 03-19-2008 08:50 AM

Re: What happens when your program crashes?
 
Doug Miller wrote:
> In article <frpiv6$i4a$1@aioe.org>, jacob@nospam.org wrote:
>> CB Falconer boasts:
>>
>>>> My point is that those units have never seen a debugger.

>>
>> Well, he should not read this article, since we are going to see what
>> happens when in the middle of the execution of your program, you see

>
> Why do you keep splitting threads and retitling them? Is it out of ignorance,
> inconsideration, or stupidity?


Are those mutually exclusive?

Richard Heathfield 03-19-2008 08:59 AM

Re: What happens when your program crashes?
 
Mark Bluemel said:

> Doug Miller wrote:


<snip>

>> Why do you keep splitting threads and retitling them? Is it out of
>> ignorance, inconsideration, or stupidity?

>
> Are those mutually exclusive?


No, that would be xor.

--
Richard Heathfield <http://www.cpax.org.uk>
Email: -http://www. +rjh@
Google users: <http://www.cpax.org.uk/prg/writings/googly.php>
"Usenet is a strange place" - dmr 29 July 1999

jacob navia 03-19-2008 09:18 AM

Re: What happens when your program crashes?
 
Mark Bluemel wrote:
> Doug Miller wrote:
>> In article <frpiv6$i4a$1@aioe.org>, jacob@nospam.org wrote:
>>> CB Falconer boasts:
>>>
>>>>> My point is that those units have never seen a debugger.
>>>
>>> Well, he should not read this article, since we are going to see what
>>> happens when in the middle of the execution of your program, you see

>>
>> Why do you keep splitting threads and retitling them? Is it out of
>> ignorance, inconsideration, or stupidity?

>
> Are those mutually exclusive?


Of course not!

You have BOTH.

Unable to contribute *anything* to the discussion, you can only
insult people, what is still within the reach of your IQ.

You and Mr Miller are a typical example of the regulars:

1) Do not contribute anything to the discussion
2) Insult to kill discussions
3) Lower the contents of the group so that only the regulars remain.
I started that series precisely to attract people into this group,
by discussing technical matters immediately related to the C
language.


--
jacob navia
jacob at jacob point remcomp point fr
logiciels/informatique
http://www.cs.virginia.edu/~lcc-win32

jacob navia 03-19-2008 09:47 AM

Re: What happens when your program crashes?
 
robertwessel2@yahoo.com wrote:
> On Mar 18, 6:24 pm, jacob navia <ja...@nospam.com> wrote:
>> CB Falconer boasts:
>>
>> >> My point is that those units have never seen a debugger.

>>
>> Well, he should not read this article, since we are going to see what
>> happens when in the middle of the execution of your program, you see
>>
>> "Segmentation fault"
>>
>> And nothing. Your program doesn't exist any more, and it is up
>> to you to find *where* in those thousands of lines lies the fault.
>>
>> First, you have to compile your program for debug, i.e. to switch on the
>> debug information generation. Personally I always leave that switch ON,
>> and only tell the linker to not include that information in the final
>> executable if I am going to ship the program to the customer. Otherwise,
>> it is always in.

>
>
> I spent a fair number of my formative years in smaller mainframe
> shops. While debuggers were available, they cost money, and tended
> *not* to be the sorts of things smaller mainframe shops bought.
>
> Anyway, almost all debugging was postmortem from either inspecting the
> incorrect output, or looking at a core dump (assuming the program did
> the equivalent of seg faulting), and then studying the source code.
> The compilers usually were set to print a map of addresses and
> statements, and we'd essentially always get linkage maps as well. If
> you did have a dump, you'd be able to hand disassemble the
> instructions leading up to the crash, figure out what they were point
> at from the compile maps, and inspect the data in question for
> interesting clues. Complex issues might take considerable spelunking
> through the dump.
>



Yes. But that is not the case today. Debuggers are no longer expensive.

My objective with this series is to attract young people and technically
interested people into this group. To show that C doesn't live in the
past but it has a future. Your post is interesting, but doesn't apply to
the environments of today.

> On occasion we would, indeed, and various trace statements ("printfs")
> and rerun, and some compilers had various diagnostic aids (one Cobol
> compiler could collect a trace of the statements executed, which was
> sometimes helpful) - although those tended to be turned off for the
> production builds. But recompiling a program tending to be something
> of a chore, so inserting printfs was something of a last resort. Not
> to mention something of an admission of failure.
>


You were writing your "ad hoc" debugger then. This is feasible, as you
prove, but it surely is time consuming.

> And many of these were quite substantial programs and systems.
>
> And I still spend a good chunk of my life in places where good
> debuggers are less than handy (device drivers, embedded stuff, code in
> the field, etc.) - although in almost all cases something, either a
> formal debugger, or something ad-hoc, can be added if necessary, in
> most cases it's rather faster to not bother.
>



Yes. Writing correct code from the starts obviates a debugger.
Problem is, it is quite difficult to do.

> A debugger is certainly handy at times, but not essential. While work
> styles vary, I see a lot of people who, IMO, have an overreliance on
> debuggers (there are a couple of programmers here who seem to fire up
> a debugger when their coffee gets cold). The point is to make the
> best use of the available tools - which, IMO, *doesn't* include firing
> up the debugger at the first sign of trouble.


In principle this is correct. But I have gotten used to develop in the
debugger. I write a part to access some data structures, and put a
breakpoint where the calculations/usage of those data structures are
to be inserted. Then, within the debuggers, I write the new code.




--
jacob navia
jacob at jacob point remcomp point fr
logiciels/informatique
http://www.cs.virginia.edu/~lcc-win32

Bart van Ingen Schenau 03-19-2008 11:33 AM

Re: What happens when your program crashes?
 
On 19 mrt, 10:47, jacob navia <ja...@nospam.com> wrote:
> robertwess...@yahoo.com wrote:
> > I spent a fair number of my formative years in smaller mainframe
> > shops. While debuggers were available, they cost money, and tended
> > *not* to be the sorts of things smaller mainframe shops bought.

>
> > Anyway, almost all debugging was postmortem from either inspecting the
> > incorrect output, or looking at a core dump (assuming the program did
> > the equivalent of seg faulting), and then studying the source code.
> > The compilers usually were set to print a map of addresses and
> > statements, and we'd essentially always get linkage maps as well. If
> > you did have a dump, you'd be able to hand disassemble the
> > instructions leading up to the crash, figure out what they were point
> > at from the compile maps, and inspect the data in question for
> > interesting clues. Complex issues might take considerable spelunking
> > through the dump.

>
> Yes. But that is not the case today. Debuggers are no longer expensive.


I beg to differ. For desktop environments debuggers might be
inexpensive, but that is not true for other environments.
I work mostly on embedded software and there debuggers are still very
expensive (upwards of EUR 10000 a piece, mostly due to the required
specialised additional hardware).
And in embedded software, I often can't use a debugger anyway because
it affects the system behaviour too much to replicate the problem.

>
> My objective with this series is to attract young people and technically
> interested people into this group. To show that C doesn't live in the
> past but it has a future. Your post is interesting, but doesn't apply to
> the environments of today.


Actually, it does still apply. Only not to the environments you are
using.
And incidentally, I agree that C is still live and kicking. It has
been the language that I have used most in the past 8 years and I
still use it on a daily basis.

>
> --
> jacob navia


Bart v Ingen Schenau

CBFalconer 03-19-2008 09:10 PM

Re: What happens when your program crashes?
 
jacob navia wrote:
>

.... snip ...
>
> I started that series precisely to attract people into this group,
> by discussing technical matters immediately related to the C
> language.


Then why put it here? You should know that debuggers are OT in
this newsgroup. Just because people put up with the occasional
digression doesn't make it on-topic.

The reason is that debuggers depend on the actual implementation,
not the language. Therefore any discussion are not controlled by a
standard referance.

--
[mail]: Chuck F (cbfalconer at maineline dot net)
[page]: <http://cbfalconer.home.att.net>
Try the download section.



--
Posted via a free Usenet account from http://www.teranews.com



All times are GMT. The time now is 01:18 AM.

Powered by vBulletin®. Copyright ©2000 - 2014, vBulletin Solutions, Inc.
SEO by vBSEO ©2010, Crawlability, Inc.