Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Perl > Perl Misc > Perl program and database locked but alive

Reply
Thread Tools

Perl program and database locked but alive

 
 
TF
Guest
Posts: n/a
 
      03-06-2004
Is there any way to intervene with a program that is still running, but
locked during database read/write subs, to see where the lockup is coming
from?
This problem cannot be repeated and is the result of web page data
corruption, or storage of said data.

Thanks


 
Reply With Quote
 
 
 
 
gnari
Guest
Posts: n/a
 
      03-06-2004
"TF" <(E-Mail Removed)> wrote in message news:404947d8$(E-Mail Removed)...
> Is there any way to intervene with a program that is still running, but
> locked during database read/write subs, to see where the lockup is coming
> from?


print debug info into a log file
when deadlock occur, read the logfile

you give no info about your application, or what kind of locking
is used, but often the risk of deadlocks can be reduced by being
careful about order of updated within transactions.
for example if in one place you have
transaction update a, update b
and in another place a transaction update b, update a
you have greatly increased risk of deadlocks

gnari



 
Reply With Quote
 
 
 
 
James Willmore
Guest
Posts: n/a
 
      03-06-2004
On Fri, 05 Mar 2004 19:05:15 -0800, TF wrote:

> Is there any way to intervene with a program that is still running, but
> locked during database read/write subs, to see where the lockup is coming
> from?
> This problem cannot be repeated and is the result of web page data
> corruption, or storage of said data.


I'm thinking you're using the DBI module, but can't be sure ... because
you have posted no code related to your issue

If you're using the DBI module and think that it's having trouble with the
database it's connecting to, use the 'trace' function to get more
information - `perldoc DBI` for more information.

Another way to do CGI script debugging is to run the script at the command
line to see what happens (again ... no code, no more specific help ).

HTH

--
Jim

Copyright notice: all code written by the author in this post is
released under the GPL. http://www.gnu.org/licenses/gpl.txt
for more information.

a fortune quote ...
Common sense is the collection of prejudices acquired by age
eighteen. -- Albert Einstein

 
Reply With Quote
 
TF
Guest
Posts: n/a
 
      03-06-2004

"TF" <(E-Mail Removed)> wrote in message news:404947d8$(E-Mail Removed)...
> Is there any way to intervene with a program that is still running, but
> locked during database read/write subs, to see where the lockup is coming
> from?
> This problem cannot be repeated and is the result of web page data
> corruption, or storage of said data.
>
> Thanks
>


O.K., more info.
Can't post any code, as it is over 15K lines.

I'm getting data from the web and storing it into an MLDBM database.

The hang is most likely one of the following:

1) Non-response from web request, though I'm not sure if this can happen.
2) The data checking routine has gone into a loop.
3) Corrupt data has been input into the database causing a lockup on future
access.

The ties & unties are all checked and O.K., so it can't be that.

If I can just intervene and find out what line of code it is hung on, would
be great.
No cgi is involved here.

Thanks


 
Reply With Quote
 
Jay Tilton
Guest
Posts: n/a
 
      03-06-2004
"TF" <(E-Mail Removed)> wrote:

: "TF" <(E-Mail Removed)> wrote in message news:404947d8$(E-Mail Removed)...
: > Is there any way to intervene with a program that is still running, but
: > locked during database read/write subs, to see where the lockup is coming
: > from?
: > This problem cannot be repeated and is the result of web page data
: > corruption, or storage of said data.
:
: O.K., more info.
: Can't post any code, as it is over 15K lines.
:
: I'm getting data from the web and storing it into an MLDBM database.
:
: The hang is most likely one of the following:
:
: 1) Non-response from web request, though I'm not sure if this can happen.
: 2) The data checking routine has gone into a loop.
: 3) Corrupt data has been input into the database causing a lockup on future
: access.
:
: The ties & unties are all checked and O.K., so it can't be that.
:
: If I can just intervene and find out what line of code it is hung on, would
: be great.

A simple signal handler could help there.

use Carp qw(confess);
$SIG{ 'INT' } = sub { confess "Signal caught" };

Then the process will terminate with a call stack backtrace when you send
that signal to it. See perlvar and perlipc for details.

: No cgi is involved here.

Huh? Where is the "web request" coming from if not through CGI?

 
Reply With Quote
 
gnari
Guest
Posts: n/a
 
      03-06-2004
"Jay Tilton" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed)...
> "TF" <(E-Mail Removed)> wrote:

[snip lockup description]
>
> : No cgi is involved here.
>
> Huh? Where is the "web request" coming from if not through CGI?


I understand it as the script is doing a series of HTTP client requests
and does some checks and stores some results to a database

in that case it is not a question of database deadlocks the original
post gave me the oimpression of , unless multiple parallel requests
are being made, in which case file locking on the database needs to
be done.

To TF:
I suggest you just do more debugging, such as logging to a file
extensively, store on disk last input file, and so on.
then when lockup happens, tail the log file to see where in the
script you where, look at the last input and so on.

then you might feed your script the last stored input again, to see if
it is the data that does it. I am sure you have better ideas than us
on how to debug your program, but the main rule is to strip away
everything that is not needed to exhibit your problem, until what
is left is so little that it becomes obvious.

gnari



 
Reply With Quote
 
TF
Guest
Posts: n/a
 
      03-06-2004

"Jay Tilton" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed)...
> "TF" <(E-Mail Removed)> wrote:
>
> : "TF" <(E-Mail Removed)> wrote in message

news:404947d8$(E-Mail Removed)...
> : > Is there any way to intervene with a program that is still running,

but
> : > locked during database read/write subs, to see where the lockup is

coming
> : > from?
> : > This problem cannot be repeated and is the result of web page data
> : > corruption, or storage of said data.
> :
> : O.K., more info.
> : Can't post any code, as it is over 15K lines.
> :
> : I'm getting data from the web and storing it into an MLDBM database.
> :
> : The hang is most likely one of the following:
> :
> : 1) Non-response from web request, though I'm not sure if this can

happen.
> : 2) The data checking routine has gone into a loop.
> : 3) Corrupt data has been input into the database causing a lockup on

future
> : access.
> :
> : The ties & unties are all checked and O.K., so it can't be that.
> :
> : If I can just intervene and find out what line of code it is hung on,

would
> : be great.
>
> A simple signal handler could help there.
>
> use Carp qw(confess);
> $SIG{ 'INT' } = sub { confess "Signal caught" };
>
> Then the process will terminate with a call stack backtrace when you send
> that signal to it. See perlvar and perlipc for details.


Thanks, I will.

>
> : No cgi is involved here.
>
> Huh? Where is the "web request" coming from if not through CGI?


LPW

>



 
Reply With Quote
 
TF
Guest
Posts: n/a
 
      03-07-2004

"gnari" <(E-Mail Removed)> wrote in message
news:c2dj1b$2qr$(E-Mail Removed)...
> "Jay Tilton" <(E-Mail Removed)> wrote in message
> news:(E-Mail Removed)...
> > "TF" <(E-Mail Removed)> wrote:

> [snip lockup description]
> >
> > : No cgi is involved here.
> >
> > Huh? Where is the "web request" coming from if not through CGI?

>
> I understand it as the script is doing a series of HTTP client requests
> and does some checks and stores some results to a database
>
> in that case it is not a question of database deadlocks the original
> post gave me the oimpression of , unless multiple parallel requests
> are being made, in which case file locking on the database needs to
> be done.
>
> To TF:
> I suggest you just do more debugging, such as logging to a file
> extensively, store on disk last input file, and so on.
> then when lockup happens, tail the log file to see where in the
> script you where, look at the last input and so on.
>
> then you might feed your script the last stored input again, to see if
> it is the data that does it. I am sure you have better ideas than us
> on how to debug your program, but the main rule is to strip away
> everything that is not needed to exhibit your problem, until what
> is left is so little that it becomes obvious.
>
> gnari
>
>

I've been basically doing this for two weeks, but it takes a whole day to
get to this point and the data from the last request and last stored plce
isn't helping much.
Once I kill the process, the dbm is screwed so I can't really see anything.
All it says is, can't find the end (closing quoute) of data entry or
something like this, MLDBM error.
However, I think that is the result of killing the process while it had the
db open.
I can add more logging, but the logfile is liable to fill the disk before
the error occurs.


>



 
Reply With Quote
 
gnari
Guest
Posts: n/a
 
      03-07-2004
"TF" <(E-Mail Removed)> wrote in message news:404a8b09$(E-Mail Removed)...
>
>
> I've been basically doing this for two weeks, but it takes a whole day to
> get to this point and the data from the last request and last stored plce
> isn't helping much.
> Once I kill the process, the dbm is screwed so I can't really see

anything.
> All it says is, can't find the end (closing quoute) of data entry or
> something like this, MLDBM error.
> However, I think that is the result of killing the process while it had

the
> db open.
> I can add more logging, but the logfile is liable to fill the disk before
> the error occurs.


you might want to try rotating logs, each one containing one request
cycle. similarly, you could make a copy of your database after each
successfull request.
(this would probably mean a close/backup/reopen sequence that
will probably kill your performance, but I suggest this just as a
debugging tool. or you could do this after every 10 requests)

gnari




 
Reply With Quote
 
Bob Walton
Guest
Posts: n/a
 
      03-07-2004
TF wrote:

....
> I've been basically doing this for two weeks, but it takes a whole day to
> get to this point and the data from the last request and last stored plce
> isn't helping much.
> Once I kill the process, the dbm is screwed so I can't really see anything.
> All it says is, can't find the end (closing quoute) of data entry or
> something like this, MLDBM error.
> However, I think that is the result of killing the process while it had the
> db open.
> I can add more logging, but the logfile is liable to fill the disk before


> the error occurs.


It sounds to me like you just need some basic debugging techniques. If
you are (as previous notes in the thread stated) using LWP to grab info
from a whole bunch of web pages, make a simple log file that writes out
the URL of every page accessed. Set the log file filehandle to
autoflush ($|=1 so you don't miss the last bufferful when you
terminate the process.

Then, when you know the URL that's generating the problem, retrieve just
that page and use the Perl debugger to see what's up with your code on
that data.

You might note that LWP by default should timeout after 180 seconds, so
if there is a web page that's not responding, the job should still
proceed, albeit slowly. You could change the timeout if it is just an
insufficient patience problem.

Also, you might check to see if the DBM-type implementation you are
using is compatible with your data. Some of the implementations (like
SDMB, for example) have limitations on key and value lengths which MLDBM
could very easily exceed. Consider using DB_File instead (if you aren't
already). You'll probably have to refer to the docs for the specific
DBM-type implementation you are using on your specific platform and OS.

Finally, note that you could close the MLDBM connection in a $SIG{INT}
handler. That way you won't corrupt your database when you kill your
program via the console, assuming you follow proper file locking
protocols in your program and in the signal handler.

HTH.
--
Bob Walton
Email: http://bwalton.com/cgi-bin/emailbob.pl

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Database Database Database Database scott93727@gmail.com Computer Information 0 09-27-2012 02:43 AM
DataBase DataBase DataBase DataBase scott93727@gmail.com Computer Information 0 09-26-2012 09:40 AM
How keep python socket alive for ever by setting Keep alive flag. hisan Python 1 06-25-2012 05:30 PM
"Keep connection alive" program chuckcar Computer Support 20 04-24-2009 01:06 PM
fork(): how can I kill forked process and all ITS children, but leaveparent alive? Alexander N. Spitzer C Programming 1 11-12-2004 01:25 PM



Advertisments