Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Re: write to the same file from multiple processes at the same time?

Reply
Thread Tools

Re: write to the same file from multiple processes at the same time?

 
 
Jp Calderone
Guest
Posts: n/a
 
      05-27-2005
On 27 May 2005 06:21:21 -0700, Paul Rubin <"http://phr.cx"@nospam.invalid> wrote:
>Peter Hansen <(E-Mail Removed)> writes:
>> The OP was probably on the right track when he suggested that things
>> like SQLite (conveniently wrapped with PySQLite) had already solved
>> this problem.

>
>But they haven't. They depend on messy things like server processes
>constantly running, which goes against the idea of a cgi that only
>runs when someone calls it.


SQLite is an in-process dbm.

Jp
 
Reply With Quote
 
 
 
 
Paul Rubin
Guest
Posts: n/a
 
      05-27-2005
Jp Calderone <(E-Mail Removed)> writes:
> >But they haven't. They depend on messy things like server processes
> >constantly running, which goes against the idea of a cgi that only
> >runs when someone calls it.

>
> SQLite is an in-process dbm.


http://www.sqlite.org/faq.html#q7

(7) Can multiple applications or multiple instances of the same
application access a single database file at the same time?

Multiple processes can have the same database open at the same
time. Multiple processes can be doing a SELECT at the same
time. But only one process can be making changes to the database
at once.

But multiple processes changing the database simultaneously is
precisely what the OP wants to do.
 
Reply With Quote
 
 
 
 
Dennis Lee Bieber
Guest
Posts: n/a
 
      05-27-2005
On Fri, 27 May 2005 14:17:58 GMT, Jp Calderone <(E-Mail Removed)>
declaimed the following in comp.lang.python:

Not a Python solution -- my apologies...

> That said, I think an RDBM is a ridiculously complex solution to this simple problem. A filesystem lock, preferably using the directory or symlink trick (but flock() is fun too, if you're into that sort of thing), is clearly the solution to go with here.
>

About 15 years ago, on my Amiga, I'd be using ARexx and message
ports.

The processes wouldn't be opening a log file. They'd be on the
nature of (psuedoARexx):


if ~show("P", "Logger") then
do
say "Logger process is not available"
/* should add code to start logger */
exit
end

/* do work */

address "Logger" "This is a message to be Logged"

/* do work */

-----
The logger process is a bit more complex

/* make sure support library is loaded */
if ~show("L", "rexxsupport.library") then
do
if ~addlib("rexxsupport.library", 0, -30, 0) then
do
say "Unable to load library"
exit
end
end

if ~openport("Logger") then
do
say "Unable to create logger port -- is it already running?"
exit
end

call open("logfile", "path/to/log/file", "APPEND")

do while waitpkt("Logger")
pkt = getpkt("Logger")
cmd = getarg(pkt) /* depending on invocation, may have */
/* multiple arguments: getarg(pkt, #) */
call reply(pkt, 0) /* 0 = return code, no error */
call writeln("logfile", cmd)
end


Believe it or not, the above /would/ be the simple method for
the Amiga OS <G> Message ports were kernel ("exec") level entities
implemented as queues, and write/read were protected operations. The
ARexx "address" command used them with ARexx specific overhead padded
into the message. The rexxsupport library added functions to let ARexx
scripts create named ARexx compatible message ports and retrieve
messages from them.

Nearest match for message ports, that I've seen, is the VMS
"mailbox" -- though mailboxes can be multiple readers, and have their
own buffers (The Amiga did not have process protected memory, so the
message ports really just had the memory address of the message buffer
itself -- the reply(message) is needed to let the process know when the
buffer is available for reuse).

I don't know if WinNT "named pipes" can be used in a similar
scheme; I do know W9x doesn't support creating "named pipes" -- one
would be down to writing TCP/IP socket operations to emulate the
ability... Much too complex for simple clients -- note how the clients,
above, aren't using /any/ low-level operations, just regular ARexx
statements ("address" is part of REXX). Only the logger itself needed
low-level support. [Non-ARexx programs could link the ARexx system to
gain call level access to the ports -- heck, as I recall, Irmen had
added ARexx support to the old AmigaPython port]

--
> ================================================== ============ <
> http://www.velocityreviews.com/forums/(E-Mail Removed) | Wulfraed Dennis Lee Bieber KD6MOG <
> (E-Mail Removed) | Bestiaria Support Staff <
> ================================================== ============ <
> Home Page: <http://www.dm.net/~wulfraed/> <
> Overflow Page: <http://wlfraed.home.netcom.com/> <

 
Reply With Quote
 
Peter Hansen
Guest
Posts: n/a
 
      05-27-2005
Paul Rubin wrote:
> http://www.sqlite.org/faq.html#q7

[snip]
> Multiple processes can have the same database open at the same
> time. Multiple processes can be doing a SELECT at the same
> time. But only one process can be making changes to the database
> at once.
>
> But multiple processes changing the database simultaneously is
> precisely what the OP wants to do.


What isn't described in the above quote from the FAQ is how SQLite
*protects* your data from corruption in this case, unlike the "raw"
approach where you just use file handles.

And PySQLite conveniently wraps the relevant calls with retries when the
database is "locked" by the writing process, making it roughly a
no-brainer to use SQLite databases as nice simple log files where you're
trying to write from multiple CGI processes like the OP wanted.

Disclaimer: I haven't actually done that myself, and have only started
playing with pysqlite2 a day ago, but I have spent a fair bit of time
experimenting and reading the relevant docs and I believe I've got this
all correct.

-Peter
 
Reply With Quote
 
Paul Rubin
Guest
Posts: n/a
 
      05-27-2005
Peter Hansen <(E-Mail Removed)> writes:
> And PySQLite conveniently wraps the relevant calls with retries when
> the database is "locked" by the writing process, making it roughly a
> no-brainer to use SQLite databases as nice simple log files where
> you're trying to write from multiple CGI processes like the OP wanted.


Oh, ok. But what kind of locks does it use?
 
Reply With Quote
 
Peter Hansen
Guest
Posts: n/a
 
      05-27-2005
Paul Rubin wrote:
> Peter Hansen <(E-Mail Removed)> writes:
>
>>And PySQLite conveniently wraps the relevant calls with retries when
>>the database is "locked" by the writing process, making it roughly a
>>no-brainer to use SQLite databases as nice simple log files where
>>you're trying to write from multiple CGI processes like the OP wanted.

>
> Oh, ok. But what kind of locks does it use?


I think the FAQ can answer that better than I can, since I'm not sure
whether you're asking about any low-level (OS) locks it might use or
higher-level (e.g. database-level locking) that it might use. In
summary, however, at the database level it provides only coarse-grained
locking on the entire database. It *is* supposed to be a relatively
simple/lightweight solution compared to typical RDBMSes...

(There's also an excrutiating level of detail about this whole area in
the page at http://www.sqlite.org/lockingv3.html ).

-Peter
 
Reply With Quote
 
Paul Rubin
Guest
Posts: n/a
 
      05-27-2005
Peter Hansen <(E-Mail Removed)> writes:
> I think the FAQ can answer that better than I can, since I'm not sure
> whether you're asking about any low-level (OS) locks it might use or
> higher-level (e.g. database-level locking) that it might use. In
> summary, however, at the database level it provides only
> coarse-grained locking on the entire database. It *is* supposed to be
> a relatively simple/lightweight solution compared to typical RDBMSes...


Compared to what the OP was asking for, which was a way to synchronize
appending to a serial log file, SQlite is very complex. It's also
much more complex than (say) the dbm module, which is what Python apps
normally use as a lightweight db.

> (There's also an excrutiating level of detail about this whole area in
> the page at http://www.sqlite.org/lockingv3.html ).


Oh ok, it says it uses some special locking system calls on Windows.
Since those calls aren't in the Python stdlib, it must be using C
extensions, which again means complexity. But it looks like the
built-in msvcrt module has ways to lock parts of files in Windows.

Really, I think the Python library is somewhat lacking in not
providing a simple, unified interface for doing stuff like this.
 
Reply With Quote
 
Mike Meyer
Guest
Posts: n/a
 
      05-28-2005
Paul Rubin <http://(E-Mail Removed)> writes:
> Really, I think the Python library is somewhat lacking in not
> providing a simple, unified interface for doing stuff like this.


It's got one. Well, three, actually.

The syslog module solves the problem quite nicely, but only works on
Unix. If the OP is working on Unix systems, that may be a good
solution.

The logging module has a SysLogHandler that talks to syslog on
Unix. It also has an NTEventLogHandler for use on NT. I'm not familiar
with NT's event log, but I presume it has the same kind of
functionality as Unix's syslog facility.

<mike
--
Mike Meyer <(E-Mail Removed)> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Two processes writing to the same XML file at the same time? darrel ASP .Net 2 04-05-2006 05:30 PM
write to the same file from multiple processes at the same time? gabor Python 19 05-31-2005 01:57 PM
Re: write to the same file from multiple processes at the same time? Jp Calderone Python 4 05-31-2005 07:54 AM
Re: write to the same file from multiple processes at the same time? Jp Calderone Python 0 05-27-2005 02:17 PM
running multiple threads and multiple processes at the same time Smegly C Programming 1 05-19-2004 02:59 AM



Advertisments