Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Re: named pipe and Linux

Reply
Thread Tools

Re: named pipe and Linux

 
 
Cameron Simpson
Guest
Posts: n/a
 
      04-07-2009
On 07Apr2009 10:08, akineko <(E-Mail Removed)> wrote:
| I'm trying to use named pipes to fuse a Python program and a C
| program.
| One side creates pipes using os.mkfifo() and both sides use the same
| named pipes (one side reads, another side writes). The read side uses
| select.select() to wait for incoming messages and read the message
| when select.select() says it is ready.
| The length of the message is unknown to the read side.

That's a serious flaw in the message protocol.

| I cannot use file.read() because it will block waiting for an EOF.
| I cannot use file.readline() because how many lines have arrived is
| unknown.
| So, I needed to use os.read() with the exact number of characters to
| read.

No!

You should use os.read() with the maximum size of a message.
It _should_ return with the number of bytes in the message, provided the
C program writes messages with a single OS-level write() call.

Forget all the fstat() stuff - it's racy.

Personally, I'd use a thread to just do continuous blocking os.read()s of
the pipe, and putting the resulting messages on a Queue for collection
by your main program. If you're the only consumer of a Queue it's safe
to poll it for emptiness or not, or to use a no-wait get().

All the above is untested, but absent a size in the protocol or other
ways of parsing message boundaries in data stream, you can only rely on
the C program writing messages with a single write() and collect using a
large os.read(), which should return with what is there.

Cheers,
--
Cameron Simpson <(E-Mail Removed)> DoD#743
http://www.cskk.ezoshosting.com/cs/

Language... has created the word "loneliness" to express the pain of
being alone. And it has created the word "solitude" to express the glory
of being alone. - Paul Johannes Tillich
 
Reply With Quote
 
 
 
 
akineko
Guest
Posts: n/a
 
      04-07-2009
Hello Cameron Simpson,

Thank you for your reply.
I now think the way I did (using fstat()) was a very bad idea (as you
pointed out).
I haven't decided how to fix this yet.
I also considered attaching the message length to the head of the
message.
It will work, too.
I need to a bit more experiment.

Thank you for providing me directions to solve this problem.

Best regards,
Aki Niimura

On Apr 7, 3:26 pm, Cameron Simpson <(E-Mail Removed)> wrote:
> On 07Apr2009 10:08, akineko <(E-Mail Removed)> wrote:
> | I'm trying to use named pipes to fuse a Python program and a C
> | program.
> | One side creates pipes using os.mkfifo() and both sides use the same
> | named pipes (one side reads, another side writes). The read side uses
> | select.select() to wait for incoming messages and read the message
> | when select.select() says it is ready.
> | The length of the message is unknown to the read side.
>
> That's a serious flaw in the message protocol.
>
> | I cannot use file.read() because it will block waiting for an EOF.
> | I cannot use file.readline() because how many lines have arrived is
> | unknown.
> | So, I needed to use os.read() with the exact number of characters to
> | read.
>
> No!
>
> You should use os.read() with the maximum size of a message.
> It _should_ return with the number of bytes in the message, provided the
> C program writes messages with a single OS-level write() call.
>
> Forget all the fstat() stuff - it's racy.
>
> Personally, I'd use a thread to just do continuous blocking os.read()s of
> the pipe, and putting the resulting messages on a Queue for collection
> by your main program. If you're the only consumer of a Queue it's safe
> to poll it for emptiness or not, or to use a no-wait get().
>
> All the above is untested, but absent a size in the protocol or other
> ways of parsing message boundaries in data stream, you can only rely on
> the C program writing messages with a single write() and collect using a
> large os.read(), which should return with what is there.
>
> Cheers,
> --
> Cameron Simpson <(E-Mail Removed)> DoD#743http://www.cskk.ezoshosting.com/cs/
>
> Language... has created the word "loneliness" to express the pain of
> being alone. And it has created the word "solitude" to express the glory
> of being alone. - Paul Johannes Tillich


 
Reply With Quote
 
 
 
 
Thomas Bellman
Guest
Posts: n/a
 
      04-08-2009
Cameron Simpson <(E-Mail Removed)> wrote:

> On 07Apr2009 10:08, akineko <(E-Mail Removed)> wrote:
>| I'm trying to use named pipes to fuse a Python program and a C
>| program.
>| One side creates pipes using os.mkfifo() and both sides use the same
>| named pipes (one side reads, another side writes). The read side uses
>| select.select() to wait for incoming messages and read the message
>| when select.select() says it is ready.
>| The length of the message is unknown to the read side.


> That's a serious flaw in the message protocol.


>| I cannot use file.read() because it will block waiting for an EOF.
>| I cannot use file.readline() because how many lines have arrived is
>| unknown.
>| So, I needed to use os.read() with the exact number of characters to
>| read.


> No!


> You should use os.read() with the maximum size of a message.
> It _should_ return with the number of bytes in the message, provided the
> C program writes messages with a single OS-level write() call.


No!

That's still broken. You can't know if the writer has managed to
write one or several messages onto the pipe. And if the messages
are large, you aren't guaranteed that the OS won't split up the
data into multiple segments that only become available to the
reader one or a few at a time.

You *need* to use a protocol where it is possible to determine
the message boundaries. You can do that by:

- Using messages of a fixed size.

- Terminate each message with an octet sequence that cannot occur
within a message. For example, a linefeed without a backslash
before it (and you would probably want a way to escape the
backslash, in case you want to end a message with a backslash).

- Have small header of a fixed size at the start of each message,
that includes the length of the message in octets.


--
Thomas Bellman, Lysator Computer Club, Linköping University, Sweden
"Life IS pain, highness. Anyone who tells ! bellman @ lysator.liu.se
differently is selling something." ! Make Love -- Nicht Wahr!
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
named pipe and Linux akineko Python 2 04-08-2009 01:33 PM
named pipe and "Text file busy" mikexf@gmail.com Perl Misc 3 11-28-2007 11:17 PM
named pipe problem on linux richard C++ 5 11-02-2004 07:44 PM
[named pipe] i wanna know about validate of pipe handle of client lee, wonsun C++ 1 11-02-2004 04:29 AM
Why does IO::Pipe::END generate an EXCEPT pipe message? lvirden@gmail.com Perl Misc 1 06-02-2004 02:17 PM



Advertisments