Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Ruby > Asynchronous process to process pipe IO

Reply
Thread Tools

Asynchronous process to process pipe IO

 
 
Pito Salas
Guest
Posts: n/a
 
      08-24-2009
This is slightly subtle question about how a process that reads from a
pipe that is being written to by another process may run asynchronously.

Here's a code snippet:

def start_processing
open("|-", "r") do |worker|
if worker
# here we are in the parent
i = 0
worker.each_line do |line|
puts "line #{i}
i = i+1
end
else
# here we are in child thread
exec("./iacommand.rb")
end
end
end

iacommand.rb is invoked as a process within ruby. It does a lot of
processing, outputs a line, does a more processing, outputs another
line, and so on.

I would like to have the "puts" calls occur right when the a line is
generated by the script running in the process.

The way it is written above, what happens is that the app waits until
iacommand.rb exits and then calls the series of puts in immediate
succession.

So in other words, how do I asynchronously read from a pipe that is
being fed by a process?
--
Posted via http://www.ruby-forum.com/.

 
Reply With Quote
 
 
 
 
Joel VanderWerf
Guest
Posts: n/a
 
      08-24-2009
Pito Salas wrote:
> This is slightly subtle question about how a process that reads from a
> pipe that is being written to by another process may run asynchronously.
>
> Here's a code snippet:
>
> def start_processing
> open("|-", "r") do |worker|
> if worker
> # here we are in the parent
> i = 0
> worker.each_line do |line|
> puts "line #{i}
> i = i+1
> end
> else
> # here we are in child thread
> exec("./iacommand.rb")
> end
> end
> end
>
> iacommand.rb is invoked as a process within ruby. It does a lot of
> processing, outputs a line, does a more processing, outputs another
> line, and so on.
>
> I would like to have the "puts" calls occur right when the a line is
> generated by the script running in the process.
>
> The way it is written above, what happens is that the app waits until
> iacommand.rb exits and then calls the series of puts in immediate
> succession.
>
> So in other words, how do I asynchronously read from a pipe that is
> being fed by a process?


Hm, the following shows output at 1 sec intervals:

open("|-", "r") do |worker|
if worker
i = 0
worker.each_line do |line|
puts "line #{i}:#{line}"
i = i+1
end
else
exec("ruby -e '3.times {|i| p i; sleep 1}'")
end
end

Does it work that way for you?

--
vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407

 
Reply With Quote
 
 
 
 
Pito Salas
Guest
Posts: n/a
 
      08-24-2009
Joel VanderWerf wrote:
> Pito Salas wrote:
>> worker.each_line do |line|
>> iacommand.rb is invoked as a process within ruby. It does a lot of
>> So in other words, how do I asynchronously read from a pipe that is
>> being fed by a process?

>
> Hm, the following shows output at 1 sec intervals:
>
> open("|-", "r") do |worker|
> if worker
> i = 0
> worker.each_line do |line|
> puts "line #{i}:#{line}"
> i = i+1
> end
> else
> exec("ruby -e '3.times {|i| p i; sleep 1}'")
> end
> end
>
> Does it work that way for you?


Yes it does. Double-hm.

My case is using a second ruby file, iacommand.rb, which is doing much
the same as what your -e is doing. I wonder if that makes the
difference? Or it must be something else.
--
Posted via http://www.ruby-forum.com/.

 
Reply With Quote
 
Pito Salas
Guest
Posts: n/a
 
      08-24-2009
Pito Salas wrote:

> Yes it does. Double-hm.
>
> My case is using a second ruby file, iacommand.rb, which is doing much
> the same as what your -e is doing. I wonder if that makes the
> difference? Or it must be something else.


It does seem to be the difference between ruby -e and ./iacommand.rb.
Can anyone see what the key is??

Thanks!!

Here's what fails (that is, blocks) (tested both in irb and in eclipse)

open("|-", "r") do |worker|
if worker
i = 0
worker.each_line do |line|
puts "line #{i}:#{line}"
i = i+1
end
else
exec("./iacommand.rb", "-t")
end
end


And here's the exact text of iacommand.rb:


require 'rubygems'
require 'getoptlong'

parser = GetoptLong.new
parser.set_options(
["-h", "--help", GetoptLong::NO_ARGUMENT],
["-t", "--test", GetoptLong::NO_ARGUMENT],
["-v", "--version", GetoptLong::NO_ARGUMENT])

valid = false
loop do

begin
opt, arg = parser.get
break if not opt
case opt
when "-h"
puts "Usage: ..."
valid = true
break
when "-t"
valid = true
break
when "-v"
puts "Version 0.0"
valid = true
break
end
end
end

if valid
puts "start"
i = 0
5.times do
puts "ballot #{i}"
delay = rand(5).to_i
sleep delay
i = i+1
end
puts "exit"
else
puts "invalid parameters for iacommand"
end




--
Posted via http://www.ruby-forum.com/.

 
Reply With Quote
 
Robert Klemme
Guest
Posts: n/a
 
      08-24-2009
On 24.08.2009 22:16, Joel VanderWerf wrote:
> Pito Salas wrote:
>> This is slightly subtle question about how a process that reads from a
>> pipe that is being written to by another process may run asynchronously.
>>
>> Here's a code snippet:
>>
>> def start_processing
>> open("|-", "r") do |worker|
>> if worker
>> # here we are in the parent
>> i = 0
>> worker.each_line do |line|
>> puts "line #{i}
>> i = i+1
>> end
>> else
>> # here we are in child thread
>> exec("./iacommand.rb")
>> end
>> end
>> end
>>
>> iacommand.rb is invoked as a process within ruby. It does a lot of
>> processing, outputs a line, does a more processing, outputs another
>> line, and so on.
>>
>> I would like to have the "puts" calls occur right when the a line is
>> generated by the script running in the process.
>>
>> The way it is written above, what happens is that the app waits until
>> iacommand.rb exits and then calls the series of puts in immediate
>> succession.
>>
>> So in other words, how do I asynchronously read from a pipe that is
>> being fed by a process?

>
> Hm, the following shows output at 1 sec intervals:
>
> open("|-", "r") do |worker|
> if worker
> i = 0
> worker.each_line do |line|
> puts "line #{i}:#{line}"
> i = i+1
> end
> else
> exec("ruby -e '3.times {|i| p i; sleep 1}'")
> end
> end
>
> Does it work that way for you?


Why are you guys using such a complex construction? If you just want to
see the output when it comes why not just:

system "./iacommand.rb"

Or maybe this, if you need to process the output:

IO.popen "./iacommand.rb" do |io|
io.each_line do |line|
puts line
end
end

Note, you may have to do $stdout.sync = true in "iacommand.rb".

Kind regards

robert

--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/
 
Reply With Quote
 
Pito Salas
Guest
Posts: n/a
 
      08-24-2009
Robert Klemme wrote:
> On 24.08.2009 22:16, Joel VanderWerf wrote:
>>> i = 0
>>>
>>>

>> i = i+1
>> end
>> else
>> exec("ruby -e '3.times {|i| p i; sleep 1}'")
>> end
>> end
>>
>> Does it work that way for you?

>
> Why are you guys using such a complex construction? If you just want to
> see the output when it comes why not just:
>
> system "./iacommand.rb"
>
> Or maybe this, if you need to process the output:
>
> IO.popen "./iacommand.rb" do |io|
> io.each_line do |line|
> puts line
> end
> end
>
> Note, you may have to do $stdout.sync = true in "iacommand.rb".
>
> Kind regards
>
> robert


Robert,

Thanks, the $stdout.sync = true was the magic bullet.

Now, I am not exactly clear (reading the doc) about the difference
between exec and system... I do need to be able to send command line
arguments to iacommand.rb and I do indeed need to process the output 'as
it appears.'

Thanks!!

Pito

--
Posted via http://www.ruby-forum.com/.

 
Reply With Quote
 
Gary Wright
Guest
Posts: n/a
 
      08-24-2009

On Aug 24, 2009, at 4:54 PM, Pito Salas wrote:
>
> Now, I am not exactly clear (reading the doc) about the difference
> between exec and system... I do need to be able to send command line
> arguments to iacommand.rb and I do indeed need to process the output
> 'as
> it appears.'


exec() causes a new program to be executed within the current process.
Unless exec() fails (e.g. program not found), exec() will never 'return'
because the current program will be discarded (i.e. the Ruby
interpreter)
in favor of the newly exec'ed one.

system() causes a new (child) process to be created and for the new
program to be executed (i.e. exec'ed) within the new child process.
The current process will wait for the child process to terminate before
proceeding.

The difference is whether the new program replaces the currently
running program or not. Note that 'program' here means the Ruby
interpreter and not the particular Ruby script that is running.

Note: I'm answering from the perspective of a Unix/Posix environment.
My windows-fu isn't sufficient to say if my description is correct
for that environment, but it is probably pretty close.

Gary Wright

 
Reply With Quote
 
Mike Barsalou
Guest
Posts: n/a
 
      08-24-2009
On Aug 24, 1:05*pm, Gary Wright <(E-Mail Removed)> wrote:
> On Aug 24, 2009, at 4:54 PM, Pito Salas wrote:
>
>
>
> > Now, I am not exactly clear (reading the doc) about the difference
> > between exec and system... I do need to be able to send command line
> > arguments to iacommand.rb and I do indeed need to process the output *
> > 'as
> > it appears.'

>
> exec() causes a new program to be executed within the current process.
> Unless exec() fails (e.g. program not found), exec() will never 'return'
> because the current program will be discarded (i.e. the Ruby *
> interpreter)
> in favor of the newly exec'ed one.
>
> system() causes a new (child) process to be created and for the new
> program to be executed (i.e. exec'ed) within the new child process.
> The current process will wait for the child process to terminate before
> proceeding.
>
> The difference is whether the new program replaces the currently
> running program or not. Note that 'program' here means the Ruby
> interpreter and not the particular Ruby script that is running.
>
> Note: I'm answering from the perspective of a Unix/Posix environment.
> My windows-fu isn't sufficient to say if my description is correct
> for that environment, but it is probably pretty close.
>
> Gary Wright


Would Ruby's fork behave exactly like system here, or would it have a
third path? Likely fork wouldn't be an option for the OP because he
wants to process the output?

Mike B.
 
Reply With Quote
 
Gary Wright
Guest
Posts: n/a
 
      08-24-2009

On Aug 24, 2009, at 6:50 PM, Mike Barsalou wrote:
>
> Would Ruby's fork behave exactly like system here, or would it have a
> third path? Likely fork wouldn't be an option for the OP because he
> wants to process the output?


fork and exec are the 'primitive' operations.

system is basically implemented by forking and having the child
process exec the new program while the parent waits for it to
complete. System() is just a convenience method for this
common pattern.

Capturing the output of a program basically means using fork/exec
along with rearranging the standout output of the exec'ed
program. For simple cases this is handled by Ruby's backtick syntax:

output = `program_to_run`

Gary Wright

 
Reply With Quote
 
Robert Klemme
Guest
Posts: n/a
 
      08-25-2009
On 24.08.2009 22:54, Pito Salas wrote:
> Robert Klemme wrote:
>> On 24.08.2009 22:16, Joel VanderWerf wrote:
>>>> i = 0
>>>>
>>>>
>>> i = i+1
>>> end
>>> else
>>> exec("ruby -e '3.times {|i| p i; sleep 1}'")
>>> end
>>> end
>>>
>>> Does it work that way for you?

>> Why are you guys using such a complex construction? If you just want to
>> see the output when it comes why not just:
>>
>> system "./iacommand.rb"
>>
>> Or maybe this, if you need to process the output:
>>
>> IO.popen "./iacommand.rb" do |io|
>> io.each_line do |line|
>> puts line
>> end
>> end
>>
>> Note, you may have to do $stdout.sync = true in "iacommand.rb".
>>
>> Kind regards
>>
>> robert

>
> Robert,
>
> Thanks, the $stdout.sync = true was the magic bullet.
>
> Now, I am not exactly clear (reading the doc) about the difference
> between exec and system... I do need to be able to send command line
> arguments to iacommand.rb and I do indeed need to process the output 'as
> it appears.'


Then you'll want to use one of the popen family of functions. Even with
IO.popen you can pass arguments:

IO.popen ["./iacommand.rb", "arg", "arg"] do |io|
...
end

Note, if you also want to write to the process's stdin, you need to
provide a file mode as second argument. See docs.

Kind regards

robert

--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Asynchronous process from asp.net page button click event? Ritha Software 0 09-29-2009 02:20 PM
How to make the system.process as asynchronous from asp.net page button click event? Ritha Software 0 09-29-2009 07:17 AM
[named pipe] i wanna know about validate of pipe handle of client lee, wonsun C++ 1 11-02-2004 04:29 AM
Why does IO::Pipe::END generate an EXCEPT pipe message? lvirden@gmail.com Perl Misc 1 06-02-2004 02:17 PM
connection.execute = asynchronous process ?? Jill Graham ASP General 4 10-04-2003 12:38 AM



Advertisments