Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Perl > Perl Misc > Parallel Parser IPC and IO - Need Direction

Reply
Thread Tools

Parallel Parser IPC and IO - Need Direction

 
 
dwknapp
Guest
Posts: n/a
 
      04-09-2004
Here is an outline of what I'm trying to do:
1. Gather a list of data text files.
2. Create say 4 child process' and divi up the file list to the
children.
3. Parent will read the parsed output.
4. Each child will open and parse its files and return the output to
the parent. Each file parsed by the child will output multiple lines
of data.
5. Parent sends the data to an output file.

Here is a dumbed down version of my code:

use IO::Select;
use Symbol;
use IO::File;
use IO::Handle;

my $sel = new IO::Select;
my $parent = 1;
my $num_children = 0;
my @filelist;
my @holdfiles;
my $files_per_proc = sprintf("%d", @filelist/4);

while (@holdfiles = splice(@filelist, 0, $files_per_proc)) {
my ($reader, $writer) = (gensym, gensym);
pipe($reader, $writer) or die "pipe: $!";

my $kid = fork();
if ($kid) {
close $writer;
$sel->add($reader);
$num_children += 1;
}
else {
close $reader;
gen_sub($writer, \@holdfiles, \@opts);
$parent = 0;
last;
}
}

while (1) {
foreach my $client ($sel->can_read()) {
my $line = <$client>;
if (!$line) {
$sel->remove($client);
next;
}
print $line; # STDOUT HERE BUT MAYBE EVENTUALLY A FILEHANDLE
}
if ($sel->count == 0) {
last;
}
}

if ($parent) {
while ($num_children) {
my $dead_child = wait;
$num_children--;
}
}

exit(0);

sub gen_sub {
my ($writer, $ra_holdfiles, $ra_options) = @_;

foreach my $file (@{$ra_holdfiles}) {
die "Error" unless (open(IN, $file));
while (my $line = <IN>) {
LOTS OF PARSING HERE

print $writer "SOME DATA THAT I'VE PARSED\n"; # MANY
TIMES PER FILE
}
close IN;
}
}

This may seem to work ok but the output of the parent is all garbaged
up. I think my problem is entirely to do with buffered data but I'm
really stuck. I'm seeing incomplete output from one child, multiple
child outputs on one line, and then also good lines. I know I can
have the children output their data to files but this is NOT what I
want to do. Any direction, help, or suggestions anyone can offer
would be greatly appreciated.

Thanks, -Dave.
 
Reply With Quote
 
 
 
 
dwknapp
Guest
Posts: n/a
 
      04-20-2004
For anyone else who is trying to do the same... I figured out what was wrong.

> use IO::Select;
> use Symbol;
> use IO::File;
> use IO::Handle;
>
> my $sel = new IO::Select;
> my $parent = 1;
> my $num_children = 0;
> my @filelist;
> my @holdfiles;
> my $files_per_proc = sprintf("%d", @filelist/4);
>
> while (@holdfiles = splice(@filelist, 0, $files_per_proc)) {
> my ($reader, $writer) = (gensym, gensym);
> pipe($reader, $writer) or die "pipe: $!";
>
> my $kid = fork();
> if ($kid) {
> close $writer;
> $sel->add($reader);
> $num_children += 1;
> }
> else {
> close $reader;
> gen_sub($writer, \@holdfiles, \@opts);
> $parent = 0;
> last;
> }
> }


> if ($parent) { # MOVED SO ONLY THE PARENT READS
>
> # HERE'S WHERE THE PROBLEM WAS. ONLY THE PARENT SHOULD BE READING. NOW THIS > # SCRIPT WORKS LIKE A CHAMP.
> while (1) {
> foreach my $client ($sel->can_read()) {
> my $line = <$client>;
> if (!$line) {
> $sel->remove($client);
> next;
> }
> print $line; # STDOUT HERE BUT MAYBE EVENTUALLY A FILEHANDLE
> }
> if ($sel->count == 0) {
> last;
> }
> }
>
> while ($num_children) {
> my $dead_child = wait;
> $num_children--;
> }
> }
>
> exit(0);
>
> sub gen_sub {
> my ($writer, $ra_holdfiles, $ra_options) = @_;
>
> foreach my $file (@{$ra_holdfiles}) {
> die "Error" unless (open(IN, $file));
> while (my $line = <IN>) {
> LOTS OF PARSING HERE
>
> print $writer "SOME DATA THAT I'VE PARSED\n"; # MANY
> TIMES PER FILE
> }
> close IN;
> }
> }
>

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Need some IPC pointers Andrew Berg Python 8 12-06-2011 02:05 PM
Re: Parallel in, Parallel out shift register Vivek Menon VHDL 0 06-10-2011 10:15 PM
Parallel in, Parallel out shift register Vivek Menon VHDL 5 06-08-2011 03:56 PM
How to use ruby, csv and webservice? Need a direction. Nick da G Ruby 2 10-08-2008 07:05 PM
Parallel port control with USB->Parallel converter Soren Python 4 02-14-2008 03:18 PM



Advertisments