Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Java > Re: Capturing stdout from shell script

Reply
Thread Tools

Re: Capturing stdout from shell script

 
 
Nigel Wade
Guest
Posts: n/a
 
      03-15-2011
On 15/03/11 13:46, Mark wrote:
> Hi,
>
> We have the following code to call a shell script:
>
> Runtime rt = Runtime.getRuntime();
> Process proc = rt.exec(scriptName );
> int exitVal = proc.waitFor();
> log.info("Returned Value is: " + exitVal);
>
> How can get access to the output from the shell script (normally from
> stdout)?


You may be better off these days using ProcessBuilder, and combining
stdout and stderr using ProcessBuilder.redirectErrorStream(). Then all
you need do is use the Process.getInputStream() to read both stdout and
stderr.

If you stick to using Runtime.exec() you'll need to use threads to read
both the stdout and stderr simultaneously to avoid potential deadlock
due to the buffered nature of the streams.

--
Nigel Wade

 
Reply With Quote
 
 
 
 
Lawrence D'Oliveiro
Guest
Posts: n/a
 
      03-15-2011
In message <(E-Mail Removed)>, Nigel Wade wrote:

> If you stick to using Runtime.exec() you'll need to use threads to read
> both the stdout and stderr simultaneously to avoid potential deadlock
> due to the buffered nature of the streams.


Don’t you have select(2) <http://docs.python.org/py3k/library/select.html>?
 
Reply With Quote
 
 
 
 
Andreas Leitgeb
Guest
Posts: n/a
 
      03-16-2011
Lawrence D'Oliveiro <(E-Mail Removed)_zealand> wrote:
> In message <(E-Mail Removed)>, Nigel Wade wrote:
>> If you stick to using Runtime.exec() you'll need to use threads to read
>> both the stdout and stderr simultaneously to avoid potential deadlock
>> due to the buffered nature of the streams.

> Don’t you have select(2) <http://docs.python.org/py3k/library/select.html>?


Iirc, with nio Java offers also the non-blocking approach to I/O.
Haven't yet used it myself, yet, as I do such tasks in Tcl

Based on how fragile thread-programming can be (forget one "synchronized"
and you end up reading a local CPU's cache instead of shared memory. Put
bad ones in and face bad performance or deadlocks), I understand this
wish for select-based IO - yet it *is* already available in Java.

Read up on java.nio.channels.Selector for a start.

PS: to be fair: event-based IO also has its pitfalls and hardships,
just different ones.
 
Reply With Quote
 
Lawrence D'Oliveiro
Guest
Posts: n/a
 
      03-17-2011
In message <(E-Mail Removed)>, Andreas Leitgeb
wrote:

> Read up on java.nio.channels.Selector for a start.


<http://developer.android.com/reference/java/nio/channels/Selector.html>

Bloody hell. How many different layers of classes and calls do you need in
Java to implement such a simple concept?

The underlying POSIX service is so simple
<http://linux.die.net/man/2/select>: pass 3 bitmasks of file descriptors to
wait for, plus an optional timeout. Python keeps this simplicity
<http://docs.python.org/py3k/library/select.html>: pass 3 sequences of
objects which can be anything you like, so long as they have valid “fileno”
methods.
 
Reply With Quote
 
Andreas Leitgeb
Guest
Posts: n/a
 
      03-17-2011
Lawrence D'Oliveiro <(E-Mail Removed)_zealand> wrote:
> In message <(E-Mail Removed)>, Andreas Leitgeb wrote:
>> Read up on java.nio.channels.Selector for a start.

> <http://developer.android.com/reference/java/nio/channels/Selector.html>
> Bloody hell. How many different layers of classes and calls do you need in
> Java to implement such a simple concept?


"simple". Yes, that describes it. Too simple in some cases, where lots of
filedescriptors are totally used by the app. (select scales badly with the
value of the latest filedescriptor, even if only a small number of fds is
really selected on.) There's also "poll" which is not that "simple", but
doesn't care if the fds you wait for are 3,4,5 or 1000,4042,4043.

I have no idea, which of select or poll is used with java's nio Selector,
but I believe that it is not user's concern, anyway. Just because the user
doesn't get to build up bitsets, himself, its probably a good abstraction
that would allow even for a dynamic (or at least system-dependent) choice
between select and poll.

Tcl also uses select, internally, but the script-interface hides this away
so it could probably be rewritten for poll any time. Also, it allows for
easier porting to systems, where such notifications for channels are handled
entirely different.

 
Reply With Quote
 
Lawrence D'Oliveiro
Guest
Posts: n/a
 
      03-18-2011
In message <(E-Mail Removed)>, Andreas Leitgeb
wrote:

> Lawrence D'Oliveiro <(E-Mail Removed)_zealand> wrote:
>
>> <http://developer.android.com/reference/java/nio/channels/Selector.html>
>> Bloody hell. How many different layers of classes and calls do you need
>> in Java to implement such a simple concept?

>
> "simple". Yes, that describes it. Too simple in some cases, where lots of
> filedescriptors are totally used by the app. (select scales badly with the
> value of the latest filedescriptor, even if only a small number of fds is
> really selected on.)


Really? I’ve written servers in Perl, C++ and Python using select, and its
performance was never a problem; any bottleneck turned out to be elsewhere.

> I have no idea, which of select or poll is used with java's nio Selector,
> but I believe that it is not user's concern, anyway. Just because the
> user doesn't get to build up bitsets, himself, its probably a good
> abstraction that would allow even for a dynamic (or at least
> system-dependent) choice between select and poll.


Python manages to avoid the bitsets in a much simpler way.

> Tcl also uses select, internally ...


Tcl is an antiquated language, where arrays are not even first-class
objects.
 
Reply With Quote
 
Andreas Leitgeb
Guest
Posts: n/a
 
      03-18-2011
Lawrence D'Oliveiro <(E-Mail Removed)_zealand> wrote:
>> [Andreas Leitgeb wrote about limitations of select]

> Really? I’ve written servers in Perl, C++ and Python using select, and its
> performance was never a problem; any bottleneck turned out to be elsewhere.

What were the access-rates on those servers? How many fds did it ever have
open at the same time? (including fds that were not incoming connections)

> Python manages to avoid the bitsets in a much simpler way.

That sounds good. Maybe it is closer to tcl, than I thought it was
after reading your mention of explicit "select" in python?

>> Tcl also uses select, internally ...

> Tcl is an antiquated language, where arrays are not even first-class
> objects.

Apart from that it is a bogus statement, it also doesn't address
Tcl's approach to event-based programming, namely hiding that ugly
select system-call beyond recognition, which was the only reason
I brought it up here.

 
Reply With Quote
 
Michael Wojcik
Guest
Posts: n/a
 
      03-18-2011
Lawrence D'Oliveiro wrote:
> In message <(E-Mail Removed)>, Andreas Leitgeb
> wrote:
>
> Really? I’ve written servers in Perl, C++ and Python using select, and its
> performance was never a problem; any bottleneck turned out to be elsewhere.


The scalability problems with select are well-documented. See for
example Stevens, _UNIX Network Programming_ 2nd ed vol 1, 6.3; that
also refers you to Wright & Stevens, _TCP/IP Illustrated_ vol 2,
16.13, which shows an implementation of select that may be enlightening.

UNP 6.3 also notes that in some implementations attempting to enlarge
the select FD_SETSIZE does not work correctly because the sets were
still restricted in the kernel; the Standard Unix Specification (v3)
does not require that implementations permit changing FD_SETISZE.

UNP 6.10 notes that poll() does not suffer from the latter problem.

Another discussion may be found at [1], which notes that neither
select() nor poll() scale very well to thousands of descriptors;
that's why OS vendors started introducing other mechanisms, such as
/dev/poll, in the mid-1990s.

Another example is the "Heuristic 2" discussion in the Winsock FAQ's
discussion of I/O strategies.[2] Of course, select in Winsock uses an
array of handles rather than a bitmask, so it's closer to Unix's
poll() than to the traditional select() in its implementation.

Or see the discussion at [3], particularly where Barry Margolin points
out that there are different scaling tradeoffs between select() and
poll(). If the descriptor set is large and dense, select() is more
efficient in switching between user and kernel mode; if either of
those doesn't hold, poll() is more efficient at that point.

The scalability issues with select() ought to be obvious. There's the
limit of FD_SETSIZE, in implementations where that limit is in the
kernel as well. (If it isn't, you can work around that at the
application level.) There's the cost of copying FD_SETs between user
and kernel space. There's the cost of linear scans of the FD_SETs. The
latter two are particularly wasteful if the sets are sparse; that's
the problem that poll() fixes, but at the cost of using far more
storage per descriptor, so whether you benefit from poll() depends on
just which descriptors you're using.

On Windows, there's just select(), and it's effectively poll(), and
(as the Winsock FAQ explains) it's pretty much the worst of the
available choices.

For a handful of clients, even up to several hundred, it likely
doesn't matter; chances are good that I/O multiplexing won't dominate
processing time, even though it adds a lot of looping within the main
loop. When a server has to handle thousands of simultaneous clients
it's a different story. I have (proprietary) profiling data from real,
commercial servers showing significant performance issues with
select() at those kinds of loads.


[1] http://www.kegel.com/c10k.html
[2] http://tangentsoft.net/wskfaq/articl...trategies.html
[3]
https://groups.google.com/group/comp...cdcda1e6c216f1

--
Michael Wojcik
Micro Focus
Rhetoric & Writing, Michigan State University
 
Reply With Quote
 
Lawrence D'Oliveiro
Guest
Posts: n/a
 
      03-18-2011
In message <(E-Mail Removed)>, Michael Wojcik wrote:

> UNP 6.10 notes that poll() does not suffer from the latter problem.


Fine. I’ve never used poll, but Python supports that too if you want, with
just one class and four methods
<http://docs.python.org/py3k/library/select.html#poll-objects>.

Contrast that to the number of different classes that Java requires
<http://developer.android.com/reference/java/nio/channels/Selector.html>:
Selector (9 methods), SelectableChannel (8 methods), SelectionKey (another
dozen methods), SelectorProvider ... I’m losing count.
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: How to pass shell variable to shell script from python D'Arcy J.M. Cain Python 0 02-27-2008 01:56 PM
Re: How to pass shell variable to shell script from python Gerardo Herzig Python 1 02-27-2008 12:19 PM
Re: How to pass shell variable to shell script from python Christian Heimes Python 0 02-27-2008 10:53 AM
execute a shell script in a shell script moongeegee Perl Misc 2 12-04-2007 12:18 AM
shell script stdout to python string? Jeff Learman Python 2 05-20-2004 09:25 PM



Advertisments