Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Perl > Perl Misc > Getting Net::SFTP handle takes FOR-FLIPPIN-EVER

Reply
Thread Tools

Getting Net::SFTP handle takes FOR-FLIPPIN-EVER

 
 
usenet@DavidFilmer.com
Guest
Posts: n/a
 
      07-07-2006
Kindly consider the following script which I am having a problem with:

#!/usr/bin/perl
use strict; use warnings;
use Net::SFTP;

my %args = qw{ user myuserid password mypassword debug 1 };
$args{'ssh_args'} = [protocol => 2];

my $sftp = Net::SFTP->new('123.456.78.90', %args) or die $!;

__END__

The script works fine, but on one particular machine it takes about a
minute to get the SFTP handle. The identical script runs instantly on
another (inferior) machine.

So, naturally, I would suspect a problem with that particular machine
that would have nothing to do with my script (maybe a routing issue,
etc), EXCEPT two things make me think the problem somehow involves the
Net::SFTP module itself:

1. "sftp" from a shell (same parameters) connects instantly.
2. I can get a Net::SSH:erl handle (same %args) instantly.

With debug mode turned on, I observe that there are long pauses at
three points. I've posted the debug output (with pauses indicated)
below. I see no obvious reason for any performance problems (I'm
running on a new and lightly loaded p520 (POWER5) AIX box). As I said,
I can get a Net::SSH:erl handle instantly (and Net::SFTP uses N:S).

If anyone has any suggestions on what the problem might be with using
Net::SFTP from this particular machine, please let me know. Here's my
Net::SFTP debug output (with pauses indicated):

server: Reading configuration data /myuserid/.ssh/config
server: Reading configuration data /etc/ssh_config
server: Connecting to server123.mycompany.com, port 22.
server: Remote protocol ver 1.99, remote ver OpenSSH_3.6.1p2
server: Net::SSH:erl Version 1.29, protocol version 2.0.
server: No compat match: OpenSSH_3.6.1p2.
server: Connection established.
server: Sent key-exchange init (KEXINIT), wait response.
server: Algorithms, c->s: 3des-cbc hmac-sha1 none
server: Algorithms, s->c: 3des-cbc hmac-sha1 none

>>long pause............................................. ...........


server: Entering Diffie-Hellman Group 1 key exchange.
server: Sent DH public key, waiting for reply.
server: Received host key, type 'ssh-dss'.
server: Host 'server123.mycompany.com' is known and matches host key
server: Computing shared secret key.

>>long pause............................................. ...........


server: Verifying server signature.

>>long pause............................................. ...........


server: Waiting for NEWKEYS message.
server: Enabling incoming encryption/MAC/compression.
server: Send NEWKEYS, enable outgoing encryption/MAC/compression.
server: Sending request for user-authentication service.
server: Service accepted: ssh-userauth.
server: Trying empty user-authentication request.
server: Authentication methods that can continue:
publickey,password,keyboard-interactive.
server: Next method to try is publickey.
server: Next method to try is password.
server: Trying password authentication.
server: Login completed, opening dummy shell channel.
server: channel 0: new [client-session]
server: Requesting channel_open for channel 0.
server: channel 0: open confirm rwindow 0 rmax 32768
Establishing SFTP connection with server server123.mycompany.com:
channel 1: new [client-session]
server: Requesting channel_open for channel 1.
server: Sending subsystem: sftp
server: Requesting service subsystem on channel 1.
server: channel 1: open confirm rwindow 0 rmax 32768
server: sftp: Sending SSH2_FXP_INIT
server: sftp: Remote version: 3
SFTP connected

--
David Filmer (http://DavidFilmer.com)

 
Reply With Quote
 
 
 
 
Sisyphus
Guest
Posts: n/a
 
      07-07-2006

<(E-Mail Removed)> wrote in message
..
..
> server: Algorithms, s->c: 3des-cbc hmac-sha1 none
>
> >>long pause............................................. ...........

>
> server: Entering Diffie-Hellman Group 1 key exchange.
> server: Sent DH public key, waiting for reply.
> server: Received host key, type 'ssh-dss'.
> server: Host 'server123.mycompany.com' is known and matches host key
> server: Computing shared secret key.
>
> >>long pause............................................. ...........

>
> server: Verifying server signature.
>
> >>long pause............................................. ...........

>
> server: Waiting for NEWKEYS message.


Looks to me that the 2nd, 3rd, and possibly also the 1st, long pauses occur
when some mathematical computations take place. Could it be that on the fast
machine those computations are being performed by compiled (XS) code, but
are being performed by comparatively slow pure perl code on the slow machine
?

Cheers,
Rob


 
Reply With Quote
 
 
 
 
usenet@DavidFilmer.com
Guest
Posts: n/a
 
      07-07-2006
Sisyphus wrote:
> Could it be that on the fast machine those computations are being performed
> by compiled (XS) code, but are being performed by comparatively slow pure
> perl code on the slow machine


I would agree with your idea if Net::SSH:erl also had these delays on
the slow machine. That's a pure Perl implementation of ssh. And,
according to the docs, Net::SFTP uses Net::SSH:erl to broker its
connection.

This problem would make a LOT more sense if Net::SSH:erl showed
similar delays. But I get instant connections with that module, even
though that very same module is used to establish the Net::SFTP
connection (and it really is - I looked at the source).

I'm baffled! It makes NO sense! If anyone has any 'try this and see
what happens' suggestions then I'm open to testing anything! If anyone
has an IBM p620 (not a 520 as I typo'd earlier) or something similar
with AIX 5.3 then I'd appreciate it if you would try my script and tell
me your results.

--
David Filmer (http://DavidFilmer.com)

 
Reply With Quote
 
rahed
Guest
Posts: n/a
 
      07-07-2006
http://www.velocityreviews.com/forums/(E-Mail Removed) writes:

> I'm baffled! It makes NO sense! If anyone has any 'try this and see
> what happens' suggestions then I'm open to testing anything! If anyone
> has an IBM p620 (not a 520 as I typo'd earlier) or something similar
> with AIX 5.3 then I'd appreciate it if you would try my script and tell
> me your results.


Try to install Math::BigInt::GMP if it's not already there. That helped
when I met weird behaviour with long ssh authentication times.

--
Radek
 
Reply With Quote
 
xhoster@gmail.com
Guest
Posts: n/a
 
      07-07-2006
(E-Mail Removed) wrote:
> Sisyphus wrote:
> > Could it be that on the fast machine those computations are being
> > performed by compiled (XS) code, but are being performed by
> > comparatively slow pure perl code on the slow machine

>
> I would agree with your idea if Net::SSH:erl also had these delays on
> the slow machine. That's a pure Perl implementation of ssh. And,
> according to the docs, Net::SFTP uses Net::SSH:erl to broker its
> connection.
>
> This problem would make a LOT more sense if Net::SSH:erl showed
> similar delays. But I get instant connections with that module, even
> though that very same module is used to establish the Net::SFTP
> connection (and it really is - I looked at the source).


Look at the %INC to make sure it is the same path to the same module.

>
> I'm baffled! It makes NO sense! If anyone has any 'try this and see
> what happens' suggestions then I'm open to testing anything!


What is the CPU doing during these long pauses? Idle or working hard?

Another thing I might do is strace it (or whatever the AIX version of that
is) and see what system calls are going on during or just before and just
after the pauses.

I looked in the Net::SFTP source put couldn't find the lines that trigger
the debug messages you showed (the ones before and after the pauses). So
presumably they are coming from one of the modules used inside it. It
might help if you track those message sources down.

Xho

--
-------------------- http://NewsReader.Com/ --------------------
Usenet Newsgroup Service $9.95/Month 30GB
 
Reply With Quote
 
usenet@DavidFilmer.com
Guest
Posts: n/a
 
      07-07-2006
rahed wrote:
> Try to install Math::BigInt::GMP if it's not already there. That helped
> when I met weird behaviour with long ssh authentication times.


THAT WAS THE PROBLEM!!!!! I owe you a beer!

FWIW, for the benefit of anyone with this same problem who may stumble
upon this thread... installing Math::BigInt::GMP on AIX 5.3 requires a
hack to Makefile.PL - namely that (for some bizarre reason) you must
change

'LIBS' => ['-lgmp'],
to
'LIBS' => ['-L/usr/local/lib -lgmp'],

For more info, see my post in comp.unix.aix (http://tinyurl.com/rtrkz)
where I described various hacks necessary to install several libraries
and Perl modules (including GPG and Pari stuff) on AIX. This particular
hack for Math::GMP (which, apparently, is also needed for
Math::BigInt::GMP) was described by Sisyphus in a thread found here:
http://tinyurl.com/cb299.

--
David Filmer (http://DavidFilmer.com)

 
Reply With Quote
 
apersaud
Guest
Posts: n/a
 
      07-25-2006
Install Math::BigInt::GMP and that should speed things up -- thats how
I fixed my slowness.


(E-Mail Removed) wrote:
> Kindly consider the following script which I am having a problem with:
>
> #!/usr/bin/perl
> use strict; use warnings;
> use Net::SFTP;
>
> my %args = qw{ user myuserid password mypassword debug 1 };
> $args{'ssh_args'} = [protocol => 2];
>
> my $sftp = Net::SFTP->new('123.456.78.90', %args) or die $!;
>
> __END__
>
> The script works fine, but on one particular machine it takes about a
> minute to get the SFTP handle. The identical script runs instantly on
> another (inferior) machine.
>
> So, naturally, I would suspect a problem with that particular machine
> that would have nothing to do with my script (maybe a routing issue,
> etc), EXCEPT two things make me think the problem somehow involves the
> Net::SFTP module itself:
>
> 1. "sftp" from a shell (same parameters) connects instantly.
> 2. I can get a Net::SSH:erl handle (same %args) instantly.
>
> With debug mode turned on, I observe that there are long pauses at
> three points. I've posted the debug output (with pauses indicated)
> below. I see no obvious reason for any performance problems (I'm
> running on a new and lightly loaded p520 (POWER5) AIX box). As I said,
> I can get a Net::SSH:erl handle instantly (and Net::SFTP uses N:S).
>
> If anyone has any suggestions on what the problem might be with using
> Net::SFTP from this particular machine, please let me know. Here's my
> Net::SFTP debug output (with pauses indicated):
>
> server: Reading configuration data /myuserid/.ssh/config
> server: Reading configuration data /etc/ssh_config
> server: Connecting to server123.mycompany.com, port 22.
> server: Remote protocol ver 1.99, remote ver OpenSSH_3.6.1p2
> server: Net::SSH:erl Version 1.29, protocol version 2.0.
> server: No compat match: OpenSSH_3.6.1p2.
> server: Connection established.
> server: Sent key-exchange init (KEXINIT), wait response.
> server: Algorithms, c->s: 3des-cbc hmac-sha1 none
> server: Algorithms, s->c: 3des-cbc hmac-sha1 none
>
> >>long pause............................................. ...........

>
> server: Entering Diffie-Hellman Group 1 key exchange.
> server: Sent DH public key, waiting for reply.
> server: Received host key, type 'ssh-dss'.
> server: Host 'server123.mycompany.com' is known and matches host key
> server: Computing shared secret key.
>
> >>long pause............................................. ...........

>
> server: Verifying server signature.
>
> >>long pause............................................. ...........

>
> server: Waiting for NEWKEYS message.
> server: Enabling incoming encryption/MAC/compression.
> server: Send NEWKEYS, enable outgoing encryption/MAC/compression.
> server: Sending request for user-authentication service.
> server: Service accepted: ssh-userauth.
> server: Trying empty user-authentication request.
> server: Authentication methods that can continue:
> publickey,password,keyboard-interactive.
> server: Next method to try is publickey.
> server: Next method to try is password.
> server: Trying password authentication.
> server: Login completed, opening dummy shell channel.
> server: channel 0: new [client-session]
> server: Requesting channel_open for channel 0.
> server: channel 0: open confirm rwindow 0 rmax 32768
> Establishing SFTP connection with server server123.mycompany.com:
> channel 1: new [client-session]
> server: Requesting channel_open for channel 1.
> server: Sending subsystem: sftp
> server: Requesting service subsystem on channel 1.
> server: channel 1: open confirm rwindow 0 rmax 32768
> server: sftp: Sending SSH2_FXP_INIT
> server: sftp: Remote version: 3
> SFTP connected
>
> --
> David Filmer (http://DavidFilmer.com)


 
Reply With Quote
 
usenet@DavidFilmer.com
Guest
Posts: n/a
 
      07-25-2006
apersaud wrote:
> Install Math::BigInt::GMP and that should speed things up -- thats how
> I fixed my slowness.


Thanks, apersaud. Another poster, Radek (rahed), kindly posed this
solution in a previous reply to this thread. And you are both correct
- installing the Math::BigInt::GMP module did solve the Net::SFTP
problems I was having.

I do wonder, though... this module is not mentioned in the Net::SSH or
Net::SFTP perldocs, and a -MCPAN install does not try to install it as
a prereq. Math::BigInt::GMP is nowhere near being a builtin module.
Absence of the module does not produce "use" or "require" errors or
warnings when using Net::SFTP. Dependency on this module is not
anything I would consider intuitive or easily discerned (and, oddly
enough, it's not REALLY a dependency, although the horrible performance
without the module would qualify it as 'dependent' AFAIAC).

So..... Why doesn't EVERYBODY have this problem with Net::SFTP?

Hmmmm.

--
David Filmer (http://DavidFilmer.com)

 
Reply With Quote
 
Sisyphus
Guest
Posts: n/a
 
      07-25-2006

<(E-Mail Removed)> wrote in message
..
..
> Absence of the module does not produce "use" or "require" errors or
> warnings when using Net::SFTP.


Somewhere in the Net::SFTP source there is presumably something like:

use Math::BigInt lib => 'GMP';

That line of code ensures that, if Math::BigInt::GMP is installed, then the
(very fast) Math::BigInt::GMP module handles the computations. If, however,
Math::BigInt::GMP is not installed, then the (very slow) Math::BigInt module
handles the computations. (I have a vague notion that a message is written
to stderr if Math::BigInt::GMP is not found, telling you that the program is
reverting to pure perl routines - but I'm not entirely sure about that.)

That accounts for the difference in the performance that you got - pure perl
Math::BigInt versus the GMP C library.

Cheers,
Rob


 
Reply With Quote
 
usenet@DavidFilmer.com
Guest
Posts: n/a
 
      07-26-2006
Sisyphus wrote:
> Somewhere in the Net::SFTP source there is presumably something like:
> use Math::BigInt lib => 'GMP';


There is no such line anywhere within my ../site_perl/5.8.6/Net
directory structure, including Net::SFTP and Net::SSH:erl. Which
further compounds my confusion........

> (I have a vague notion that a message is written
> to stderr if Math::BigInt::GMP is not found, telling you that the program is
> reverting to pure perl routines - but I'm not entirely sure about that.)


That would have been nice in helping me solve my problem, but no such
message came on STDERR before I installed the GMP module.

--
David Filmer (http://DavidFilmer.com)

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Hi, I am getting the following errorTypeError: sort() takes nokeyword arguments gaurav kashyap Python 2 10-17-2008 07:46 AM
Package to handle table text render (handle space or tab betweenthe columns) ? =?ISO-8859-1?Q?KLEIN_St=E9phane?= Python 3 10-06-2006 08:46 AM
Possible to handle web requests without an ASPX page? i.e. have DLL handle request. jdlwright@shaw.ca ASP .Net 2 05-31-2005 05:42 PM
how to handle command line output(not terminal handle) Leon Python 2 11-04-2004 05:16 AM
File Handle Reading Blues: Rereading a File Handle for Input Dietrich Perl 1 07-22-2004 10:02 AM



Advertisments