LWP with proxy problem

12-20-2003

When running the sample code below without a proxy the GET returns the
expected data. When run with the $proxy uncommented the GET returns
the content of the login page for the site being accessed. The site
in the code is valid for ease of testing. I also am including the LWP
debug info for each attempt.

Can anyone explain this behavior and what can I do to correct it?

Thanks in advance

Trace without proxy:

LWP::UserAgent::new: ()
LWP::UserAgent::request: ()
LWP::UserAgent::send_request: GET
https://squid.servebeer.com/getservi...ing&format=csv
LWP::UserAgent::_need_proxy: Not proxied
LWP:

rotocol::http::request: ()
LWP:

rotocol::collect: read 28 bytes
LWP::UserAgent::request: Simple response: OK

Home
PE

Trace with proxy:

LWP::UserAgent::new: ()
LWP::UserAgent:

roxy: https http://148.245.207.85:8080
LWP::UserAgent::request: ()
LWP::UserAgent::send_request: GET
https://squid.servebeer.com/getservi...ing&format=csv
LWP::UserAgent::_need_proxy: Proxied to http://148.245.207.85:8080
LWP:

rotocol::http::request: ()
LWP:

rotocol::collect: read 236 bytes
LWP:

rotocol::collect: read 594 bytes
LWP:

rotocol::collect: read 416 bytes
LWP:

rotocol::collect: read 450 bytes
LWP:

rotocol::collect: read 1017 bytes
LWP:

rotocol::collect: read 443 bytes
LWP:

rotocol::collect: read 643 bytes
LWP::UserAgent::request: Simple response: OK

<html lang="en">


<head>
<title>

Error

</title>
<script language="JavaScript">
....

Here is the sample code:

#!/usr/bin/perl -w

use LWP::UserAgent;
use HTTP::Request;
use HTTP::Response;
use Crypt::SSLeay;

LWP:

ebug::level('+');

$url = "https://squid.servebeer.com/getservices.do?user=Guest&password=JustLooking&for mat=csv";
#$proxy="http://xxx.xxx.xxx.xxx:8080";

$ua = LWP::UserAgent->new();

if (defined $proxy)
{
$ENV{HTTPS_PROXY} = $proxy;

# initialize from environment variables
$ua->env_proxy;
}

$req = HTTP::Request->new(GET => $url);
$response = $ua->request($req);
if ($response->is_error())
{
printf " %s\n", $response->status_line;
}

else
{
$content = $response->content();
print $content;
}

exit;

12-22-2003

Note: you'll reach more people at comp.lang.perl.misc. This newsgroup
does not officially exist, and so is not universally propogated.

LWP::UserAgent sends the wrong request for HTTPS connections by proxy.
In the docs for Crypt::SSLeay, you'll find this:
http://search.cpan.org/~chamas/Crypt...#PROXY_SUPPORT
LWP::UserAgent has its own methods of proxying which may work for you
and is likely incompatible with Crypt::SSLeay proxy support. To use
LWP::UserAgent proxy support, try something like:

my $ua = new LWP::UserAgent;
$ua->proxy([qw( https http )], "$proxy_ip:$proxy_port");

At the time of this writing, libwww v5.6 seems to proxy https requests
fine with an Apache mod_proxy server. It sends a line like:

GET https://www.nodeworks.com HTTP/1.1

to the proxy server, which is not the CONNECT request that some
proxies would expect, so this may not work with other proxy servers
than mod_proxy. The CONNECT method is used by Crypt::SSLeay's internal
proxy support.
Crypt::SSLeay Proxy Support

For native Crypt::SSLeay proxy support of https requests, you need to
set an environment variable HTTPS_PROXY to your proxy server & port,
as in:

# PROXY SUPPORT
$ENV{HTTPS_PROXY} = 'http://proxy_hostname_or_ip

ort';
$ENV{HTTPS_PROXY} = '127.0.0.1:8080';

Use of the HTTPS_PROXY environment variable in this way is similar to
LWP::UserAgent->env_proxy() usage, but calling that method will likely
override or break the Crypt::SSLeay support, so do not mix the two.
</excerpt>

This is the code I use, *after* the UserAgent has been created:

# LWP proxying does not work for HTTPS
# This must cannot be set at the time $a is created
$ENV{HTTPS_PROXY} = $proxy;

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Proxy and LWP::UserAgent	Mike	Perl Misc	1	06-30-2005 06:03 PM
sanity checking proxy URL before passing to LWP	buildmorelines	Perl Misc	2	11-06-2004 05:02 PM
Perl LWP content request behind firewall and proxy server	demonhunter	Perl	3	08-16-2004 05:01 PM
LWP with proxy problem	Woogie	Perl Misc	4	12-27-2003 03:21 PM
How to use proxy in Net::HTTP, not in LWP::UserAgent?	Great Deals	Perl Misc	1	10-02-2003 07:27 PM