Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Perl > Perl Misc > LWP::UserAgent and redirected page responses

Reply
Thread Tools

LWP::UserAgent and redirected page responses

 
 
Bill
Guest
Posts: n/a
 
      10-21-2005
Hello. This concerns LWP::UserAgent. If a request is sent to a certain
web site, the response in the browser comes back as a completely
different domain and site due to redirection. How do I find out, from
the UserAgent module, what the redirected url is? A uri() method as in
WWW::Mechanize seems like a good candidate but when checked the uri()
method seems to return the original request uri, not the redirected
one. I need to know exactly what would be in the url box of a web
browser after the redirected response happens. Anyone know how to do
this? I will post code if requested.

 
Reply With Quote
 
 
 
 
J. Gleixner
Guest
Posts: n/a
 
      10-21-2005
Bill wrote:
> Hello. This concerns LWP::UserAgent. If a request is sent to a certain
> web site, the response in the browser comes back as a completely
> different domain and site due to redirection. How do I find out, from
> the UserAgent module, what the redirected url is? A uri() method as in
> WWW::Mechanize seems like a good candidate but when checked the uri()
> method seems to return the original request uri, not the redirected
> one. I need to know exactly what would be in the url box of a web
> browser after the redirected response happens. Anyone know how to do
> this? I will post code if requested.
>


Check the requests_redirectable method in "perldoc LWP::UserAgent". By
default, it doesn't redirect on a POST.
 
Reply With Quote
 
 
 
 
Paul Lalli
Guest
Posts: n/a
 
      10-21-2005
Bill wrote:
> Hello. This concerns LWP::UserAgent. If a request is sent to a certain
> web site, the response in the browser comes back as a completely
> different domain and site due to redirection. How do I find out, from
> the UserAgent module, what the redirected url is? A uri() method as in
> WWW::Mechanize seems like a good candidate but when checked the uri()
> method seems to return the original request uri, not the redirected
> one. I need to know exactly what would be in the url box of a web
> browser after the redirected response happens. Anyone know how to do
> this? I will post code if requested.


[Disclaimer: All of the below is gleaned from reading the relevant
docs. I have not tried any LWP code myself ]

The LWP::UserAgent object sends a request to a server by means of the
post() method. The return value of the post() method is an object of
HTTP::Response. The HTTP::Response man page shows that one of its
methods is request(), which is defined as follows:
$r->request
$r->request( $request )

This is used to get/set the request attribute. The request attribute
is a
reference to the the request that caused this response. It does not
have
to be the same request passed to the $ua->request() method, because
there
might have been redirects and authorization retries in between.

To find out what we can get from that object, we look to HTTP::Request,
which has this method:
$r->uri
$r->uri( $val )

This is used to get/set the uri attribute. The $val can be a
reference to
a URI object or a plain string. If a string is given, then it should
be
parseable as an absolute URI.

Putting it altogether then:

my $ua = new HTTP::UserAgent;
my $response = $ua->post($url);
my $request = $response->request();
my $found_url = $request->uri();

Hope this helps,
Paul Lalli

 
Reply With Quote
 
William Herrera
Guest
Posts: n/a
 
      10-22-2005
Paul Lalli wrote:

> To find out what we can get from that object, we look to HTTP::Request,
> which has this method:
> $r->uri
> $r->uri( $val )
>
> This is used to get/set the uri attribute. The $val can be a
> reference to
> a URI object or a plain string. If a string is given, then it should
> be
> parseable as an absolute URI.
>
> Putting it altogether then:
>
> my $ua = new HTTP::UserAgent;
> my $response = $ua->post($url);
> my $request = $response->request();
> my $found_url = $request->uri();
>
> Hope this helps,
> Paul Lalli
>


Sorry that I did not post code the last time.
Here's an excerpt from the method in question:

-----------------------------------------
# log in to my.tmobile.com (T-Mobile USA) and
# return hashref keyed to total charged minutes (not free) and
# total charged SMS messaging. Keys are 'calls' and 'messages'
sub get_billing {
my ($self) = @_;
$self->{start_page} = $base_uri;
$self->{agent} =
new WWW::Mechanize(
agent => "Mozilla/4.0 (compatible; MSIE 7.0b; Perl $])",
);
$self->{agent}->get($base_uri);
$self->{agent}->form_name("Form1") or croak $self->content;

# Even though WWW:Mechanize does most of the work, we have to
# manually change readonly on hidden fields. Annoying.
my $input = $self->{agent}->current_form->find_input('__EVENTTARGET')
or $self->_err("Cannot find hidden field for signin in Form1");
no warnings;
$input->readonly(0);
use warnings;
$self->{agent}->set_fields(
'txtMSISDN' => $self->{user_number},
'txtPassword' => $self->{password},
'__EVENTTARGET' => 'signin',
);
$self->{agent}->submit
or $self->_err("Could not submit form1 successfully");
$self->{agent}->get("https://my.t-mobile.com/Billing/")
or $self->_err("Cannot get Billing page: ");
print "Line uri: ", $self->{agent}->uri, "\n";

# cut for brevity here....
}

The problem is that the second uri printed is NOT the same as the uri
displayed in the url line of the browser doing the same tasks, even
though the CONTENT of the FIRST request's response text is correct. As a
result, the user agent fails to correctly submit the next click, since
the base URL is now incorrect. I cannot just plug in a fixed url there,
since the redirected URL contains some cookie-like values needed by the
host.

Ideas?
 
Reply With Quote
 
William Herrera
Guest
Posts: n/a
 
      10-22-2005
J. Gleixner wrote:
> Check the requests_redirectable method in "perldoc LWP::UserAgent". By
> default, it doesn't redirect on a POST.


Thanks, I'll try that.
 
Reply With Quote
 
John Bokma
Guest
Posts: n/a
 
      10-22-2005
William Herrera
<(E-Mail Removed)> wrote:

> The problem is that the second uri printed is NOT the same as the uri
> displayed in the url line of the browser doing the same tasks, even
> though the CONTENT of the FIRST request's response text is correct. As
> a result, the user agent fails to correctly submit the next click,
> since the base URL is now incorrect. I cannot just plug in a fixed url
> there, since the redirected URL contains some cookie-like values
> needed by the host.
>
> Ideas?


Might be the UserAgent or any other header that triggers this behaviour. If
the uri you get back contains the "cookie-like" values, you can tweak them
into the URL you know.

--
John Small Perl scripts: http://johnbokma.com/perl/
Perl programmer available: http://castleamber.com/
I ploink googlegroups.com

 
Reply With Quote
 
William Herrera
Guest
Posts: n/a
 
      10-22-2005
John Bokma wrote:
>>The problem is that the second uri printed is NOT the same as the uri
>>displayed in the url line of the browser doing the same tasks, even
>>though the CONTENT of the FIRST request's response text is correct. As
>>a result, the user agent fails to correctly submit the next click,
>>since the base URL is now incorrect. I cannot just plug in a fixed url
>>there, since the redirected URL contains some cookie-like values
>>needed by the host.
>>
>>Ideas?

>
>
> Might be the UserAgent or any other header that triggers this behaviour. If
> the uri you get back contains the "cookie-like" values, you can tweak them
> into the URL you know.


Yes, and the "cookie-like" values seem to be a per-session ID that
changes. So, I need to know that the uri is that I get back. Which LWP
does not seem to keep anywhere--it keeps the original, non-redirected
uri instead?
 
Reply With Quote
 
William Herrera
Guest
Posts: n/a
 
      10-22-2005
John Bokma wrote:

> If there is a redirect, LWP stores this info. IIRC in debug mode you can
> see what is happening. Another trick is to set the redirect level to 0, to
> 1, etc.
>


Thanks. Using LWP:ebugFile shows that LWP correctly GETS the URL which
the browser displays, yet the uri() method returns the initial URL, not
the finally redirected one. Weird. I suppose I could check the tail of
the LWP:ebugFile as the program progresses, but that seems so clumsy.
There ought to be a method or value inside UserAgent that I can use?
 
Reply With Quote
 
William Herrera
Guest
Posts: n/a
 
      10-23-2005

>
> If there is a redirect, LWP stores this info. IIRC in debug mode you can
> see what is happening. Another trick is to set the redirect level to 0, to
> 1, etc.
>
> I am sure there are little (Perl) proxy programs available that show you
> exactly what is being send out, and comes back.
>
> Also, try with a browser with JavaScript off, since that is what LWP is
> doing.
>


I wrote an itty bitty module to fix the problem (currently calling it
LWP::LastURI). So now things work okay. Thanks for the suggestion to
look at the LWP debug output.

--Bill
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Passing variable between asp login page and the redirected page Steve ASP .Net 2 11-01-2007 11:32 AM
problem with index.html .(page is automatically gettin redirected to index.html) karthikeyavenkat Java 2 03-17-2005 10:01 PM
Grabbing referrer page on a redirected error page darrel ASP .Net 4 09-29-2004 08:25 PM
:o( .... Client not redirected to login page. M O J O ASP .Net 5 09-27-2004 09:24 AM
Using Form based Authenication and still getting redirected back to the login page Irishmaninusa ASP .Net Security 0 09-23-2004 03:02 PM



Advertisments