Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Perl > Perl Misc > help with LWP and log in after redirect

Reply
Thread Tools

help with LWP and log in after redirect

 
 
eatmoreoats
Guest
Posts: n/a
 
      03-04-2008
I'm trying to automate the log in to a website, which I've done before
successfully with lwp. However in this case I'm having problems and
need some help. After posting the username/password, a redirect
occurs to a new url where the username and password have been encoded.
With post redirects enabled, I end up back at the log in page, rather
than following the redirect.

Here are the request and response headers, as observed with Firefox
plugins (HTTP LIve Headers, and Tamper Data). These are observed
during a successful login with my firefox browser.

REQUEST HEADER

Host=www.mywebsite.com
User-Agent=Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:
1.8.1.12) Gecko/20080201 Firefox/2.0.0.12
Accept=text/xml,application/xml,application/xhtml+xml,text/
html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language=en-us,en;q=0.5
Accept-Encoding=gzip,deflate
Accept-Charset=ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive=300
Connection=keep-alive
Referer=http://www.mywebsite.com/newmywebsite/logon.aspx
Cookie=MyCookie=; mysrv=abc;
JSESSIONID=DC638726E6831E7D6EEE996823FEECA0; JOSSO_SESSIONID=-
Content-Type=application/x-www-form-urlencoded
Content-Length=171
POSTDATA=username=theusername&password=thepassword


RESPONSE HEADER

Status=Moved Temporarily - 302
Server=Apache-Coyote/1.1
X-Powered-By=Servlet 2.4; JBoss-4.0.2 (build: CVSTag=JBoss_4_0_2
date=200505022023)/Tomcat-5.5
X-Node-Name=abc
Set-Cookie=SERVER-X=abc; Domain=.website.com; Expires=Tue, 04-Mar-2008
15:03:03 GMT; Path=/
NewMywebsite=null; Expires=Thu, 01-Jan-1970 00:00:10 GMT
Location=http://www.mywebsite.com/newmywebsite/somenewurl/
somenewscript.wotever?
cmd=login&username=YWR2YW5jZWludGVybmV0&password=c 2FuaXR5
Content-Type=text/html
Content-Length=0
Date=Tue, 04 Mar 2008 14:03:03 GMT

Notice the Location in the response - if I take that url and do a get
request on it, I'm logged in. However, I can't seem to get the code to
follow the redirect down this url path. If I disable redirects, and
grab the Location header value, it is Location=http://
http://www.mywebsite.com/newmywebsit...&username=null.
I'd like the script to just follow the redirect to the new location
but its not clear why thats not working. Any ideas how to make this
work ?

Here is the code example :

use HTTP::Cookies;
use LWP::UserAgent;

$ua = LWP::UserAgent->new;
$cookies = new HTTP::Cookies();
$ua->cookie_jar($cookies);
$ua->timeout(300);
$ua->requests_redirectable (['GET', 'HEAD', 'POST']);

# step 1 - hit the login page first
$response = $ua->get('http://www.mywebsite.com/newmywebsite/
logon.aspx');

# step 2 - post in the username/password
$response = $ua->post('http://www.mywebsite.com/newmywebsite/
logonvalidate.aspx',
[
username=>"theusername",
password=>"thepassword",
]
);
if ($response->is_success) { print $response->content; } else { print
$response->status_line; }

# the content at this point is the same as the logon.aspx page in step
1 - why is the redirect not working here ?

Thanks
-emo
 
Reply With Quote
 
 
 
 
Ben Morrow
Guest
Posts: n/a
 
      03-04-2008

Quoth eatmoreoats <(E-Mail Removed)>:
> I'm trying to automate the log in to a website, which I've done before
> successfully with lwp. However in this case I'm having problems and
> need some help. After posting the username/password, a redirect
> occurs to a new url where the username and password have been encoded.
> With post redirects enabled, I end up back at the log in page, rather
> than following the redirect.
>
> Here are the request and response headers, as observed with Firefox
> plugins (HTTP LIve Headers, and Tamper Data). These are observed
> during a successful login with my firefox browser.
>
> REQUEST HEADER
>

<snip>
> Referer=http://www.mywebsite.com/newmywebsite/logon.aspx

<snip>
>
> RESPONSE HEADER
>
> Status=Moved Temporarily - 302

<snip>
> Location=http://www.mywebsite.com/newmywebsite/somenewurl/
> somenewscript.wotever?
> cmd=login&username=YWR2YW5jZWludGVybmV0&password=c 2FuaXR5

<snip>
>
> Notice the Location in the response - if I take that url and do a get
> request on it, I'm logged in. However, I can't seem to get the code to
> follow the redirect down this url path. If I disable redirects, and
> grab the Location header value, it is Location=http://
> http://www.mywebsite.com/newmywebsit...cript.wotever?
> cmd=login&username=null.


So whatever LWP is sending, the server doesn't like it and doesn't
accept it as a valid logon.

> I'd like the script to just follow the redirect to the new location
> but its not clear why thats not working. Any ideas how to make this
> work ?
>
> Here is the code example :
>
> use HTTP::Cookies;
> use LWP::UserAgent;
>
> $ua = LWP::UserAgent->new;
> $cookies = new HTTP::Cookies();
> $ua->cookie_jar($cookies);
> $ua->timeout(300);
> $ua->requests_redirectable (['GET', 'HEAD', 'POST']);


You shouldn't need this. requests_redirectable refers to the method used
to follow the redirect, not the method the redirect is a reply to; and
LWP treats 302 as 'GET this URL' rather than 'repeat the previous
request with a different URL', in line with the behaviour of most (all?)
actual browsers (this is strictly against the letter of the standard,
but RFC2616 recognises that this is the actual implemented behaviour).

> # step 1 - hit the login page first
> $response = $ua->get('http://www.mywebsite.com/newmywebsite/
> logon.aspx');
>
> # step 2 - post in the username/password
> $response = $ua->post('http://www.mywebsite.com/newmywebsite/
> logonvalidate.aspx',
> [
> username=>"theusername",
> password=>"thepassword",
> ]
> );


This will not set the Referer header. I would recommend using
WWW::Mechanize instead, which tries as hard as possible to emulate an
actual browser.

Ben

 
Reply With Quote
 
 
 
 
eatmoreoats
Guest
Posts: n/a
 
      03-04-2008
On Mar 4, 11:05 am, Ben Morrow <(E-Mail Removed)> wrote:
> Quotheatmoreoats<(E-Mail Removed)>:
>
>
>
> > I'm trying to automate the log in to a website, which I've done before
> > successfully with lwp. However in this case I'm having problems and
> > need some help. After posting the username/password, a redirect
> > occurs to a new url where the username and password have been encoded.
> > With post redirects enabled, I end up back at the log in page, rather
> > than following the redirect.

>
> > Here are the request and response headers, as observed with Firefox
> > plugins (HTTP LIve Headers, and Tamper Data). These are observed
> > during a successful login with my firefox browser.

>
> > REQUEST HEADER

>
> <snip>
> > Referer=http://www.mywebsite.com/newmywebsite/logon.aspx

> <snip>
>
> > RESPONSE HEADER

>
> > Status=Moved Temporarily - 302

> <snip>
> > Location=http://www.mywebsite.com/newmywebsite/somenewurl/
> > somenewscript.wotever?
> > cmd=login&username=YWR2YW5jZWludGVybmV0&password=c 2FuaXR5

> <snip>
>
> > Notice the Location in the response - if I take that url and do a get
> > request on it, I'm logged in. However, I can't seem to get the code to
> > follow the redirect down this url path. If I disable redirects, and
> > grab the Location header value, it is Location=http://
> >http://www.mywebsite.com/newmywebsit...cript.wotever?
> > cmd=login&username=null.

>
> So whatever LWP is sending, the server doesn't like it and doesn't
> accept it as a valid logon.
>
> > I'd like the script to just follow the redirect to the new location
> > but its not clear why thats not working. Any ideas how to make this
> > work ?

>
> > Here is the code example :

>
> > use HTTP::Cookies;
> > use LWP::UserAgent;

>
> > $ua = LWP::UserAgent->new;
> > $cookies = new HTTP::Cookies();
> > $ua->cookie_jar($cookies);
> > $ua->timeout(300);
> > $ua->requests_redirectable (['GET', 'HEAD', 'POST']);

>
> You shouldn't need this. requests_redirectable refers to the method used
> to follow the redirect, not the method the redirect is a reply to; and
> LWP treats 302 as 'GET this URL' rather than 'repeat the previous
> request with a different URL', in line with the behaviour of most (all?)
> actual browsers (this is strictly against the letter of the standard,
> but RFC2616 recognises that this is the actual implemented behaviour).
>
> > # step 1 - hit the login page first
> > $response = $ua->get('http://www.mywebsite.com/newmywebsite/
> > logon.aspx');

>
> > # step 2 - post in the username/password
> > $response = $ua->post('http://www.mywebsite.com/newmywebsite/
> > logonvalidate.aspx',
> > [
> > username=>"theusername",
> > password=>"thepassword",
> > ]
> > );

>
> This will not set the Referer header. I would recommend using
> WWW::Mechanize instead, which tries as hard as possible to emulate an
> actual browser.
>
> Ben


Hi Ben,
Thanks for the suggestion and I will give it a try. However, I did
actually try setting all of the other header key-value pairs in the
post request but without success.
Dom
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Help Needed - LWP Redirect problemss mark1.thompson45@btinternet.com Perl Misc 8 10-24-2005 02:37 PM
LWP gives 302 Found after update? John Bokma Perl Misc 5 08-16-2005 07:35 PM
Basic Q - Response.Redirect, all redirect to first Response.Redirect statement Sal ASP .Net Web Controls 1 05-15-2004 03:46 PM
Need help on the Permissions needed to log to Event Log from ASP.NET? Henrik_the_boss ASP .Net 0 11-05-2003 10:14 AM
LWP::UserAgent to POST and redirect browser at the same time La Jesus Perl 9 10-27-2003 09:55 PM



Advertisments