Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > urlgrabber cookie handling?

Reply
Thread Tools

urlgrabber cookie handling?

 
 
Devraj
Guest
Posts: n/a
 
      10-17-2007
Hi everyone,

I have been battling to make my code work with a HTTPS proxy, current
my code uses urllib2 to to most things and works well, except that
urllib2 doesn't handle HTTPS proxies.

Urlgrabber (http://linux.duke.edu/projects/urlgrabber/help/
urlgrabber.grabber.html) looks very promising except that I can find a
way to handle cookies in urlgrabber. Is there a way urlgrabber can use
a HTTPCookieProcess or cookielib.CookieJar object to handle cookies?

Thanks for sparing your time. Thanks.

--
Code referenced in this email is available at http://code.google.com/p/gdatacopier

 
Reply With Quote
 
 
 
 
John J. Lee
Guest
Posts: n/a
 
      10-17-2007
Devraj <(E-Mail Removed)> writes:

> Hi everyone,
>
> I have been battling to make my code work with a HTTPS proxy, current
> my code uses urllib2 to to most things and works well, except that
> urllib2 doesn't handle HTTPS proxies.
>
> Urlgrabber (http://linux.duke.edu/projects/urlgrabber/help/
> urlgrabber.grabber.html) looks very promising except that I can find a
> way to handle cookies in urlgrabber. Is there a way urlgrabber can use
> a HTTPCookieProcess or cookielib.CookieJar object to handle cookies?


I don't see a nice way. But then I don't see any HTTPS proxy support
in urlgrabber... (I looked at version 3.1.0).

There is a recipe or two on ASPN showing how to support HTTPS proxies
with urllib2, which gives an idea how to do it, though the code is a
bit rough (I'd post some code myself, but I did it for work).


John
 
Reply With Quote
 
 
 
 
Devraj
Guest
Posts: n/a
 
      10-18-2007
Hi John,

Thanks for getting back to me. I did find the ASPN article. If I
figure this out then I will make sure I post the code somewhere for
public consumption.

On Oct 18, 6:13 am, (E-Mail Removed) (John J. Lee) wrote:
> Devraj <(E-Mail Removed)> writes:
> > Hi everyone,

>
> > I have been battling to make my code work with a HTTPS proxy, current
> > my code uses urllib2 to to most things and works well, except that
> > urllib2 doesn't handle HTTPS proxies.

>
> > Urlgrabber (http://linux.duke.edu/projects/urlgrabber/help/
> > urlgrabber.grabber.html) looks very promising except that I can find a
> > way to handle cookies in urlgrabber. Is there a way urlgrabber can use
> > a HTTPCookieProcess or cookielib.CookieJar object to handle cookies?

>
> I don't see a nice way. But then I don't see any HTTPS proxy support
> in urlgrabber... (I looked at version 3.1.0).
>
> There is a recipe or two on ASPN showing how to support HTTPS proxies
> with urllib2, which gives an idea how to do it, though the code is a
> bit rough (I'd post some code myself, but I did it for work).
>
> John



 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
urlgrabber for Python 3.0 Robert Dailey Python 1 04-30-2009 09:26 PM
Cookie Question (IP as domain and cookie file location) =?Utf-8?B?UGF1bA==?= ASP .Net 1 01-10-2006 08:37 PM
Any downsides to cookie assignment inside custom class using HttpContext.Current? ASP.NET 2.0 cookie fix? ASP .Net 2 08-17-2005 06:43 AM
Cookie and Session Cookie Questions. Shapper ASP .Net 1 04-27-2005 11:20 AM
Session cookie? Browser instance cookie? Ben ASP .Net 3 06-03-2004 03:41 AM



Advertisments