Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Using a proxy with urllib2

Reply
Thread Tools

Using a proxy with urllib2

 
 
Jack
Guest
Posts: n/a
 
      01-10-2008
I'm trying to use a proxy server with urllib2.
So I have managed to get it to work by setting the environment
variable:
export HTTP_PROXY=127.0.0.1:8081

But I wanted to set it from the code. However, this does not set the proxy:
httpproxy = '127.0.0.1:3129'
proxy_support = urllib2.ProxyHandler({"http":"http://" + httpproxy})
opener = urllib2.build_opener(proxy_support, urllib2.HTTPHandler)
urllib2.install_opener(opener)
I'm using it from a web.py URL handler file, not sure if it matters.

I have another question though. It seems that using either of the
methods above, the proxy will be global. What if I want to use
a proxy with one site, but not with another site? Or even use a
proxy for some URLs but not others? The proxy having to be global
is really not convenient. Is there any way to do per-fetch proxy?


 
Reply With Quote
 
 
 
 
Rob Wolfe
Guest
Posts: n/a
 
      01-10-2008
"Jack" <(E-Mail Removed)> writes:

> I'm trying to use a proxy server with urllib2.
> So I have managed to get it to work by setting the environment
> variable:
> export HTTP_PROXY=127.0.0.1:8081
>
> But I wanted to set it from the code. However, this does not set the proxy:
> httpproxy = '127.0.0.1:3129'
> proxy_support = urllib2.ProxyHandler({"http":"http://" + httpproxy})
> opener = urllib2.build_opener(proxy_support, urllib2.HTTPHandler)
> urllib2.install_opener(opener)


Works for me.
How do you know that the proxy is not set?

> I'm using it from a web.py URL handler file, not sure if it matters.


I don't think so.

> I have another question though. It seems that using either of the
> methods above, the proxy will be global. What if I want to use
> a proxy with one site, but not with another site? Or even use a
> proxy for some URLs but not others? The proxy having to be global
> is really not convenient. Is there any way to do per-fetch proxy?


Try this:

<code>
import urllib2

def getopener(proxy=None):
opener = urllib2.build_opener(urllib2.HTTPHandler)
if proxy:
proxy_support = urllib2.ProxyHandler({"http": "http://" + proxy})
opener.add_handler(proxy_support)
return opener

def fetchurl(url, opener):
f = opener.open(url)
data = f.read()
f.close()
return data

print fetchurl('http://www.python.org', getopener('127.0.0.1:8081'))
</code>

HTH,
Rob
 
Reply With Quote
 
 
 
 
Jack
Guest
Posts: n/a
 
      01-11-2008

> Works for me.
> How do you know that the proxy is not set?


The proxy drops some URLs and the URLs were not being dropped when I did
this

> Try this:


Thank you. I'll give it a try.


 
Reply With Quote
 
Jack
Guest
Posts: n/a
 
      01-11-2008
Rob,

I tried your code snippet and it worked great. I'm just wondering if
getopener( ) call
is lightweight so I can just call it in every call to fetchurl( )? Or I
should try to share
the opener object among fetchurl( ) calls?

Thanks,
Jack


"Rob Wolfe" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed)...
> Try this:
>
> <code>
> import urllib2
>
> def getopener(proxy=None):
> opener = urllib2.build_opener(urllib2.HTTPHandler)
> if proxy:
> proxy_support = urllib2.ProxyHandler({"http": "http://" + proxy})
> opener.add_handler(proxy_support)
> return opener
>
> def fetchurl(url, opener):
> f = opener.open(url)
> data = f.read()
> f.close()
> return data
>
> print fetchurl('http://www.python.org', getopener('127.0.0.1:8081'))
> </code>
>
> HTH,
> Rob



 
Reply With Quote
 
Rob Wolfe
Guest
Posts: n/a
 
      01-11-2008
"Jack" <(E-Mail Removed)> writes:

> Rob,
>
> I tried your code snippet and it worked great. I'm just wondering if
> getopener( ) call
> is lightweight so I can just call it in every call to fetchurl( )? Or I
> should try to share
> the opener object among fetchurl( ) calls?


Creating an opener for every url is rather not reasonable way to go.
Sharing is the better approach. In your case you might
create e.g. two instances: simple_opener and proxy_opener.

Regards,
Rob
 
Reply With Quote
 
Jack
Guest
Posts: n/a
 
      01-12-2008

>> I'm trying to use a proxy server with urllib2.
>> So I have managed to get it to work by setting the environment
>> variable:
>> export HTTP_PROXY=127.0.0.1:8081
>>
>> But I wanted to set it from the code. However, this does not set the
>> proxy:
>> httpproxy = '127.0.0.1:3129'
>> proxy_support = urllib2.ProxyHandler({"http":"http://" + httpproxy})
>> opener = urllib2.build_opener(proxy_support, urllib2.HTTPHandler)
>> urllib2.install_opener(opener)


I find out why it doesn't work in my code but I don't have a solution -
somewhere
else in the code calls these two lines:

opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(c j))
urllib2.install_opener(opener)

and they override the proxy opener. Could anyone tell me how to use both
openers?


 
Reply With Quote
 
Rob Wolfe
Guest
Posts: n/a
 
      01-12-2008
"Jack" <(E-Mail Removed)> writes:

>>> I'm trying to use a proxy server with urllib2.
>>> So I have managed to get it to work by setting the environment
>>> variable:
>>> export HTTP_PROXY=127.0.0.1:8081
>>>
>>> But I wanted to set it from the code. However, this does not set the
>>> proxy:
>>> httpproxy = '127.0.0.1:3129'
>>> proxy_support = urllib2.ProxyHandler({"http":"http://" + httpproxy})
>>> opener = urllib2.build_opener(proxy_support, urllib2.HTTPHandler)
>>> urllib2.install_opener(opener)

>
> I find out why it doesn't work in my code but I don't have a solution -
> somewhere
> else in the code calls these two lines:
>
> opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(c j))
> urllib2.install_opener(opener)
>
> and they override the proxy opener. Could anyone tell me how to use both
> openers?
>


You don't have to create another opener if you only want to add
some handler. You can use `add_handler` method, e.g.:
opener.add_handler(urllib2.HTTPCookieProcessor(cj) )

HTH,
Rob

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
How to configure proxy authentication when using urllib2? Licheng Fang Python 2 02-19-2006 01:11 PM
Problem with: urllib2.build_opener(urllib2.HTTPCookieProcessor(cj)) Josef Cihal Python 0 09-05-2005 11:26 AM
% in POST when using URLLIB2.URLOPEN with PROXY Herman Geldenhuys Python 1 07-14-2004 06:42 PM
Proxy Authentication using urllib2 Andre Bocchini Python 2 09-23-2003 01:18 AM
FTP with urllib2 behind a proxy O. Koch Python 4 08-14-2003 11:54 AM



Advertisments