Velocity Reviews

Velocity Reviews (http://www.velocityreviews.com/forums/index.php)
-   Python (http://www.velocityreviews.com/forums/f43-python.html)
-   -   FTP with urllib2 behind a proxy (http://www.velocityreviews.com/forums/t320706-ftp-with-urllib2-behind-a-proxy.html)

O. Koch 08-07-2003 11:04 AM

FTP with urllib2 behind a proxy
 
Until now, i know that ftplib doesn't support proxies and that i have
to use urllib2. But i don't know how to use the urllib2 correct. I
found some examples, but i don't understand them.

Is there anyone who can help me?

Anand Pillai 08-13-2003 10:37 AM

Re: FTP with urllib2 behind a proxy
 
You need to install a proxyhandler, authhandler,
ftphandler and httphandler. Then build yourself
an opener, that opens the doors for you ... :-)

The following does the trick.

proxy_handler = urllib2.ProxyHandler( {'http': 'myhttpproxy:80',
'https' : 'myhttpsproxy:443',
'ftp' : 'myftpproxy:21' } )

opener= urllib2.build_opener(proxy_handler, urllib2.HTTPBasicAuthHandler(),
urllib2.HTTPHandler, urllib2.HTTPSHandler,
urllib2.FTPHandler)

# install this opener
urllib2.install_opener(opener)

# Go ahead, knock knock!

req=urlli2.Request('ftp://ftp.gnu.org')
data=urllib2.urlopen(req).read()

Of course, replace the arbit proxy values I wrote
with your proxy values. If your proxy need authentication
, you will need to do a bit more here.

proxyauth='http://' + username + '@' + password + 'myproxy:myproxport'

Then your proxy handler becomes ( I am assuming a generic proxy
for all protocols here!)

proxy_handler = urllib2.ProxyHandler ( {'http' : proxyauth,
'https' : proxyauth,
'ftp' : proxyauth } )

HTH.

-Anand


jjl@pobox.com (John J. Lee) wrote in message news:<87d6feykl8.fsf@pobox.com>...
> Zappelphillip@gmx.de (O. Koch) writes:
>
> > Until now, i know that ftplib doesn't support proxies and that i have
> > to use urllib2. But i don't know how to use the urllib2 correct. I
> > found some examples, but i don't understand them.
> >
> > Is there anyone who can help me?

>
> import urllib2
> response = urllib2.urlopen("ftp://ftp.example.com/pub/myfile")
> data = response.read()
> response.close()
>
>
> Does that do the trick?
>
>
> John


John J. Lee 08-13-2003 12:50 PM

Re: FTP with urllib2 behind a proxy
 
pythonguy@Hotpop.com (Anand Pillai) writes:

> You need to install a proxyhandler, authhandler,
> ftphandler and httphandler. Then build yourself
> an opener, that opens the doors for you ... :-)
>
> The following does the trick.
>
> proxy_handler = urllib2.ProxyHandler( {'http': 'myhttpproxy:80',
> 'https' : 'myhttpsproxy:443',
> 'ftp' : 'myftpproxy:21' } )
>
> opener= urllib2.build_opener(proxy_handler, urllib2.HTTPBasicAuthHandler(),
> urllib2.HTTPHandler, urllib2.HTTPSHandler,
> urllib2.FTPHandler)
>
> # install this opener
> urllib2.install_opener(opener)
>
> # Go ahead, knock knock!
>
> req=urlli2.Request('ftp://ftp.gnu.org')
> data=urllib2.urlopen(req).read()


A couple of things to add: you don't need to add handlers that already
get added by default by build_opener (FTPHandler and HTTPHandler, for
example). ProxyHandler is one of these default handlers, so if your
environment is set up for it (http_proxy, etc. environment variables),
you don't need to supply a ProxyHandler (of course, if your
environ. *isn't*, then you do need to supply one, to give it the proxy
details). You don't need Request objects (unless you want to add
headers to a Request, or pass Requests around). You don't need to
install a global opener, unless your code expects it -- it's just a
convenience (or an inconvenience, sometimes).

Actually, did the OP say proxy basic auth. was involved? Don't
recall. I've never needed it for proxies, but there seems to be a
ProxyBasicAuthHandler in urllib2, so I guess that's what you meant to
use, rather than HTTPBasicAuthHandler (which is for website auth., not
proxy auth).

So, after all that, you end up with:

opener = urllib2.build_opener(urllib2.ProxyBasicAuthHandler )
data = opener.open('ftp://ftp.gnu.org').read()

(I like to close the response explicitly, though)


John

Anand Pillai 08-14-2003 06:10 AM

Re: FTP with urllib2 behind a proxy
 
jjl@pobox.com (John J. Lee) wrote in message news:<87wudhoh5c.fsf@pobox.com>...
> pythonguy@Hotpop.com (Anand Pillai) writes:

The code obviously caters to the bottom-line, in case
the programmer does not want to worry about environment
variables, would like to install the handlers himself
and his proxy needs authentication.

In the last case, the HTTP_PROXY env variable does not
help and you need to install the proxy handler yourself.

I copied this stuff from my program which needs all these
and more ( a USER-AGENT header for example ).

-Anand

> > You need to install a proxyhandler, authhandler,
> > ftphandler and httphandler. Then build yourself
> > an opener, that opens the doors for you ... :-)
> >
> > The following does the trick.
> >
> > proxy_handler = urllib2.ProxyHandler( {'http': 'myhttpproxy:80',
> > 'https' : 'myhttpsproxy:443',
> > 'ftp' : 'myftpproxy:21' } )
> >
> > opener= urllib2.build_opener(proxy_handler, urllib2.HTTPBasicAuthHandler(),
> > urllib2.HTTPHandler, urllib2.HTTPSHandler,
> > urllib2.FTPHandler)
> >
> > # install this opener
> > urllib2.install_opener(opener)
> >
> > # Go ahead, knock knock!
> >
> > req=urlli2.Request('ftp://ftp.gnu.org')
> > data=urllib2.urlopen(req).read()

>
> A couple of things to add: you don't need to add handlers that already
> get added by default by build_opener (FTPHandler and HTTPHandler, for
> example). ProxyHandler is one of these default handlers, so if your
> environment is set up for it (http_proxy, etc. environment variables),
> you don't need to supply a ProxyHandler (of course, if your
> environ. *isn't*, then you do need to supply one, to give it the proxy
> details). You don't need Request objects (unless you want to add
> headers to a Request, or pass Requests around). You don't need to
> install a global opener, unless your code expects it -- it's just a
> convenience (or an inconvenience, sometimes).
>
> Actually, did the OP say proxy basic auth. was involved? Don't
> recall. I've never needed it for proxies, but there seems to be a
> ProxyBasicAuthHandler in urllib2, so I guess that's what you meant to
> use, rather than HTTPBasicAuthHandler (which is for website auth., not
> proxy auth).
>
> So, after all that, you end up with:
>
> opener = urllib2.build_opener(urllib2.ProxyBasicAuthHandler )
> data = opener.open('ftp://ftp.gnu.org').read()
>
> (I like to close the response explicitly, though)
>
>
> John


John J. Lee 08-14-2003 11:54 AM

Re: FTP with urllib2 behind a proxy
 
pythonguy@Hotpop.com (Anand Pillai) writes:

> jjl@pobox.com (John J. Lee) wrote in message news:<87wudhoh5c.fsf@pobox.com>...
> > pythonguy@Hotpop.com (Anand Pillai) writes:

> The code obviously caters to the bottom-line, in case
> the programmer does not want to worry about environment
> variables, would like to install the handlers himself
> and his proxy needs authentication.


Sure, I understand that. I was just elaborating, no criticism was
intended!

It seems people are frightened to use urllib2 (perhaps partly because
the documentation has lots of sections -- though they're all very
short), so I like to point out that most of the time it's just a
matter of:

import urllib2
urllib2.urlopen("http://example.com/")


And most of the rest of the time, the only complications above that
are passing a Request instead of a URL (for adding headers and passing
around requests as first-class objects), and building your own opener
with build_opener (for choosing what features you want and don't want,
like authentication -- and adding your own features, of course).


John


All times are GMT. The time now is 03:16 AM.

Powered by vBulletin®. Copyright ©2000 - 2014, vBulletin Solutions, Inc.
SEO by vBSEO ©2010, Crawlability, Inc.