Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > FTP with urllib2 behind a proxy

Reply
Thread Tools

FTP with urllib2 behind a proxy

 
 
O. Koch
Guest
Posts: n/a
 
      08-07-2003
Until now, i know that ftplib doesn't support proxies and that i have
to use urllib2. But i don't know how to use the urllib2 correct. I
found some examples, but i don't understand them.

Is there anyone who can help me?
 
Reply With Quote
 
 
 
 
Anand Pillai
Guest
Posts: n/a
 
      08-13-2003
You need to install a proxyhandler, authhandler,
ftphandler and httphandler. Then build yourself
an opener, that opens the doors for you ...

The following does the trick.

proxy_handler = urllib2.ProxyHandler( {'http': 'myhttpproxy:80',
'https' : 'myhttpsproxy:443',
'ftp' : 'myftpproxy:21' } )

opener= urllib2.build_opener(proxy_handler, urllib2.HTTPBasicAuthHandler(),
urllib2.HTTPHandler, urllib2.HTTPSHandler,
urllib2.FTPHandler)

# install this opener
urllib2.install_opener(opener)

# Go ahead, knock knock!

req=urlli2.Request('ftp://ftp.gnu.org')
data=urllib2.urlopen(req).read()

Of course, replace the arbit proxy values I wrote
with your proxy values. If your proxy need authentication
, you will need to do a bit more here.

proxyauth='http://' + username + '@' + password + 'myproxy:myproxport'

Then your proxy handler becomes ( I am assuming a generic proxy
for all protocols here!)

proxy_handler = urllib2.ProxyHandler ( {'http' : proxyauth,
'https' : proxyauth,
'ftp' : proxyauth } )

HTH.

-Anand


http://www.velocityreviews.com/forums/(E-Mail Removed) (John J. Lee) wrote in message news:<(E-Mail Removed)>...
> (E-Mail Removed) (O. Koch) writes:
>
> > Until now, i know that ftplib doesn't support proxies and that i have
> > to use urllib2. But i don't know how to use the urllib2 correct. I
> > found some examples, but i don't understand them.
> >
> > Is there anyone who can help me?

>
> import urllib2
> response = urllib2.urlopen("ftp://ftp.example.com/pub/myfile")
> data = response.read()
> response.close()
>
>
> Does that do the trick?
>
>
> John

 
Reply With Quote
 
 
 
 
John J. Lee
Guest
Posts: n/a
 
      08-13-2003
(E-Mail Removed) (Anand Pillai) writes:

> You need to install a proxyhandler, authhandler,
> ftphandler and httphandler. Then build yourself
> an opener, that opens the doors for you ...
>
> The following does the trick.
>
> proxy_handler = urllib2.ProxyHandler( {'http': 'myhttpproxy:80',
> 'https' : 'myhttpsproxy:443',
> 'ftp' : 'myftpproxy:21' } )
>
> opener= urllib2.build_opener(proxy_handler, urllib2.HTTPBasicAuthHandler(),
> urllib2.HTTPHandler, urllib2.HTTPSHandler,
> urllib2.FTPHandler)
>
> # install this opener
> urllib2.install_opener(opener)
>
> # Go ahead, knock knock!
>
> req=urlli2.Request('ftp://ftp.gnu.org')
> data=urllib2.urlopen(req).read()


A couple of things to add: you don't need to add handlers that already
get added by default by build_opener (FTPHandler and HTTPHandler, for
example). ProxyHandler is one of these default handlers, so if your
environment is set up for it (http_proxy, etc. environment variables),
you don't need to supply a ProxyHandler (of course, if your
environ. *isn't*, then you do need to supply one, to give it the proxy
details). You don't need Request objects (unless you want to add
headers to a Request, or pass Requests around). You don't need to
install a global opener, unless your code expects it -- it's just a
convenience (or an inconvenience, sometimes).

Actually, did the OP say proxy basic auth. was involved? Don't
recall. I've never needed it for proxies, but there seems to be a
ProxyBasicAuthHandler in urllib2, so I guess that's what you meant to
use, rather than HTTPBasicAuthHandler (which is for website auth., not
proxy auth).

So, after all that, you end up with:

opener = urllib2.build_opener(urllib2.ProxyBasicAuthHandler )
data = opener.open('ftp://ftp.gnu.org').read()

(I like to close the response explicitly, though)


John
 
Reply With Quote
 
Anand Pillai
Guest
Posts: n/a
 
      08-14-2003
(E-Mail Removed) (John J. Lee) wrote in message news:<(E-Mail Removed)>...
> (E-Mail Removed) (Anand Pillai) writes:

The code obviously caters to the bottom-line, in case
the programmer does not want to worry about environment
variables, would like to install the handlers himself
and his proxy needs authentication.

In the last case, the HTTP_PROXY env variable does not
help and you need to install the proxy handler yourself.

I copied this stuff from my program which needs all these
and more ( a USER-AGENT header for example ).

-Anand

> > You need to install a proxyhandler, authhandler,
> > ftphandler and httphandler. Then build yourself
> > an opener, that opens the doors for you ...
> >
> > The following does the trick.
> >
> > proxy_handler = urllib2.ProxyHandler( {'http': 'myhttpproxy:80',
> > 'https' : 'myhttpsproxy:443',
> > 'ftp' : 'myftpproxy:21' } )
> >
> > opener= urllib2.build_opener(proxy_handler, urllib2.HTTPBasicAuthHandler(),
> > urllib2.HTTPHandler, urllib2.HTTPSHandler,
> > urllib2.FTPHandler)
> >
> > # install this opener
> > urllib2.install_opener(opener)
> >
> > # Go ahead, knock knock!
> >
> > req=urlli2.Request('ftp://ftp.gnu.org')
> > data=urllib2.urlopen(req).read()

>
> A couple of things to add: you don't need to add handlers that already
> get added by default by build_opener (FTPHandler and HTTPHandler, for
> example). ProxyHandler is one of these default handlers, so if your
> environment is set up for it (http_proxy, etc. environment variables),
> you don't need to supply a ProxyHandler (of course, if your
> environ. *isn't*, then you do need to supply one, to give it the proxy
> details). You don't need Request objects (unless you want to add
> headers to a Request, or pass Requests around). You don't need to
> install a global opener, unless your code expects it -- it's just a
> convenience (or an inconvenience, sometimes).
>
> Actually, did the OP say proxy basic auth. was involved? Don't
> recall. I've never needed it for proxies, but there seems to be a
> ProxyBasicAuthHandler in urllib2, so I guess that's what you meant to
> use, rather than HTTPBasicAuthHandler (which is for website auth., not
> proxy auth).
>
> So, after all that, you end up with:
>
> opener = urllib2.build_opener(urllib2.ProxyBasicAuthHandler )
> data = opener.open('ftp://ftp.gnu.org').read()
>
> (I like to close the response explicitly, though)
>
>
> John

 
Reply With Quote
 
John J. Lee
Guest
Posts: n/a
 
      08-14-2003
(E-Mail Removed) (Anand Pillai) writes:

> (E-Mail Removed) (John J. Lee) wrote in message news:<(E-Mail Removed)>...
> > (E-Mail Removed) (Anand Pillai) writes:

> The code obviously caters to the bottom-line, in case
> the programmer does not want to worry about environment
> variables, would like to install the handlers himself
> and his proxy needs authentication.


Sure, I understand that. I was just elaborating, no criticism was
intended!

It seems people are frightened to use urllib2 (perhaps partly because
the documentation has lots of sections -- though they're all very
short), so I like to point out that most of the time it's just a
matter of:

import urllib2
urllib2.urlopen("http://example.com/")


And most of the rest of the time, the only complications above that
are passing a Request instead of a URL (for adding headers and passing
around requests as first-class objects), and building your own opener
with build_opener (for choosing what features you want and don't want,
like authentication -- and adding your own features, of course).


John
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Web Service invocation from behind proxy behind firewall Kumarforg ASP .Net Web Services 0 08-03-2004 07:15 AM
FTP, Proxy and urllib2 Martin Brodbeck Python 4 06-08-2004 05:13 PM
Help to set up tomcat 4.1.27 behind socks proxy to access database outside the proxy vidy5300 Java 0 06-07-2004 02:23 PM
Re: FTP with urllib2 behind a proxy Bob McCormick Python 1 02-25-2004 09:53 PM
Re: FTP with urllib2 behind a proxy Bob McCormick Python 0 02-25-2004 01:56 AM



Advertisments