Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Wierd M2Crypto bug - phony "peer did not return certificate" error

Reply
Thread Tools

Wierd M2Crypto bug - phony "peer did not return certificate" error

 
 
John Nagle
Guest
Posts: n/a
 
      01-10-2007
Here's a wierd problem:

I have a little test case for M2Crypto, which just opens up SSL connections to
web servers and reads their certificates. This works fine.

But if I execute

socket.setdefaulttimeout(5.0)

so that the sockets don't wait too long if there's no SSL server, I get
a "peer did not return certificate" exception every time.

Environment:
Windows 2000 SP 4
M2Crypto: M2Crypto-0.17.win32-py2.4.exe
Python: Python 2.4.4 (#71, Oct 18 2006, 08:34:43)
OpenSSL: Win32 OpenSSL v0.9.8d

Notes:
Running M2Crypto does NOT use the version of OpenSSL that comes
with Python. You have to install OpenSSL separately, or
M2Crypto's DLL won't find the OpenSSL DLLS.

There's a good chance that this may be related to:

[ python-Bugs-1098618 ] socket.setdefaulttimeout() breaks smtplib.starttls()

But that should have been fixed in Python 2.4.4, right? Or did that patch
not make it in?

John Nagle

(And no, I can't run Python 2.5, because MySQLdb support doesn't work for
2.5 yet.)



 
Reply With Quote
 
 
 
 
Heikki Toivonen
Guest
Posts: n/a
 
      01-10-2007
John Nagle wrote:
> I have a little test case for M2Crypto, which just opens up SSL
> connections to
> web servers and reads their certificates. This works fine.
>
> But if I execute
>
> socket.setdefaulttimeout(5.0)


Yup, this is a known problem, this breaks all M2Crypto code that uses
sockets. The Twisted wrapper part still works, as it hands the network
activity to Twisted.

If you can make do without setdefaulttimeout you should be fine.

This is bug https://bugzilla.osafoundation.org/show_bug.cgi?id=2341

The bug has a patch that works on Linux, but it would need Windows and
Mac specific parts before it can be checked in.

--
Heikki Toivonen
 
Reply With Quote
 
 
 
 
nagle@animats.com
Guest
Posts: n/a
 
      01-11-2007
That's a problem for me. I need short timeouts; I'm accessing sites
that might or might not have SSL support, and I need to quickly time
out when there's no SSL server.

PyOpenSSL handles timeout correctly, but M2Crypto does not. On the
other hand, M2Crypto actually checks certificates, which PyOpenSSL does
not. So we have two broken implementations.

Python needs a merge here. Read Guido van Rossum's rant on M2Crypto:

http://www.artima.com/weblogs/viewpost.jsp?thread=95863

Both of these packages contain wrappers for OpenSSL, but both wrappers
are incompatible and buggy. M2Crypto also has some Python components.
The PyOpenSSL wrapper, which ships with Python, doesn't expose enough
of the OpenSSL API. M2Crypto exposes more of the API, but doesn't work
as well.

What's probably needed is to implement the additional API functions of
the M2Crypto wrapper in the PyOpenSSL wrapper, so that the M2Crypto
components written in Python could be used with it. Then we'd have one
good implementation instead of two broken ones.

John Nagle
http://www.velocityreviews.com/forums/(E-Mail Removed)

(News feed broken, using Google Groups as backup.)

 
Reply With Quote
 
Heikki Toivonen
Guest
Posts: n/a
 
      01-11-2007
(E-Mail Removed) wrote:
> That's a problem for me. I need short timeouts; I'm accessing sites
> that might or might not have SSL support, and I need to quickly time
> out when there's no SSL server.


You should be able to do short timeouts, just not using the global
setdefaulttimeout. Have you tried Connection.set_socket_read/write_timeout?

Also like I mentioned before, if you use the Twisted wrapper and let
Twisted handle network stuff you should be fine.

> Python needs a merge here. Read Guido van Rossum's rant on M2Crypto:
>
> http://www.artima.com/weblogs/viewpost.jsp?thread=95863


That's old news, I believe I have fixed all of the issues mentioned
there already in the 0.16 release.

> What's probably needed is to implement the additional API functions of
> the M2Crypto wrapper in the PyOpenSSL wrapper, so that the M2Crypto
> components written in Python could be used with it. Then we'd have one
> good implementation instead of two broken ones.


M2Crypto and pyOpenSSL (and pyOpenSSL-extended, which you might want to
take a look at as well) are implemented pretty differently, so merging
seems unlikely. They are pretty small code wise, though, so it is not an
impossible task.

Personally I think I'd prefer if Python stdlib contained a better SSL
module that did at least all the checks required for safe SSL
connection. (Yeah, yeah, maybe I need to write it myself if nobody else
gets to it

--
Heikki Toivonen
 
Reply With Quote
 
John Nagle
Guest
Posts: n/a
 
      01-12-2007
Heikki Toivonen wrote:
> (E-Mail Removed) wrote:
>
>> That's a problem for me. I need short timeouts; I'm accessing sites
>>that might or might not have SSL support, and I need to quickly time
>>out when there's no SSL server.

>
>
> You should be able to do short timeouts, just not using the global
> setdefaulttimeout. Have you tried Connection.set_socket_read/write_timeout?


Yes. That does not affect the connect timeout; it's only effective once
the connection has been opened. And adjusting the session timeout
just recreates the blocking/non blocking problem.

Incidentally, "get_socket_read_timeout()" doesn't work. Generates
"EXCEPTION at socket level: unpack str size does not match format",
every time, at least with Python 2.4 on Windows. The lower level
function returns one number as a string, like "7200", and the unpack
function tries to unpack it as "ll", which fails.

> Also like I mentioned before, if you use the Twisted wrapper and let
> Twisted handle network stuff you should be fine.


That would mean struggling with Twisted and dealing with its bugs.
(For example, has the MySQLdb mess been resolved for Twisted's API?)
This isn't a long-running application; it usually runs as a CGI program.
So Twisted is inappropriate.

> M2Crypto and pyOpenSSL (and pyOpenSSL-extended, which you might want to
> take a look at as well) are implemented pretty differently, so merging
> seems unlikely. They are pretty small code wise, though, so it is not an
> impossible task.


That seened good idea, and so I downloaded the source and tried to build
it on a Windows machine to run with Python 2.4. But that build needs
(exactly) Visual Studio 7.1, which I don't have. It also wants a specific
version of OpenSSL, and has a build file which seems to assume a 1998 version
of Visual C++. The last change was in 1995, and it's still at an 0.x version,
so it's effectively abandonware. I have some doubts that it really works
any more. I saw some Python 2.2/2.3 specific code in there.

I could try building on a Linux system, but it's useful to me if Python
works on both Windows and Linux.
>
> Personally I think I'd prefer if Python stdlib contained a better SSL
> module that did at least all the checks required for safe SSL
> connection. (Yeah, yeah, maybe I need to write it myself if nobody else
> gets to it


What I need is proper SSL operation, certificate chain validation,
useful exception info when a connection fails (including why), and
read acess to the certificate itself in some reasonably sane form.
And I need to time out an SSL connection if it can't connect and
verify within 4 seconds. It's embarassing that this doesn't work,
despite four different wrappers for OpenSSL.

I'm willing to spend a few hundred dollars towards making this happen.

John Nagle
 
Reply With Quote
 
John Nagle
Guest
Posts: n/a
 
      01-12-2007
OK, I have a one-line fix.

To Connection.py of M2Crypto:

def connect(self, addr):
self.socket.connect(addr)
self.addr = addr
self.socket.settimeout(None) # Back to normal timeout (NEW)
self.setup_ssl()
self.set_connect_state()
ret = self.connect_ssl()
check = getattr(self, 'postConnectionCheck', self.clientPostConnectionCheck)
if check is not None:
if not check(self.get_peer_cert(), self.addr[0]):
raise Checker.SSLVerificationError, 'post connection check failed'
return ret


After the socket is connected, we turn its timeout off.

The effect is that if the caller sets

socket.setdefaulttimeout(timeoutsecs)

before creating the Connection object, that will set the connection
timeout. The socket will be created as non-blocking, but before
any reads or writes are done, we clear the timeout, making it blocking
again. So the connection will time out as requested, but we won't
get errors because the socket is non-blocking and is being used by
code that expects it to block.

Gets usable connection timeouts, and prevents bogus "peer did not
return certificate" errors.

John Nagle
Animats

 
Reply With Quote
 
Heikki Toivonen
Guest
Posts: n/a
 
      01-13-2007
John Nagle wrote:
> def connect(self, addr):
> self.socket.connect(addr)
> self.addr = addr
> self.socket.settimeout(None) # Back to normal timeout (NEW)


I am not sure if this is always appropriate. In fact, doesn't this just
eliminate the timeout completely from this connection?

In your case you could achieve what you want by calling
connection_object.socket.settimeout(4.0) method just before calling
connect, without needing to edit M2Crypto itself.

setdefaulttimeout, socket.settimeout and socket.setblocking are pretty
confusing...

--
Heikki Toivonen
 
Reply With Quote
 
John Nagle
Guest
Posts: n/a
 
      01-13-2007
Heikki Toivonen wrote:
> John Nagle wrote:
>
>>def connect(self, addr):
>> self.socket.connect(addr)
>> self.addr = addr
>> self.socket.settimeout(None) # Back to normal timeout (NEW)

>
>
> I am not sure if this is always appropriate. In fact, doesn't this just
> eliminate the timeout completely from this connection?
>
> In your case you could achieve what you want by calling
> connection_object.socket.settimeout(4.0) method just before calling
> connect, without needing to edit M2Crypto itself.


No, this actually works right. It seems that the default timeout
controls the timeout on TCP connect, but puts the socket into
non-blocking mode. So, if as soon as a TCP connection is
opened, but before the SSL handshake starts, the socket timeout
is set to None, then the socket returns to blocking mode before
its first read.

At least on Windows. More later on how this works on Linux.

John Nagle
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Nauseating. Even Leica playing the phony "I.S." game Rich Digital Photography 12 04-10-2007 05:54 AM
Re: Wierd M2Crypto bug - phony "peer did not return certificate" error Jean-Paul Calderone Python 0 01-19-2007 03:16 PM
creating Phony links Buddy Ackerman ASP .Net 2 03-09-2005 03:55 AM
Phony sites about phoney towns. bambam@nospam.tnx Computer Support 16 02-12-2005 06:08 PM



Advertisments