Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Pickling over a socket

Reply
Thread Tools

Pickling over a socket

 
 
Roger Alexander
Guest
Posts: n/a
 
      04-19-2011
Hi,

I'm trying to understand how to pickle Python objects over a TCP
socket.

In the example below (based on code from Foundations of Python Network
Programming), a client creates a dictionary (lines 34-3 and uses
pickle.dump at line 42 to write the pickled object using file handle
make from a socket. The server de-pickles with pickle.load (line 24),
again using a file handle made from a socket.

When I run the program, the following output is produced:

Listening at ('127.0.0.1', 1060)
Accepted connection from ('127.0.0.1', 4993
Traceback (most recent call last):
File "pickles.py", line 24, in <module>
d = pickle.load( s_fh )
File "/usr/local/lib/python2.7/pickle.py", line 1378, in load
return Unpickler(file).load()
File "/usr/local/lib/python2.7/pickle.py", line 857, in load
key = read(1)
File "/usr/local/lib/python2.7/socket.py", line 380, in read
data = self._sock.recv(left)
socket.error: [Errno 107] Transport endpoint is not connected

I'm at a loss, can anyone provide any guidance?

Thanks,

Roger Alexander

1 import pickle
2 import socket, sys
3
4 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
5
6 HOST = sys.argv.pop() if len(sys.argv) == 3 else '127.0.0.1'
7 PORT = 1060
8
9 if sys.argv[1:] == ['server']:
10
11 s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
12 s.bind((HOST, PORT))
13 s.listen(1)
14
15 print 'Listening at', s.getsockname()
16
17 sc, sockname = s.accept()
18
19 print 'Accepted connection from', sockname
20
21 sc.shutdown(socket.SHUT_WR)
22 sf = s.makefile( "rb" )
23
24 d = pickle.load(sf)
25
26 sc.close()
27 s.close()
28
29 elif sys.argv[1:] == ['client']:
30
31 s.connect((HOST, PORT))
32 s.shutdown(socket.SHUT_RD)
33
34 d = dict()
35
36 d[ 'Name' ] = 'Jake Thompson.'
37 d[ 'Age' ] = 25
38 d[ 'Location' ] = 'Washington, D.C.'
39
40 sf = s.makefile( "wb" )
41
42 pickle.dump( d, sf, pickle.HIGHEST_PROTOCOL )
43
44 s.close()
45
46 else:
47 print >>sys.stderr, 'usage: streamer.py server|client [host]'
 
Reply With Quote
 
 
 
 
Chris Rebert
Guest
Posts: n/a
 
      04-19-2011
On Tue, Apr 19, 2011 at 11:53 AM, Roger Alexander <(E-Mail Removed)> wrote:
> Hi,
>
> I'm trying to understand how to pickle Python objects over a TCP
> socket.
>
> In the example below (based on code from Foundations of Python Network
> Programming), a client creates a dictionary (lines 34-3 and uses
> pickle.dump at line 42 to write the pickled object using file handle
> make from a socket. The server de-pickles with pickle.load *(line 24),
> again using a file handle made from a socket.
>
> When I run the program, the following output is produced:
>
> * *Listening at ('127.0.0.1', 1060)
> * *Accepted connection from ('127.0.0.1', 4993
> * *Traceback (most recent call last):
> * *File "pickles.py", line 24, in <module>
> * * * *d = pickle.load( s_fh )
> * *File "/usr/local/lib/python2.7/pickle.py", line 1378, in load
> * * * *return Unpickler(file).load()
> * *File "/usr/local/lib/python2.7/pickle.py", line 857, in load
> * * * *key = read(1)
> * *File "/usr/local/lib/python2.7/socket.py", line 380, in read
> * * * *data = self._sock.recv(left)
> * *socket.error: [Errno 107] Transport endpoint is not connected
>
> I'm at a loss, can anyone provide any guidance?
>
> Thanks,
>
> Roger Alexander
>
> *1 *import pickle
> *2 *import socket, sys
> *3
> *4 *s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
> *5
> *6 *HOST = sys.argv.pop() if len(sys.argv) == 3 else '127..0.0.1'
> *7 *PORT = 1060
> *8
> *9 *if sys.argv[1:] == ['server']:
> 10
> 11 * * *s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
> 12 * * *s.bind((HOST, PORT))
> 13 * * *s.listen(1)
> 14
> 15 * * *print 'Listening at', s.getsockname()
> 16
> 17 * * *sc, sockname = s.accept()
> 18
> 19 * * *print 'Accepted connection from', sockname
> 20
> 21 * * *sc.shutdown(socket.SHUT_WR)


[Haven't done any network programming, so please excuse the naivete of
this suggestion.]

Have you tried removing line #21 and/or #32?

http://docs.python.org/library/socke...ocket.shutdown :
"socket.shutdown(how) - Shut down one or both halves of the
connection. [...] Depending on the platform, shutting down one half of
the connection can also close the opposite half"

Cheers,
Chris
--
http://blog.rebertia.com

> 22 * * *sf = s.makefile( "rb" )
> 23
> 24 * * *d = pickle.load(sf)
> 25
> 26 * * *sc.close()
> 27 * * *s.close()
> 28
> 29 *elif sys.argv[1:] == ['client']:
> 30
> 31 * * *s.connect((HOST, PORT))
> 32 * * *s.shutdown(socket.SHUT_RD)

<snip>
> 42 * * *pickle.dump( d, sf, pickle.HIGHEST_PROTOCOL )
> 43
> 44 * * *s.close()

 
Reply With Quote
 
 
 
 
Chris Angelico
Guest
Posts: n/a
 
      04-19-2011
On Wed, Apr 20, 2011 at 4:53 AM, Roger Alexander <(E-Mail Removed)> wrote:
> Hi,
>
> I'm trying to understand how to pickle Python objects over a TCP
> socket.
>
> In the example below (based on code from Foundations of Python Network
> Programming), a client creates a dictionary (lines 34-3 and uses
> pickle.dump at line 42 to write the pickled object using file handle
> make from a socket. The server de-pickles with pickle.load *(line 24),
> again using a file handle made from a socket.


Whenever there's a problem, create simplicity. I would recommend not
using the file-from-socket method, and simply using pickle.dumps() and
pickle.loads() to pickle to/from strings; those strings can then be
sent/received over the socket using standard recv/send functions.

Also, Chris Rebert's idea is a good one, and worth trying.

Chris Angelico
 
Reply With Quote
 
Dan Stromberg
Guest
Posts: n/a
 
      04-19-2011
On Tue, Apr 19, 2011 at 11:53 AM, Roger Alexander <(E-Mail Removed)> wrote:
> Hi,
>
> I'm trying to understand how to pickle Python objects over a TCP
> socket.
>
> In the example below (based on code from Foundations of Python Network
> Programming), a client creates a dictionary (lines 34-3 and uses
> pickle.dump at line 42 to write the pickled object using file handle
> make from a socket. The server de-pickles with pickle.load *(line 24),
> again using a file handle made from a socket.
>
> When I run the program, the following output is produced:
>
> * *Listening at ('127.0.0.1', 1060)
> * *Accepted connection from ('127.0.0.1', 4993
> * *Traceback (most recent call last):
> * *File "pickles.py", line 24, in <module>
> * * * *d = pickle.load( s_fh )
> * *File "/usr/local/lib/python2.7/pickle.py", line 1378, in load
> * * * *return Unpickler(file).load()
> * *File "/usr/local/lib/python2.7/pickle.py", line 857, in load
> * * * *key = read(1)
> * *File "/usr/local/lib/python2.7/socket.py", line 380, in read
> * * * *data = self._sock.recv(left)
> * *socket.error: [Errno 107] Transport endpoint is not connected
>
> I'm at a loss, can anyone provide any guidance?
>
> Thanks,
>
> Roger Alexander


I played around with it until something worked, and ended up with the
below. The most significant change was probably using sc.makefile
instead of s.makefile in the server, but I seemed to need some data
framing too despite the pickling. It's possible you won't need that
if you just flush your file in the client; I don't much pickling
experience - in particular, I don't know if you can concatenate
pickled objects and load them serially from a file-like object without
any (additional) framing.

I like to use bufsock for this sort of stuff, but I'm probably unique
in that. ^_^ http://stromberg.dnsalias.org/~strombrg/bufsock.html

#!/usr/bin/python

import time
import pickle
import socket, sys
import pprint

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

HOST = sys.argv.pop() if len(sys.argv) == 3 else '127.0.0.1'
PORT = 1060

if sys.argv[1:] == ['server']:

s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((HOST, PORT))
s.listen(1)

print 'Listening at', s.getsockname()

sc, sockname = s.accept()

print 'Accepted connection from', sockname

sf = sc.makefile( "rb" )

length_list = []
while True:
char = sf.read(1)
if char == '\n':
break
else:
length_list.append(int(char))
length = 0
for digit in length_list:
length = length * 10 + digit
data = sf.read(length)

d = pickle.loads(data)

pprint.pprint(d)

sc.shutdown(socket.SHUT_RDWR)
sc.close()
s.close()

elif sys.argv[1:] == ['client']:

s.connect((HOST, PORT))
# s.shutdown(socket.SHUT_RD)

d = dict()

d[ 'Name' ] = 'Jake Thompson.'
d[ 'Age' ] = 25
d[ 'Location' ] = 'Washington, D.C.'

sf = s.makefile( "wb" )

string = pickle.dumps( d, pickle.HIGHEST_PROTOCOL )
sf.write('%d\n' % len(string))
sf.write(string)
sf.flush()

#time.sleep(10)
sf.close()
s.shutdown(socket.SHUT_RDWR)
# s.close()

else:
print >>sys.stderr, 'usage: streamer.py server|client [host]'
 
Reply With Quote
 
Chris Angelico
Guest
Posts: n/a
 
      04-19-2011
On Wed, Apr 20, 2011 at 5:30 AM, Dan Stromberg <(E-Mail Removed)> wrote:
> I played around with it until something worked, and ended up with the
> below. *The most significant change was probably using sc.makefile
> instead of s.makefile in the server...


Oh! I didn't notice that in the OP. Yep, that would do it!

ChrisA
 
Reply With Quote
 
Roger Alexander
Guest
Posts: n/a
 
      04-19-2011
Thanks everybody, got it working.

I appreciate the help!

Roger.
 
Reply With Quote
 
Jean-Paul Calderone
Guest
Posts: n/a
 
      04-20-2011
On Apr 19, 6:27*pm, Roger Alexander <(E-Mail Removed)> wrote:
> Thanks everybody, got it working.
>
> *I appreciate the help!
>
> Roger.


It's too bad none of the other respondents pointed out to you that you
_shouldn't do this_! Pickle is not suitable for use over the network
like this. Your server accepts arbitrary code from clients and
executes it. It is completely insecure. Do not use pickle and
sockets together. Notice the large red box at the top of <http://
docs.python.org/library/pickle.html>.

Jean-Paul
 
Reply With Quote
 
Bastian Ballmann
Guest
Posts: n/a
 
      04-20-2011
Am Tue, 19 Apr 2011 19:28:50 -0700 (PDT)
schrieb Jean-Paul Calderone <(E-Mail Removed)>:

> It is completely insecure. Do not use pickle and
> sockets together.


Yes pickle is like eval, but that doesnt mean that one should never
ever use it over a socket connection.
What about ssl sockets where client and server authenticate each other?
Or you encrypt the pickle dump with symmetric encryption and only load
it if you can decrypt it? There are ways to ensure that the data you
get can be handled as trusted.
Greets

Basti

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAk2ugM8ACgkQEQHD8bvs9q1IIACfeHPy6/Zd7jQ0haEZO7JreUYt
pCYAnRBZEXSwxYBpHlqRtWjJZRdGk0qv
=TaGe
-----END PGP SIGNATURE-----

 
Reply With Quote
 
Chris Angelico
Guest
Posts: n/a
 
      04-20-2011
On Wed, Apr 20, 2011 at 4:44 PM, Bastian Ballmann <(E-Mail Removed)> wrote:
> Yes pickle is like eval, but that doesnt mean that one should never
> ever use it over a socket connection.
> What about ssl sockets where client and server authenticate each other?
> Or you encrypt the pickle dump with symmetric encryption and only load
> it if you can decrypt it? There are ways to ensure that the data you
> get can be handled as trusted.


No, I disagree. And I'll cite Caesary as evidence of why.

Caesary is a multiplayer game that uses Flash as its client. (I'm told
the back end is Java, which would explain why it starts lagging
horribly when everyone's online at once.) It has some measure of
authentication of the client, but it's not difficult to spoof;
obviously you could go more elaborate and harder to spoof, but that
still doesn't solve the problem. Even public/private key systems won't
work here; someone could get hold of your client and its private key,
and poof.

Caesary uses an Adobe Message Format system, whereby complex objects
get serialized and transmitted in both directions. It's fundamentally
the same as pickling. When I started poking around with things, it
took me very little time to start transmitting my own requests to the
server; my requests were benign (asking it for information), but other
people figured out the same thing and were rather less ethical.

That's why I tend to use and create much simpler protocols for network
transmission. Also, I like to use a MUD client to test my servers,
ergo textual protocols similar to SMTP. Sure, it may be a tad more
verbose than some, but it's usually easy to parse and verify.

Chris Angelico
 
Reply With Quote
 
Bastian Ballmann
Guest
Posts: n/a
 
      04-20-2011
Am Wed, 20 Apr 2011 16:59:19 +1000
schrieb Chris Angelico <(E-Mail Removed)>:

> Even public/private key systems won't
> work here; someone could get hold of your client and its private key,
> and poof.


Oh yeah but than all kinds of trusted computing wont work. Sure
one can see it on the net these days looking at the rsa or commodo or
ps3 hack and the like.

No system is totally secure. You can _always_ poke around if a program
uses user input. For example one can totally own a complete computer by
nothing more than a single sql injection attack even if the programmer
implemented some filters. Now would you say one shouldnt use sql
databases cause of that?

My point is using ssl authentication / encryption together with another
symmetric encryption builds up two layers, which I would say is secure
enough to handle the data as trusted.

Greets

Basti

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAk2ujHsACgkQEQHD8bvs9q3a+gCgt7OPN8CJqh em9hMa77a7+Ud+
U4UAn2uHBqOWYaC94xY8RwM9OPhZCXqk
=isJb
-----END PGP SIGNATURE-----

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: socket.unbind or socket.unlisten? - socket.error: (48, 'Addressalready in use') Steve Holden Python 0 02-01-2009 12:45 PM
Re: socket.unbind or socket.unlisten? - socket.error: (48, 'Addressalready in use') Laszlo Nagy Python 0 02-01-2009 07:37 AM
socket.unbind or socket.unlisten? - socket.error: (48, 'Addressalready in use') Laszlo Nagy Python 1 01-27-2009 05:05 PM
Re: socket.unbind or socket.unlisten? - socket.error: (48,'Address already in use') Jean-Paul Calderone Python 0 01-27-2009 01:41 PM
VOIP over VPN over TCP over WAP over 3G Theo Markettos UK VOIP 2 02-14-2008 03:27 PM



Advertisments