Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > recv_into(bytearray) complains about a "pinned buffer"

Reply
Thread Tools

recv_into(bytearray) complains about a "pinned buffer"

 
 
Andrew Dalke
Guest
Posts: n/a
 
      01-31-2010
In Python 2.6 I can't socket.recv_into(a byte array instance). I get a
TypeError which complains about a "pinned buffer". I have only an
inkling of what that means. Since an array.array("b") works there, and
since it works in Python 3.1.1, and since I thought the point of a
bytearray was to make things like recv_into easier, I think this
exception is a bug in Python 2.6.

I want to double check before posting it to the tracker.

Here's my reproducibles:

Python 2.6.1 (r261:67515, Jul 7 2009, 23:51:51)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> sock = socket.socket()
>>> sock.connect( ("python.org", 80) )
>>> sock.send(b"GET / HTTP/1.0\r\n\r\n")

18
>>> buf = bytearray(b" " * 10)
>>> sock.recv_into(buf)

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: recv_into() argument 1 must be pinned buffer, not bytearray
>>>


I expected a bytearray to work there. In fact, I thought the point of
bytearray was to allow this to work.

By comparison, an array of bytes does work:

>>> import array
>>> arr = array.array("b")
>>> arr.extend(map(ord, "This is a test"))
>>> len(arr)

14
>>> sock.recv_into(arr)

14
>>> arr

array('b', [72, 84, 84, 80, 47, 49, 46, 49, 32, 51, 48, 50, 32, 70])
>>> "".join(map(chr, arr))

'HTTP/1.1 302 F'

I don't even know what a "pinned buffer" means, and searching
python.org isn't helpful.

Using a bytearray in Python 3.1.1 *does* work:

Python 3.1.1 (r311:74480, Jan 31 2010, 23:07:16)
[GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> sock = socket.socket()
>>> sock.connect( ("python.org", 80) )
>>> sock.send(b"GET / HTTP/1.0\r\n\r\n")

18
>>> buf = bytearray(b" " * 10)
>>> sock.recv_into(buf)

10
>>> buf

bytearray(b'HTTP/1.1 3')
>>>


Is this a bug in Python 2.6 or a deliberate choice regarding
implementation concerns I don't know about?

If it's a bug, I'll add it to the tracker.

Andrew Dalke
http://www.velocityreviews.com/forums/(E-Mail Removed)
 
Reply With Quote
 
 
 
 
Antoine Pitrou
Guest
Posts: n/a
 
      02-01-2010

Hello Andrew,

> I don't even know what a "pinned buffer" means, and searching python.org
> isn't helpful.
>
> Using a bytearray in Python 3.1.1 *does* work:
> [...]


Agreed, the error message is cryptic.
The problem is that socket.recv_into() in 2.6 doesn't recognize the new
buffer API which is needed to accept bytearray objects.
(it does in 3.1, because the old buffer API doesn't exist anymore there)

You could open an issue on the bug tracker for this.

Thank you

Antoine.

 
Reply With Quote
 
 
 
 
Andrew Dalke
Guest
Posts: n/a
 
      02-01-2010
On Feb 1, 1:04*am, Antoine Pitrou <(E-Mail Removed)> wrote:
> The problem is that socket.recv_into() in 2.6 doesn't recognize the new
> buffer API which is needed to accept bytearray objects.
> (it does in 3.1, because the old buffer API doesn't exist anymore there)


That's about what I thought it was, but I don't know if this was a
deliberate choice or accidental.

BTW, 2.7 (freshly built from version control) also has the same
exception.

> You could open an issue on the bug tracker for this.


I've done that. It's http://bugs.python.org/issue7827 .

Cheers!
Andrew
(E-Mail Removed)

 
Reply With Quote
 
Martin v. Loewis
Guest
Posts: n/a
 
      02-01-2010
> In Python 2.6 I can't socket.recv_into(a byte array instance). I get a
> TypeError which complains about a "pinned buffer". I have only an
> inkling of what that means.


A pinned buffer is one that cannot move in memory, even if another
thread tries to behind your back. Typically, resizable containers
are not inherently pinned, and "a user" (i.e. the API function) must
explicitly pin it, which recv_into fails to do.

> Is this a bug in Python 2.6 or a deliberate choice regarding
> implementation concerns I don't know about?


It's actually a bug also that you pass an array; doing so *should*
give the very same error.

It may be that the bugs surrounding buffers will never get fully
resolved in the lifetime of Python 2.x, so I would probably just ignore
recv_into.

Regards,
Martin
 
Reply With Quote
 
Antoine Pitrou
Guest
Posts: n/a
 
      02-01-2010
Le Mon, 01 Feb 2010 03:30:56 +0100, Martin v. Loewis a √©crit¬*:
>
>> Is this a bug in Python 2.6 or a deliberate choice regarding
>> implementation concerns I don't know about?

>
> It's actually a bug also that you pass an array; doing so *should* give
> the very same error.


Well, if you can give neither an array nor a bytearray to recv_into(),
what *could* you give it?

recv_into() should simply be fixed to use the new buffer API, as it does
in 3.x.

 
Reply With Quote
 
Martin v. Loewis
Guest
Posts: n/a
 
      02-01-2010
Antoine Pitrou wrote:
> Le Mon, 01 Feb 2010 03:30:56 +0100, Martin v. Loewis a écrit :
>>> Is this a bug in Python 2.6 or a deliberate choice regarding
>>> implementation concerns I don't know about?

>> It's actually a bug also that you pass an array; doing so *should* give
>> the very same error.

>
> Well, if you can give neither an array nor a bytearray to recv_into(),
> what *could* you give it?


My recommendation would be to not use recv_into in 2.x, but only in 3.x.

> recv_into() should simply be fixed to use the new buffer API, as it does
> in 3.x.


I don't think that's the full solution. The array module should also
implement the new buffer API, so that it would also fail with the old
recv_into.

Regards,
Martin
 
Reply With Quote
 
Andrew Dalke
Guest
Posts: n/a
 
      02-02-2010
On Feb 2, 12:12*am, Martin v. Loewis wrote:
> My recommendation would be to not use recv_into in 2.x, but only in 3.x.


> I don't think that's the full solution. The array module should also
> implement the new buffer API, so that it would also fail with the old
> recv_into.


Okay. But recv_into was added in 2.5 and the test case in
2.6's test_socket.py clearly allows an array there:


def testRecvInto(self):
buf = array.array('c', ' '*1024)
nbytes = self.cli_conn.recv_into(buf)
self.assertEqual(nbytes, len(MSG))
msg = buf.tostring()[:len(MSG)]
self.assertEqual(msg, MSG)

Checking koders and Google Code search engines, I found one project
which used recv_into, with the filename bmpreceiver.py . It
uses a array.array("B", [0] * length) .

Clearly it was added to work with an array, and it's
being used with an array. Why shouldn't people use it
with Python 2.x?

Andrew
(E-Mail Removed)
 
Reply With Quote
 
Antoine Pitrou
Guest
Posts: n/a
 
      02-02-2010
Le Tue, 02 Feb 2010 00:12:34 +0100, Martin v. Loewis a √©crit¬*:
>> recv_into() should simply be fixed to use the new buffer API, as it
>> does in 3.x.

>
> I don't think that's the full solution. The array module should also
> implement the new buffer API, so that it would also fail with the old
> recv_into.


There was a patch for this in http://bugs.python.org/issue6071 , but the
bug was closed when hashlib was "fixed".


 
Reply With Quote
 
Martin v. Loewis
Guest
Posts: n/a
 
      02-02-2010
> Clearly it was added to work with an array, and it's
> being used with an array. Why shouldn't people use it
> with Python 2.x?


Because it's not thread-safe; it may crash the interpreter if used
incorrectly.

Of course, if you don't share the array across threads, it can be safe
to use.

Regards,
Martin
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Numeric CSS class - Firefox complains but Opera OK john@starmarkassociates.co.uk HTML 1 06-22-2006 07:22 PM
debugger complains: auto-attach to process 'aspnet_wp.exe' =?Utf-8?B?c2lycGVsaWRvcg==?= ASP .Net 0 08-29-2005 04:41 PM
Linker complains about symbol it can't possibly use Old Wolf C++ 4 05-25-2005 03:21 PM
CP 3200 complains about memory card Cooter Digital Photography 4 08-25-2004 12:42 AM
an ingrate newbie complains Elaine Jackson Python 10 02-07-2004 12:58 PM



Advertisments