Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > yEnc implementation in Python, bit slow

Reply
Thread Tools

yEnc implementation in Python, bit slow

 
 
Freddie
Guest
Posts: n/a
 
      08-04-2003
Hi,

I posted a while ago for some help with my word finder program, which is now
quite a lot faster than I could manage. Thanks to all who helped

This time, I've written a basic batch binary usenet poster in Python, but
encoding the data into yEnc format is fairly slow. Is it possible to improve
the routine any, WITHOUT using non-standard libraries? I don't want to have
to rely on something strange

yEncode1 tends to be slightly faster here for me on my K6/2 500:

$ python2.3 testyenc.py
yEncode1 401563 1.82
yEncode1 401563 1.83
yEncode2 401562 1.83
yEncode2 401562 1.83

Any help would be greatly appreciated

Freddie


import struct
import time
from zlib import crc32

def timing(f, n, a):
print f.__name__,
r = range(n)
t1 = time.clock()
for i in r:
#f(a); f(a); f(a); f(a); f(a); f(a); f(a); f(a); f(a); f(a)
f(a)
t2 = time.clock()
print round(t2-t1, 3)

def yEncSetup():
global YENC
YENC = [''] * 256

for I in range(256):
O = (I + 42) % 256
if O in (0, 10, 13, 61):
# Supposed to modulo 256, but err, why bother?
O += 64
YENC[I] = '=%c' % O
else:
YENC[I] = '%c' % O

def yEncode1(data):
global YENC
yenc = YENC

encoded = []
datalen = len(data)
n = 0
while n < datalen:
chunk = data[n:n+256]
n += len(chunk)
encoded.extend([yenc[ord(c)] for c in chunk])
encoded.append('\n')

print len(''.join(encoded)),

def yEncode2(data):
global YENC
yenc = YENC

lines = []
datalen = len(data)
n = 0

bits = divmod(datalen, 256)
format = '256s' * bits[0]
parts = struct.unpack(format, data[:-bits[1]])
for part in parts:
lines.append(''.join([yenc[ord(c)] for c in part]))

lines.append(''.join([yenc[ord(c)] for c in data[-bits[1]:]]))
print len('\n'.join(lines) + '\n'),


yEncSetup()

teststr1 = 'a' * 400000
teststr2 = 'b' * 400000

for meth in (yEncode1, yEncode2):
timing(meth, 1, teststr1)
timing(meth, 1, teststr2)

--
Remove the oinks!
 
Reply With Quote
 
 
 
 
Oren Tirosh
Guest
Posts: n/a
 
      08-04-2003
On Tue, Aug 05, 2003 at 12:50:58AM +1000, Freddie wrote:
> Hi,
>
> I posted a while ago for some help with my word finder program, which is now
> quite a lot faster than I could manage. Thanks to all who helped
>
> This time, I've written a basic batch binary usenet poster in Python, but
> encoding the data into yEnc format is fairly slow. Is it possible to improve
> the routine any, WITHOUT using non-standard libraries? I don't want to have
> to rely on something strange


Python is pretty quick as long as you avoid loops that operate character
by character. Try to use functions that operate on longer strings.

Suggestions:

For the (x+42)%256 build a translation table and use str.translate.
To encode characters as escape sequences use str.replace or re.sub.

Oren

 
Reply With Quote
 
 
 
 
Freddie
Guest
Posts: n/a
 
      08-05-2003
Oren Tirosh <(E-Mail Removed)> wrote in
news:(E-Mail Removed):

> Suggestions:
>
> For the (x+42)%256 build a translation table and use str.translate.
> To encode characters as escape sequences use str.replace or re.sub.
>
> Oren


Aahh. I couldn't work out how to use translate() at 4am this morning, but I
worked it out now This version is a whoooole lot faster, and actually
meets the yEnc line splitting spec. Bonus!

$ python2.3 testyenc.py
yEncode1 407682 1.98
yEncode2 407707 0.18

I'm not sure how to use re.sub to escape the characters, I assume it would
also be 4 seperate replaces? Also, it needs a slightly more random input
string than 'a' * 400000, so here we go.


test = []
for i in xrange(256):
test.append(chr(i))
teststr = ''.join(test*1562)


def yEncode2(data):
trans = ''
for i in range(256):
trans += chr((i+42)%256)

translated = data.translate(trans)

# escape =, NUL, LF, CR
for i in (61, 0, 10, 13):
j = '=%c' % (i + 64)
translated = translated.replace(chr(i), j)


encoded = []
n = 0
for i in range(0, len(translated), 256):
chunk = translated[n+i:n+i+256]
if chunk[-1] == '=':
chunk += translated[n+i+256+1]
n += 1
encoded.append(chunk)
encoded.append('\n')

result = ''.join(encoded)

print len(result),
return result

--
-----------------------------------------------------------
Remove the oinks!
 
Reply With Quote
 
Freddie
Guest
Posts: n/a
 
      08-05-2003
Freddie <(E-Mail Removed)> wrote in
news:Xns93CE8D81747C5freddiethescaryeleph@218.100. 3.9:

Arr. There's an error here, the [n+i+256+1] shouldn't have a 1. I always get
that wrong The posted files actually decode now, and the yEncode()
overhead is a lot lower.

<snip>

> encoded = []
> n = 0
> for i in range(0, len(translated), 256):
> chunk = translated[n+i:n+i+256]
> if chunk[-1] == '=':
> chunk += translated[n+i+256] <<< this line
> n += 1
> encoded.append(chunk)
> encoded.append('\n')


--
Remove the oinks!
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
What is the point of having 16 bit colour if a computer monitor can only display 8 bit colour? How do you edit 16 bit colour when you can only see 8 bit? Scotius Digital Photography 6 07-13-2010 03:33 AM
Re: slow slow slow! Expert lino fitter Computer Support 5 12-12-2008 04:00 PM
Re: slow slow slow! Expert lino fitter Computer Support 0 12-10-2008 02:33 PM
64 bit - Windows Liberty 64bit, Windows Limited Edition 64 Bit, Microsoft SQL Server 2000 Developer Edition 64 Bit, IBM DB2 64 bit - new ! vvcd Computer Support 0 09-17-2004 08:15 PM
64 bit - Windows Liberty 64bit, Windows Limited Edition 64 Bit,Microsoft SQL Server 2000 Developer Edition 64 Bit, IBM DB2 64 bit - new! Ionizer Computer Support 1 01-01-2004 07:27 PM



Advertisments