Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > unicode mystery/problem

Reply
Thread Tools

unicode mystery/problem

 
 
Petr Jakes
Guest
Posts: n/a
 
      09-20-2006
Hi,
I am using Python 2.4.3 on Fedora Core4 and "Eric3" Python IDE
..
Below mentioned code works fine in the Eric3 environment. While trying
to start it from the command line, it returns:

Traceback (most recent call last):
File "pokus_1.py", line 5, in ?
print str(a)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xc1' in
position 6: ordinal not in range(12

========== 8< =============
#!/usr/bin python
# -*- Encoding: utf_8 -*-

a= u'DISKOV\xc1 POLE'
print a
print str(a)
========== 8< =============

Even it looks strange, I have to use str(a) syntax even I know the "a"
variable is a string.
I am trying to use ChartDirector for Python (charts for Python) and the
method "layer.addDataSet()" needs above mentioned syntax otherwise it
returns an Error.

layer.addDataSet(data, colour, str(dataName))

Thanks for your comments

Petr Jakes

 
Reply With Quote
 
 
 
 
John Machin
Guest
Posts: n/a
 
      09-20-2006
Petr Jakes wrote:
> Hi,
> I am using Python 2.4.3 on Fedora Core4 and "Eric3" Python IDE
> .
> Below mentioned code works fine in the Eric3 environment. While trying
> to start it from the command line, it returns:
>
> Traceback (most recent call last):
> File "pokus_1.py", line 5, in ?
> print str(a)
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xc1' in
> position 6: ordinal not in range(12


So print a works, but print str(a) crashes.

Instead, insert this:
import sys
print "default", sys.getdefaultencoding()
print "stdout", sys.stdout.encoding
and run your script at the command line. It should print:
default ascii
stdout x
here, and crash at the later use of str(a).
Step 2: run your script under Eric3. It will print:
default y
stdout z
and then should work properly. It is probable that x == y == z ==
'utf-8'
Step 3: see below.

>
> ========== 8< =============
> #!/usr/bin python
> # -*- Encoding: utf_8 -*-


There is no UTF8-encoded text in this short test script. Is the above
encoding comment merely a carry-over from your real script, or do you
believe it is necessary or useful in this test script?

>
> a= u'DISKOV\xc1 POLE'
> print a
> print str(a)
> ========== 8< =============
>
> Even it looks strange, I have to use str(a) syntax even I know the "a"
> variable is a string.


Some concepts you need to understand:
(a) "a" is not a string, it is a reference to a string.
(b) It is a reference to a unicode object (an implementation of a
conceptual Unicode string) ...
(c) which must be distinguished from a str object, which represents a
conceptual string of bytes.
(d) str(a) is trying to produce a str object from a unicode object. Not
being told what encoding to use, it uses the default encoding
(typically ascii) and naturally this will crash if there are non-ascii
characters in the unicode object.

> I am trying to use ChartDirector for Python (charts for Python) and the
> method "layer.addDataSet()" needs above mentioned syntax otherwise it
> returns an Error.


Care to tell us which error???

>
> layer.addDataSet(data, colour, str(dataName))


The method presumably expects a str object (8-bit string). What does
its documentation say? Again, what error message do you get if you feed
it a unicode object with non-ascii characters?

[Step 3] For foo in set(['x', 'y', 'z']):
Change str(dataName) to dataName.encode(foo). Change any debugging
display to use repr(a) instead of str(a). Test it with both Eric3 and
the command line.

[Aside: it's entirely possible that your problem will go away if you
remove the letter u from the line a= u'DISKOV\xc1 POLE' -- however if
you want to understand what is happening generally, I suggest you don't
do that]

HTH,
John

 
Reply With Quote
 
 
 
 
=?windows-1250?Q?Petr_Jake=9A?=
Guest
Posts: n/a
 
      09-22-2006
John, thanks for your extensive answer.
>> Hi,
>> I am using Python 2.4.3 on Fedora Core4 and "Eric3" Python IDE
>> .
>> Below mentioned code works fine in the Eric3 environment. While trying
>> to start it from the command line, it returns:
>>
>> Traceback (most recent call last):
>> File "pokus_1.py", line 5, in ?
>> print str(a)
>> UnicodeEncodeError: 'ascii' codec can't encode character u'\xc1' in
>> position 6: ordinal not in range(12


JM> So print a works, but print str(a) crashes.

JM> Instead, insert this:
JM> import sys
JM> print "default", sys.getdefaultencoding()
JM> print "stdout", sys.stdout.encoding
JM> and run your script at the command line. It should print:
JM> default ascii
JM> stdout x
**** in the command line it prints: *****
default ascii
stdout UTF-8
JM> here, and crash at the later use of str(a).
JM> Step 2: run your script under Eric3. It will print:
JM> default y
JM> stdout z

**** in the Eric3 it prints: ****
if the # -*- Eencoding: utf_8 -*- is set than:

default utf_8
stdout
unhandled AttributeError, "AsyncFile instance has no attribute
'encoding' "

if the encoding is not set than it prints:

DeprecationWarning: Non-ASCII character '\xc3' in file
/root/eric/analyza_dat_TPC/pokus_1.py on line 26, but no encoding
declared; see http://www.python.org/peps/pep-0263.html for details execfile(sys.argv[0], self.debugMod.__dict__)

default latin-1
stdout
unhandled AttributeError, "AsyncFile instance has no attribute
'encoding' "

JM> and then should work properly. It is probable that x == y == z ==
JM> 'utf-8'
JM> Step 3: see below.

>>
>> ========== 8< =============
>> #!/usr/bin python
>> # -*- Encoding: utf_8 -*-


JM> There is no UTF8-encoded text in this short test script. Is the above
JM> encoding comment merely a carry-over from your real script, or do you
JM> believe it is necessary or useful in this test script?
Generally, I am working with string like u'DISKOV\xc1 POLE' (I am
getting it from the database)

My intention to use >> # -*- Encoding: utf_8 -*- was to suppress
DeprecationWarnings if I use utf_8 in the code (like u'DISKOV\xc1 POLE')

>>
>> a= u'DISKOV\xc1 POLE'
>> print a
>> print str(a)
>> ========== 8< =============
>>
>> Even it looks strange, I have to use str(a) syntax even I know the "a"
>> variable is a string.


JM> Some concepts you need to understand:
JM> (a) "a" is not a string, it is a reference to a string.
JM> (b) It is a reference to a unicode object (an implementation of a
JM> conceptual Unicode string) ...
JM> (c) which must be distinguished from a str object, which represents a
JM> conceptual string of bytes.
JM> (d) str(a) is trying to produce a str object from a unicode object. Not
JM> being told what encoding to use, it uses the default encoding
JM> (typically ascii) and naturally this will crash if there are non-ascii
JM> characters in the unicode object.

>> I am trying to use ChartDirector for Python (charts for Python) and the
>> method "layer.addDataSet()" needs above mentioned syntax otherwise it
>> returns an Error.


JM> Care to tell us which error???
you can see the Error description and author comments here:
http://tinyurl.com/ezohe

>>
>> layer.addDataSet(data, colour, str(dataName))

I have try to experiment with the code a bit.
the simplest code where I can demonstrate my problems:
#!/usr/bin python
import sys
print "default", sys.getdefaultencoding()
print "stdout", sys.stdout.encoding

a=['P\xc5\x99\xc3\xad','Petr Jake\xc5\xa1']
b="my nice try %s" % ''.join(a).encode("utf-8")
print b

When I run it from the command line i am getting:
sys:1: DeprecationWarning: Non-ASCII character '\xc3' in file pokus_1.py on line 26, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details

default ascii
stdout UTF-8

Traceback (most recent call last):
File "pokus_1.py", line 8, in ?
b="my nice try %s" % ''.join(a).encode("utf-8")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 1: ordinal not in range(12



JM> The method presumably expects a str object (8-bit string). What does
JM> its documentation say? Again, what error message do you get if you feed
JM> it a unicode object with non-ascii characters?

JM> [Step 3] For foo in set(['x', 'y', 'z']):
JM> Change str(dataName) to dataName.encode(foo). Change any debugging
JM> display to use repr(a) instead of str(a). Test it with both Eric3 and
JM> the command line.

JM> [Aside: it's entirely possible that your problem will go away if you
JM> remove the letter u from the line a= u'DISKOV\xc1 POLE' -- however if
JM> you want to understand what is happening generally, I suggest you don't
JM> do that]

JM> HTH,
JM> John


 
Reply With Quote
 
Marc 'BlackJack' Rintsch
Guest
Posts: n/a
 
      09-22-2006
In <(E-Mail Removed)>, Petr JakeŇ°
wrote:

> I have try to experiment with the code a bit.
> the simplest code where I can demonstrate my problems:
> #!/usr/bin python
> import sys
> print "default", sys.getdefaultencoding()
> print "stdout", sys.stdout.encoding
>
> a=['P\xc5\x99\xc3\xad','Petr Jake\xc5\xa1']
> b="my nice try %s" % ''.join(a).encode("utf-8")


You have two byte strings in the list `a` and try to *encode* them as
utf-8. That does not work. You can make the example even a bit simpler::

'P\xc5\x99\xc3\xadPetr Jake\xc5\xa1'.encode('utf-8')

You cant't *encode* byte strings, just *decode* them. What happens is
that Python tries to make a unicode string from the byte string to encode
that in utf-8. But it decodes as ASCII as that is the default.

Don't mix byte strings and unicode strings. Put an encoding declaration
at the top of your file and convert everything to unicode on the "way in"
and to the proper encoding on the "way out" of your program.

Ciao,
Marc 'BlackJack' Rintsch
 
Reply With Quote
 
John Machin
Guest
Posts: n/a
 
      09-22-2006

Petr Jakeö wrote:
> John, thanks for your extensive answer.
> >> Hi,
> >> I am using Python 2.4.3 on Fedora Core4 and "Eric3" Python IDE
> >> .
> >> Below mentioned code works fine in the Eric3 environment. While trying
> >> to start it from the command line, it returns:
> >>
> >> Traceback (most recent call last):
> >> File "pokus_1.py", line 5, in ?
> >> print str(a)
> >> UnicodeEncodeError: 'ascii' codec can't encode character u'\xc1' in
> >> position 6: ordinal not in range(12

>
> JM> So print a works, but print str(a) crashes.
>
> JM> Instead, insert this:
> JM> import sys
> JM> print "default", sys.getdefaultencoding()
> JM> print "stdout", sys.stdout.encoding
> JM> and run your script at the command line. It should print:
> JM> default ascii
> JM> stdout x
> **** in the command line it prints: *****
> default ascii
> stdout UTF-8
> JM> here, and crash at the later use of str(a).
> JM> Step 2: run your script under Eric3. It will print:
> JM> default y
> JM> stdout z
>
> **** in the Eric3 it prints: ****
> if the # -*- Eencoding: utf_8 -*- is set than:
>
> default utf_8
> stdout
> unhandled AttributeError, "AsyncFile instance has no attribute
> 'encoding' "
>
> if the encoding is not set than it prints:
>
> DeprecationWarning: Non-ASCII character '\xc3' in file
> /root/eric/analyza_dat_TPC/pokus_1.py on line 26, but no encoding
> declared; see http://www.python.org/peps/pep-0263.html for details execfile(sys.argv[0], self.debugMod.__dict__)
>
> default latin-1
> stdout
> unhandled AttributeError, "AsyncFile instance has no attribute
> 'encoding' "
>
> JM> and then should work properly. It is probable that x == y == z ==
> JM> 'utf-8'
> JM> Step 3: see below.
>
> >>
> >> ========== 8< =============
> >> #!/usr/bin python
> >> # -*- Encoding: utf_8 -*-

>
> JM> There is no UTF8-encoded text in this short test script. Is the above
> JM> encoding comment merely a carry-over from your real script, or do you
> JM> believe it is necessary or useful in this test script?
> Generally, I am working with string like u'DISKOV\xc1 POLE' (I am
> getting it from the database)
>
> My intention to use >> # -*- Encoding: utf_8 -*- was to suppress
> DeprecationWarnings if I use utf_8 in the code (like u'DISKOV\xc1 POLE')
>
> >>
> >> a= u'DISKOV\xc1 POLE'
> >> print a
> >> print str(a)
> >> ========== 8< =============
> >>
> >> Even it looks strange, I have to use str(a) syntax even I know the "a"
> >> variable is a string.

>
> JM> Some concepts you need to understand:
> JM> (a) "a" is not a string, it is a reference to a string.
> JM> (b) It is a reference to a unicode object (an implementation of a
> JM> conceptual Unicode string) ...
> JM> (c) which must be distinguished from a str object, which represents a
> JM> conceptual string of bytes.
> JM> (d) str(a) is trying to produce a str object from a unicode object. Not
> JM> being told what encoding to use, it uses the default encoding
> JM> (typically ascii) and naturally this will crash if there are non-ascii
> JM> characters in the unicode object.
>
> >> I am trying to use ChartDirector for Python (charts for Python) and the
> >> method "layer.addDataSet()" needs above mentioned syntax otherwise it
> >> returns an Error.

>
> JM> Care to tell us which error???
> you can see the Error description and author comments here:
> http://tinyurl.com/ezohe


You have two different episodes on that website; adding the one we have
been discussing gives *three* different stories:

Episode 1:

The error description: "TypeError: Error converting argument 1 to type
PCc" -- you should ask him "What is type PCc???" If arg 1 is an
arbitrary str object, which byte values could it possibly be objecting
to?

The author comments: "The error code usually means the filename is not
a text string, ..." (1) Input file or output file? Is it possible that
one or more bytes are not allowable in a filename? (2) Is it possible
for you to give him the exact args that you are passing in (use print
repr(arg) before the call), and for him to tell you the *exact* reason,
not the "usual" reason?

Episode 2: Evidently arg is a str object, but passing in str(arg) and
just plain arg give different results??? I doubt it. print repr(arg)
and type(arg) and see what you've actually got there.

>
> >>
> >> layer.addDataSet(data, colour, str(dataName))

> I have try to experiment with the code a bit.
> the simplest code where I can demonstrate my problems:
> #!/usr/bin python
> import sys
> print "default", sys.getdefaultencoding()
> print "stdout", sys.stdout.encoding
>
> a=['P\xc5\x99\xc3\xad','Petr Jake\xc5\xa1']
> b="my nice try %s" % ''.join(a).encode("utf-8")


So ''.join(a) is a str object, encoded in utf-8 *already*.
Please try to understand:
(1) unicode_object.encode('utf-8') produces a str_object # in utf-8
encoding
(2) str_object.decode('utf-8') produces a unicode object # if
str_object contains valid utf-8.
(3) str_object.encode('anything') is a nonsense; it is the equivalent
of str_object.decode('ascii').encode('anything') and will typically
fail, as your next error message shows.

What were you trying to do?? I don't understand the relationship
between this little exercise and Episodes 1, 2, & 3.

Try to concentrate on what your data is (u"DISKOetcetc" is a unicode
string, but then you say that str(x) should be unnecessary because x is
already a str object!?) and what you need to have to get it passed
through to that package's methods.


> print b
>
> When I run it from the command line i am getting:
> sys:1: DeprecationWarning: Non-ASCII character '\xc3' in file pokus_1.py on line 26, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details
>
> default ascii
> stdout UTF-8
>
> Traceback (most recent call last):
> File "pokus_1.py", line 8, in ?
> b="my nice try %s" % ''.join(a).encode("utf-8")
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 1: ordinal not in range(12
>


As expected.

Regards,
John

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: os.lisdir, gets unicode, returns unicode... USUALLY?!?!? Jean-Paul Calderone Python 23 11-21-2006 10:25 AM
os.lisdir, gets unicode, returns unicode... USUALLY?!?!? gabor Python 13 11-18-2006 09:23 AM
Unicode digit to unicode string Gabriele *darkbard* Farina Python 2 05-16-2006 01:15 PM
unicode wrap unicode object? ygao Python 6 04-08-2006 09:54 AM
Unicode + jsp + mysql + tomcat = unicode still not displaying Robert Mark Bram Java 0 09-28-2003 05:37 AM



Advertisments