Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Unicode Question

Reply
Thread Tools

Unicode Question

 
 
David Pratt
Guest
Posts: n/a
 
      01-10-2006
Hi. I am working through some tutorials on unicode and am hoping that
someone can help explain this for me. I am on mac platform using python
2.4.1 at the moment. I am experimenting with unicode with the 3/4 symbol.

I want to prepare strings for db storage that come from normal Windows
machine (cp1252) so my understanding is to unicode and encode to utf-8
and to store properly. Since data will be used on the web I would not
have to change my encoding when extracting from the database. This first
example I believe simulates this with the 3/4 symbol. Here I want to
store '\xc2\xbe' in my database.

>>> tq = u'\xbe'
>>> tq_utf = tq.encode('utf8')
>>> tq, tq_utf

(u'\xbe', '\xc2\xbe')

To unicode withat a valiable, my understanding is that I can unicode and
encode at the same time

>>> tq = '\xbe'
>>> tq_utf = unicode(tq, 'utf-8')

Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeDecodeError: 'utf8' codec can't decode byte 0xbe in position 0:
unexpected code byte

This is not working for me. Can someone explain why. Many thanks.

Regards,
David
 
Reply With Quote
 
 
 
 
Erik Max Francis
Guest
Posts: n/a
 
      01-10-2006
David Pratt wrote:

> This is not working for me. Can someone explain why. Many thanks.


Because '\xbe' isn't UTF-8 for the character you want, '\xc2\xbe' is, as
you just showed yourself in the code snippet.

--
Erik Max Francis && http://www.velocityreviews.com/forums/(E-Mail Removed) && http://www.alcyone.com/max/
San Jose, CA, USA && 37 20 N 121 53 W && AIM erikmaxfrancis
Where are they?
-- Enrico Fermi, 1901-1954
 
Reply With Quote
 
 
 
 
=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=
Guest
Posts: n/a
 
      01-10-2006
David Pratt wrote:
> I want to prepare strings for db storage that come from normal Windows
> machine (cp1252) so my understanding is to unicode and encode to utf-8
> and to store properly.


That also depends on the database. The database must accept
UTF-8-encoded strings, and must not modify them in any form or way.
Some databases fail here, and work better if you pass Unicode objects
to them directly.

> Since data will be used on the web I would not
> have to change my encoding when extracting from the database. This first
> example I believe simulates this with the 3/4 symbol. Here I want tox
> store '\xc2\xbe' in my database.
>
>>>> tq = u'\xbe'


You can verify that this is really 3/4:

py> import unicodedata
py> unicodedata.name(u"\xbe")
'VULGAR FRACTION THREE QUARTERS'

>>>> tq_utf = tq.encode('utf8')
>>>> tq, tq_utf

> (u'\xbe', '\xc2\xbe')


So it should be clear now that '\xc2\xbe' is the UTF-8 encoding
of that character.

> To unicode withat a valiable, my understanding is that I can unicode and
> encode at the same time


Not sure what you mean by "same time" (I'm not even sure what
"I can unicode" means - unicode is not a verb, it's a noun).

>>>> tq = '\xbe'
>>>> tq_utf = unicode(tq, 'utf-8')

> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> UnicodeDecodeError: 'utf8' codec can't decode byte 0xbe in position 0:
> unexpected code byte
>
> This is not working for me. Can someone explain why. Many thanks.


Of course not. The UTF-8 encoding of the character, as we have seen
earlier, is '\xc2\xbe'. So you should write

py> unicode('\xc2\xbe', 'utf-8')
u'\xbe'

You mentioned windows-1252 at some point. If you are given windows-1252
bytes, you can do

py> unicode('\xbe', 'windows-1252')
u'\xbe'

If you are looking for "at the same time", perhaps this is also
interesting:

py> unicode('\xbe', 'windows-1252').encode('utf-8')
'\xc2\xbe'

Regards,
Martin
 
Reply With Quote
 
David Pratt
Guest
Posts: n/a
 
      01-10-2006
Hi Martin. Many thanks for your reply. What I am reall after, the
following accomplishes.
>
> If you are looking for "at the same time", perhaps this is also
> interesting:
>
> py> unicode('\xbe', 'windows-1252').encode('utf-8')
> '\xc2\xbe'
>


Your answer really helped quite a bit to clarify this for me. I am using
sqlite3 so it is very happy to have utf-8 encoded unicode.

The examples you provided were the additional help I needed. Thank you.

Regards,
David
 
Reply With Quote
 
David Pratt
Guest
Posts: n/a
 
      01-10-2006
Hi Erik. Thank you for your reply. The advice I has helped clarify this
for me.

Regards,
David

Erik Max Francis wrote:
> David Pratt wrote:
>
>
>>This is not working for me. Can someone explain why. Many thanks.

>
>
> Because '\xbe' isn't UTF-8 for the character you want, '\xc2\xbe' is, as
> you just showed yourself in the code snippet.
>

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: os.lisdir, gets unicode, returns unicode... USUALLY?!?!? Jean-Paul Calderone Python 23 11-21-2006 10:25 AM
os.lisdir, gets unicode, returns unicode... USUALLY?!?!? gabor Python 13 11-18-2006 09:23 AM
Unicode digit to unicode string Gabriele *darkbard* Farina Python 2 05-16-2006 01:15 PM
unicode wrap unicode object? ygao Python 6 04-08-2006 09:54 AM
Unicode + jsp + mysql + tomcat = unicode still not displaying Robert Mark Bram Java 0 09-28-2003 05:37 AM



Advertisments