Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > pep 277, Unicode filenames & mbcs encoding &c.

Reply
Thread Tools

pep 277, Unicode filenames & mbcs encoding &c.

 
 
Edward K. Ream
Guest
Posts: n/a
 
      10-21-2003
Am I reading pep 277 correctly? On Windows NT/XP, should filenames always
be converted to Unicode using the mbcs encoding? For example,

myFile = unicode(__file__, "mbcs", "strict")

This seems to work, and I'm wondering whether there are any other details to
consider.

My experiments with Idle for Python 2.2 indicate that os.path.join doesn't
work as I expect when one of the args is a Unicode string. Everything
before the Unicode string gets thrown away. But this is probably moot: pep
277 implies Python 2.3...

Am I correct that conversions to Unicode (using "mbcs" on Windows) should be
done before passing arguments to os.path.join, os.path.split,
os.path.normpath, etc. ? Presumably os.path functions use the default
system encoding to convert strings to Unicode, which isn't likely to be
"mbcs" or anything else useful

Are there any situations where some other encoding should be used instead on
Windows? What about other platforms? For instance, does Linux allow
non-ascii file names? If so, what encoding should be specified when
converting to Unicode? Thanks.

Edward
--------------------------------------------------------------------
Edward K. Ream email: http://www.velocityreviews.com/forums/(E-Mail Removed)
Leo: Literate Editor with Outlines
Leo: http://webpages.charter.net/edreamleo/front.html
--------------------------------------------------------------------


 
Reply With Quote
 
 
 
 
vincent wehren
Guest
Posts: n/a
 
      10-21-2003
"Edward K. Ream" <(E-Mail Removed)> schrieb im Newsbeitrag
news:(E-Mail Removed)...
| Am I reading pep 277 correctly? On Windows NT/XP, should filenames always
| be converted to Unicode using the mbcs encoding? For example,
|
| myFile = unicode(__file__, "mbcs", "strict")

No and no. You can *still* use regular byte strings. Python will do the
conversion to Unicode for you using "mbcs" as encoding.

|
| This seems to work, and I'm wondering whether there are any other details
to
| consider.
|
| My experiments with Idle for Python 2.2 indicate that os.path.join doesn't
| work as I expect when one of the args is a Unicode string. Everything
| before the Unicode string gets thrown away. But this is probably moot:
pep
| 277 implies Python 2.3...

Exactly. Python Unicode file name support has arrived with 2.3.

|
....
|
| Are there any situations where some other encoding should be used instead
on
| Windows? What about other platforms? For instance, does Linux allow
| non-ascii file names?

You can use "os.path.supports_unicode_filenames" to check...


HTH

Vincent Wehren

If so, what encoding should be specified when
| converting to Unicode? Thanks.
Propably the default encoding, on Linux

|
| Edward
| --------------------------------------------------------------------
| Edward K. Ream email: (E-Mail Removed)
| Leo: Literate Editor with Outlines
| Leo: http://webpages.charter.net/edreamleo/front.html
| --------------------------------------------------------------------
|
|


 
Reply With Quote
 
 
 
 
Just
Guest
Posts: n/a
 
      10-21-2003
In article <bn3jfo$df$(E-Mail Removed)1.nb.home.nl>,
"vincent wehren" <(E-Mail Removed)> wrote:

> | Are there any situations where some other encoding should be used instead
> on
> | Windows? What about other platforms? For instance, does Linux allow
> | non-ascii file names?
>
> You can use "os.path.supports_unicode_filenames" to check...


Actually, you can't, see:

http://python.org/sf/767645

The only two platforms that currently support unicode filenames properly
are Windows NT/XP and MacOSX, and for one of them
os.path.supports_unicode_filenames returns False

Just
 
Reply With Quote
 
Martin v. =?iso-8859-15?q?L=F6wis?=
Guest
Posts: n/a
 
      10-21-2003
"Edward K. Ream" <(E-Mail Removed)> writes:

> Am I reading pep 277 correctly? On Windows NT/XP, should filenames always
> be converted to Unicode using the mbcs encoding?


What do you mean with "should"? "Should Python always..." or "Should
the application always"?

PEP 277 actually answers neither question. As Vincent explains,
nothing changes with respect to using byte strings on the API. The
changes only affect Unicode strings passed to functions expecting file names.

> For example,
>
> myFile = unicode(__file__, "mbcs", "strict")
>
> This seems to work


And it has nothing to do with PEP 277: You are not passing myFile to
any API function.

If you mean to use myFile as a file name, then yes: this is intended
to work. However, using plain __file__ directly should also work.

> Am I correct that conversions to Unicode (using "mbcs" on Windows) should be
> done before passing arguments to os.path.join, os.path.split,
> os.path.normpath, etc. ?


You should either use only Unicode strings, or only byte strings. The
functions of os.path are not all affected by the PEP 277
implementation (although they probably should).

> Presumably os.path functions use the default
> system encoding to convert strings to Unicode, which isn't likely to be
> "mbcs" or anything else useful


Right. This is actually unfortunate.

> Are there any situations where some other encoding should be used instead on
> Windows?


If you get data from a cmd.exe Window.

> What about other platforms? For instance, does Linux allow non-ascii
> file names?


Yes, it does.

> If so, what encoding should be specified when converting to Unicode?


Nobody knows, but the convention is to use the locale's encoding, as
returned by locale.getpreferredencoding().

Regards,
Martin
 
Reply With Quote
 
Edward K. Ream
Guest
Posts: n/a
 
      10-22-2003
Many thanks, Martin, for these comments. They are so helpful...

> You should either use only Unicode strings, or only byte strings. The
> functions of os.path are not all affected by the PEP 277
> implementation (although they probably should).


My working assumption is that all strings in my app must be Unicode strings.
For example, the crashes happening right now trying to support Unicode
filenames occur when a string is converted to Unicode in situations like:

if fileName1 == fileName2:

where one fileName is a unicode string and the other isn't yet. That's why
I wanted to do:

myFile = unicode(__file__, "mbcs", "strict")

The challenge in my app is to make sure the proper encoding is used in the
more than 30 situations where a filename gets created somehow. Naturally,
that's not your problem, nor PEP 277's problem either

> > If so, what encoding should be specified when converting to Unicode?

>
> Nobody knows, but the convention is to use the locale's encoding, as
> returned by locale.getpreferredencoding().


Thanks for this.

Edward
--------------------------------------------------------------------
Edward K. Ream email: (E-Mail Removed)
Leo: Literate Editor with Outlines
Leo: http://webpages.charter.net/edreamleo/front.html
--------------------------------------------------------------------


 
Reply With Quote
 
Martin v. =?iso-8859-15?q?L=F6wis?=
Guest
Posts: n/a
 
      10-23-2003
"Edward K. Ream" <(E-Mail Removed)> writes:

> if fileName1 == fileName2:
>
> where one fileName is a unicode string and the other isn't yet. That's why
> I wanted to do:
>
> myFile = unicode(__file__, "mbcs", "strict")


Ah, I see. Instead of "mbcs", you should use
sys.getfilesystemencoding(). This is what Python will use when
converting the Unicode strings back to byte strings before passing
them to the system (in case it converts back at all, which it doesn't
on Windows thanks to PEP 277).

Regards,
Martin
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: 2to3 on Mac - unknown encoding: mbcs Gabriel Genellina Python 0 11-06-2009 06:42 AM
2to3 on Mac - unknown encoding: mbcs Skip Montanaro Python 0 11-06-2009 01:50 AM
strftime return value encoding (mbcs, locale, etc.) Giovanni Bajo Python 2 01-27-2008 10:25 PM
ldap_get_values: converting UTF8 encoding to ANSI MBCS string on UNIX systems Tejas C++ 1 11-14-2007 06:19 PM
problem with filenames, Filenames and FILENAMES B.J. HTML 4 04-23-2005 08:13 PM



Advertisments