«

I've got a UTF-8 encoded text file from Linux with standard newlines
("\n").

I'm reading this file on Win32 with Python 2.6:

codecs.open("whatever.txt","r","utf-8").read()

Inexplicably, all the newlines ("\n") are replaced with CR+LF ("\r
\n") ... Why?

As a workaround I'm having to do this:

open("whatever.txt","r").read().decode("utf-8")

which appropriately does not alter my newlines.

What really gets me confused though is the Python docs for
codecs.open:

"Files are always opened in binary mode, even if no binary mode was
specified. This is done to avoid data loss due to encodings using 8-
bit values. This means that no automatic conversion of '\n' is done on
reading and writing."

The way I read that, codecs.open should not touch my newlines. What am
I doing wrong? Is this a bug in Python, or in the docs, or both?

On Aug 26, 2009, at 10:52 PM, Ryan McGuire wrote:

> I've got a UTF-8 encoded text file from Linux with standard newlines
> ("\n").
>
> I'm reading this file on Win32 with Python 2.6:
>
> codecs.open("whatever.txt","r","utf-8").read()
>
> Inexplicably, all the newlines ("\n") are replaced with CR+LF ("\r
> \n") ... Why?

Try using "rb" instead of "r" for the mode in the call to open().

HTH
Philip

On Aug 26, 11:04*pm, Philip Semanchuk <phi...@semanchuk.com> wrote:
> Try using "rb" instead of "r" for the mode in the call to open().
>
> HTH
> Philip

That does indeed fix the problem, thanks! Still seems like the docs
are wrong though.

On Wed, Aug 26, 2009 at 8:40 PM, Ryan McGuire<> wrote:
> On Aug 26, 11:04Â*pm, Philip Semanchuk <phi...@semanchuk.com> wrote:
>> Try using "rb" instead of "r" for the mode in the call to open().
>>
>> HTH
>> Philip
>
> That does indeed fix the problem, thanks! Still seems like the docs
> are wrong though.

Yeah, the need to specify "b" does seem rather incongruous:

codecs.open(filename, mode[, encoding[, errors[, buffering]]])
[...]
Note: Files are always opened in binary mode, even if no binary
mode was specified. This is done to avoid data loss due to encodings
using 8-bit values. This means that no automatic conversion of b'\n'
is done on reading and writing.

File a bug perhaps?: http://bugs.python.org/

Cheers,
Chris
--
http://blog.rebertia.com

On Wed, Aug 26, 2009 at 11:06 PM, Chris Rebert<> wrote:
> On Wed, Aug 26, 2009 at 8:40 PM, Ryan McGuire<> wrote:
>> On Aug 26, 11:04Â*pm, Philip Semanchuk <phi...@semanchuk.com> wrote:
>>> Try using "rb" instead of "r" for the mode in the call to open().
>>>
>>> HTH
>>> Philip
>>
>> That does indeed fix the problem, thanks! Still seems like the docs
>> are wrong though.
>
> Yeah, the need to specify "b" does seem rather incongruous:
>
> codecs.open(filename, mode[, encoding[, errors[, buffering]]])
> Â* Â*[...]
> Â* Â*Note: Files are always opened in binary mode, even if no binary
> mode was specified. This is done to avoid data loss due to encodings
> using 8-bit values. This means that no automatic conversion of b'\n'
> is done on reading and writing.
>
> File a bug perhaps?: http://bugs.python.org/

Ah, I see you already did: http://bugs.python.org/issue6788

- Chris