«

"Colin S. Miller" <colinsm.spam-me-> writes:

> Where have I gone wrong, and what is the correct method
> to verify the BOM mark?

readline is not supported in the UTF-16 codec. You have to read the
entire file, and perform .split. Looking at the BOM should not be
necessary, as the UTF-16 codec will do so on its own.

Regards,
Martin

Martin v. Löwis wrote:
> "Colin S. Miller" <colinsm.spam-me-> writes:
>
>
>>Where have I gone wrong, and what is the correct method
>>to verify the BOM mark?
>
>
> readline is not supported in the UTF-16 codec. You have to read the
> entire file, and perform .split. Looking at the BOM should not be
> necessary, as the UTF-16 codec will do so on its own.
Is there any reason why readline() isn't supported?
AFAIK,
the prefered UNICODE standard line endings are
0x2028 (Line seperator)
0x2029 (Paragraph seperator)
but 0x10 (Line feed) and 0x13 (carriage return) are
also supported for legacy support.


I'm using
file.read().splitlines() now, but am slightly worried
about perfomance/memory when there a few hundered lines.

TIA,
Colin S. Miller


>
> Regards,
> Martin
>

"Colin S. Miller" <colinsm.spam-me-> writes:

> Is there any reason why readline() isn't supported?

Because it hasn't been implemented. The naive approach of calling the
readline of the underlying stream (as all other codecs do) does not
work for UTF-16.

> AFAIK, the prefered UNICODE standard line endings are 0x2028 (Line
> seperator) 0x2029 (Paragraph seperator) but 0x10 (Line feed) and
> 0x13 (carriage return) are also supported for legacy support.

Add that on top of that. One should support all line breaking
characters for UTF-16, atleast in Universal Newline (U) mode.

> I'm using file.read().splitlines() now, but am slightly worried
> about perfomance/memory when there a few hundered lines.

Feel free to implement and contribute a patch. It has been that way
for some years now, and it likely will stay the same for the coming
years unless somebody contributes a patch.

Regards,
Martin