Velocity Reviews

Velocity Reviews (http://www.velocityreviews.com/forums/index.php)
-   Python (http://www.velocityreviews.com/forums/f43-python.html)
-   -   Python 2.6 StreamReader.readline() (http://www.velocityreviews.com/forums/t948490-python-2-6-streamreader-readline.html)

cpppwner@gmail.com 07-24-2012 03:01 PM

Python 2.6 StreamReader.readline()
 
Hi,

I have a simple question, I'm using something like the following lines in python 2.6.2

reader = codecs.getreader(encoding)
lines = []
with open(filename, 'rb') as f:
lines = reader(f, 'strict').readlines(keepends=False)

where encoding == 'utf-16-be'
Everything works fine, except that lines[0] is equal to codecs.BOM_UTF16_BE
Is this behaviour correct, that the BOM is still present?

Thanks in advance for your help.

Best,
Stefan

Ulrich Eckhardt 07-25-2012 06:09 AM

Re: Python 2.6 StreamReader.readline()
 
Am 24.07.2012 17:01, schrieb cpppwner@gmail.com:
> reader = codecs.getreader(encoding)
> lines = []
> with open(filename, 'rb') as f:
> lines = reader(f, 'strict').readlines(keepends=False)
>
> where encoding == 'utf-16-be'
> Everything works fine, except that lines[0] is equal to codecs.BOM_UTF16_BE
> Is this behaviour correct, that the BOM is still present?


Yes, assuming the first line only contains that BOM. Technically it's a
space character, and why should those be removed?

Uli

Walter Dörwald 07-25-2012 09:02 AM

Re: Python 2.6 StreamReader.readline()
 
On 25.07.12 08:09, Ulrich Eckhardt wrote:

> Am 24.07.2012 17:01, schrieb cpppwner@gmail.com:
>> reader = codecs.getreader(encoding)
>> lines = []
>> with open(filename, 'rb') as f:
>> lines = reader(f, 'strict').readlines(keepends=False)
>>
>> where encoding == 'utf-16-be'
>> Everything works fine, except that lines[0] is equal to
>> codecs.BOM_UTF16_BE
>> Is this behaviour correct, that the BOM is still present?

>
> Yes, assuming the first line only contains that BOM. Technically it's a
> space character, and why should those be removed?


If the first "character" in the file is a BOM the file encoding is
probably not utf-16-be but utf-16.

Servus,
Walter


wxjmfauth@gmail.com 07-25-2012 10:26 AM

Re: Python 2.6 StreamReader.readline()
 
On Wednesday, July 25, 2012 11:02:01 AM UTC+2, Walter Dörwald wrote:
> On 25.07.12 08:09, Ulrich Eckhardt wrote:
>
> > Am 24.07.2012 17:01, schrieb cpppwner@gmail.com:
> >> reader = codecs.getreader(encoding)
> >> lines = []
> >> with open(filename, 'rb') as f:
> >> lines = reader(f, 'strict').readlines(keepends=False)
> >>
> >> where encoding == 'utf-16-be'
> >> Everything works fine, except that lines[0] is equal to
> >> codecs.BOM_UTF16_BE
> >> Is this behaviour correct, that the BOM is still present?
> >
> > Yes, assuming the first line only contains that BOM. Technically it's a
> > space character, and why should those be removed?
>
> If the first "character" in the file is a BOM the file encodingis
> probably not utf-16-be but utf-16.
>
> Servus,
> Walter


The byte order mark, if present, is nothing else than
an encoded

>>> ud.name('\ufeff')

'ZERO WIDTH NO-BREAK SPACE'

*code point*.

Five "BOM" are possible (Unicode consortium). utf-8-sig, utf-16-be,
utf-16-le, utf-32-be, utf-32-le. The codecs module provide many
aliases.

The fact that utf-16/32 does correspond to -le or to -be may
vary according to the platforms, the compilers, ...

>>> sys.version

'3.2.3 (default, Apr 11 2012, 07:15:24) [MSC v.1500 32 bit
(Intel)]'
>>> codecs.BOM_UTF16_BE

b'\xfe\xff'
>>> codecs.BOM_UTF16_LE

b'\xff\xfe'
>>> codecs.BOM_UTF16

b'\xff\xfe'
>>>


---

As far as I know, Py 2.7 or Py 3.2 never return a "BOM" when
a file is read correctly.

>>> with open('a-utf-16-be.txt', 'r', encoding='utf-16-be') as f:

.... r = f.readlines()
.... for zeile in r:
.... print(zeile.rstrip())
....
abc
élève
cœur
€uro
>>>



jmf


wxjmfauth@gmail.com 07-25-2012 10:26 AM

Re: Python 2.6 StreamReader.readline()
 
On Wednesday, July 25, 2012 11:02:01 AM UTC+2, Walter Dörwald wrote:
> On 25.07.12 08:09, Ulrich Eckhardt wrote:
>
> > Am 24.07.2012 17:01, schrieb cpppwner@gmail.com:
> >> reader = codecs.getreader(encoding)
> >> lines = []
> >> with open(filename, 'rb') as f:
> >> lines = reader(f, 'strict').readlines(keepends=False)
> >>
> >> where encoding == 'utf-16-be'
> >> Everything works fine, except that lines[0] is equal to
> >> codecs.BOM_UTF16_BE
> >> Is this behaviour correct, that the BOM is still present?
> >
> > Yes, assuming the first line only contains that BOM. Technically it's a
> > space character, and why should those be removed?
>
> If the first "character" in the file is a BOM the file encodingis
> probably not utf-16-be but utf-16.
>
> Servus,
> Walter


The byte order mark, if present, is nothing else than
an encoded

>>> ud.name('\ufeff')

'ZERO WIDTH NO-BREAK SPACE'

*code point*.

Five "BOM" are possible (Unicode consortium). utf-8-sig, utf-16-be,
utf-16-le, utf-32-be, utf-32-le. The codecs module provide many
aliases.

The fact that utf-16/32 does correspond to -le or to -be may
vary according to the platforms, the compilers, ...

>>> sys.version

'3.2.3 (default, Apr 11 2012, 07:15:24) [MSC v.1500 32 bit
(Intel)]'
>>> codecs.BOM_UTF16_BE

b'\xfe\xff'
>>> codecs.BOM_UTF16_LE

b'\xff\xfe'
>>> codecs.BOM_UTF16

b'\xff\xfe'
>>>


---

As far as I know, Py 2.7 or Py 3.2 never return a "BOM" when
a file is read correctly.

>>> with open('a-utf-16-be.txt', 'r', encoding='utf-16-be') as f:

.... r = f.readlines()
.... for zeile in r:
.... print(zeile.rstrip())
....
abc
élève
cœur
€uro
>>>



jmf



All times are GMT. The time now is 08:06 PM.

Powered by vBulletin®. Copyright ©2000 - 2014, vBulletin Solutions, Inc.
SEO by vBSEO ©2010, Crawlability, Inc.