Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Odd csv column-name truncation with only one column

Reply
Thread Tools

Odd csv column-name truncation with only one column

 
 
Tim Chase
Guest
Posts: n/a
 
      07-19-2012
tim@laptop:~/tmp$ python
Python 2.6.6 (r266:84292, Dec 26 2010, 22:31:4
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import csv
>>> from cStringIO import StringIO
>>> s = StringIO('Email\(E-Mail Removed)\(E-Mail Removed)\ n')
>>> s.seek(0)
>>> d = csv.Sniffer().sniff(s.read())
>>> s.seek(0)
>>> r = csv.DictReader(s, dialect=d)
>>> r.fieldnames

['Emai', '']

I get the same results using Python 3.1.3 (also readily available on
Debian Stable), as well as working directly on a file rather than a
StringIO.

Any reason I'm getting ['Emai', ''] (note the missing ell) instead
of ['Email'] as my resulting fieldnames? Did I miss something in
the docs?

-tkc






 
Reply With Quote
 
 
 
 
Steven D'Aprano
Guest
Posts: n/a
 
      07-19-2012
On Thu, 19 Jul 2012 06:21:58 -0500, Tim Chase wrote:

> tim@laptop:~/tmp$ python
> Python 2.6.6 (r266:84292, Dec 26 2010, 22:31:4 [GCC 4.4.5] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import csv
>>>> from cStringIO import StringIO
>>>> s = StringIO('Email\(E-Mail Removed)\(E-Mail Removed)\ n') s.seek(0)
>>>> d = csv.Sniffer().sniff(s.read())
>>>> s.seek(0)
>>>> r = csv.DictReader(s, dialect=d)
>>>> r.fieldnames

> ['Emai', '']



I get the same results for Python 2.6 and 2.7. Curiously, 2.5 returns
fieldnames as None.

I'm not entirely sure that a single column is legitimate for CSV -- if
there's only one column, it is hardly comma-separated, or any other
separated for that matter. But perhaps the csv module should raise an
exception in that case.

I think you've found a weird corner case where the sniffer goes nuts. You
should probably report it as a bug:

py> s = StringIO('Email\(E-Mail Removed)\(E-Mail Removed)\ n')
py> s.seek(0)
py> d = csv.Sniffer().sniff(s.read())
py> d.delimiter
'l'

py> s = StringIO('Spam\(E-Mail Removed)\(E-Mail Removed)\n ')
py> s.seek(0)
py> d = csv.Sniffer().sniff(s.read())
py> d.delimiter
'p'

py> s = StringIO('Spam\nham\ncheese\n')
py> s.seek(0)
py> d = csv.Sniffer().sniff(s.read())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/csv.py", line 184, in sniff
raise Error, "Could not determine delimiter"
_csv.Error: Could not determine delimiter


--
Steven
 
Reply With Quote
 
 
 
 
Hans Mulder
Guest
Posts: n/a
 
      07-19-2012
On 19/07/12 13:21:58, Tim Chase wrote:
> tim@laptop:~/tmp$ python
> Python 2.6.6 (r266:84292, Dec 26 2010, 22:31:4
> [GCC 4.4.5] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import csv
>>>> from cStringIO import StringIO
>>>> s = StringIO('Email\(E-Mail Removed)\(E-Mail Removed)\ n')
>>>> s.seek(0)
>>>> d = csv.Sniffer().sniff(s.read())
>>>> s.seek(0)
>>>> r = csv.DictReader(s, dialect=d)
>>>> r.fieldnames

> ['Emai', '']
>
> I get the same results using Python 3.1.3 (also readily available on
> Debian Stable), as well as working directly on a file rather than a
> StringIO.
>
> Any reason I'm getting ['Emai', ''] (note the missing ell) instead
> of ['Email'] as my resulting fieldnames? Did I miss something in
> the docs?


The sniffer tries to guess the column separator. If none of the
usual suspects seems to work, it tries to find a character that
occurs with the same frequency in every row. In your sample,
the letter 'l' occurs exactly once on each line, so it is the
most plausible separator, or so the Sniffer thinks.

Perhaps it should be documented that the Sniffer doesn't work
on single-column data.

If you really need to read a one-column csv file, you'll have
to find some other way to produce a Dialect object. Perhaps the
predefined 'cvs.excel' dialect matches your data. If not, the
easiest way might be to manually define a csv.Dialect subclass.

Hope this helps,

-- HansM

 
Reply With Quote
 
Tim Chase
Guest
Posts: n/a
 
      07-19-2012
On 07/19/12 08:52, Hans Mulder wrote:
> Perhaps it should be documented that the Sniffer doesn't work
> on single-column data.


I think this would involve the least change in existing code, and
go a long way towards removing my surprise.

> If you really need to read a one-column csv file, you'll have
> to find some other way to produce a Dialect object. Perhaps the
> predefined 'cvs.excel' dialect matches your data. If not, the
> easiest way might be to manually define a csv.Dialect subclass.


The problem I'm trying to solve is "here's a filename that might be
comma/pipe/tab delimited, it has an 'email' column at minimum, and
perhaps a couple others of interest if they were included" It's
improbable that it's ONLY an email column, but my tests happened to
snag this edge case. I can likely do my own sniffing by reading the
first line, checking for tabs then pipes then commas (perhaps
biasing the order based on the file-extension of .csv vs. .txt), and
then building my own dialect information to pass to csv.DictReader
It just seems unfortunate that the sniffer would ever consider
[a-zA-Z0-9] as a valid delimiter.

-tkc


 
Reply With Quote
 
Dennis Lee Bieber
Guest
Posts: n/a
 
      07-19-2012
On Thu, 19 Jul 2012 13:01:37 -0500, Tim Chase
<(E-Mail Removed)> declaimed the following in
gmane.comp.python.general:

> It just seems unfortunate that the sniffer would ever consider
> [a-zA-Z0-9] as a valid delimiter.
>

I'd suspect the sniffer logic does not do any special casing -- any
/byte value/ is a candidate for the delimiter. This would allow for
usage of some old ASCII control characters -- things like x1F (unit
separator)

{Next is to rig the sniffer to identify x1F for fields, and x1E for
records <G>}
--
Wulfraed Dennis Lee Bieber AF6VN
http://www.velocityreviews.com/forums/(E-Mail Removed) HTTP://wlfraed.home.netcom.com/

 
Reply With Quote
 
Steven D'Aprano
Guest
Posts: n/a
 
      07-20-2012
On Thu, 19 Jul 2012 15:52:12 +0200, Hans Mulder wrote:

> Perhaps it should be documented that the Sniffer doesn't work on
> single-column data.
>
> If you really need to read a one-column csv file, you'll have to find
> some other way to produce a Dialect object. Perhaps the predefined
> 'cvs.excel' dialect matches your data. If not, the easiest way might be
> to manually define a csv.Dialect subclass.


Perhaps the csv module could do with a pre-defined "one column" dialect.
If anyone comes up with one, do consider proposing it as a patch on the
bug tracker.


--
Steven
 
Reply With Quote
 
Hans Mulder
Guest
Posts: n/a
 
      07-20-2012
On 19/07/12 23:10:04, Dennis Lee Bieber wrote:
> On Thu, 19 Jul 2012 13:01:37 -0500, Tim Chase
> <(E-Mail Removed)> declaimed the following in
> gmane.comp.python.general:
>
>> It just seems unfortunate that the sniffer would ever consider
>> [a-zA-Z0-9] as a valid delimiter.


+1

> I'd suspect the sniffer logic does not do any special casing
> -- any /byte value/ is a candidate for the delimiter.


The sniffer prefers [',', '\t', ';', ' ', ':'] (in that order).
If none of those is found, it goes to the other extreme and considers
all characters equally likely.

> This would allow for usage of some old ASCII control characters --
> things like x1F (unit separator)


If the Sniffer excludes [a-zA-Z0-9] (or all alphanumerics) as
potential delimiters, than control characters such as "\x1F" are
still possible.

> {Next is to rig the sniffer to identify x1F for fields, and x1E
> for records <G>}


The sniffer will always guess '\r\n' as the line terminator.

That should not stop you from creating a dialect with '\x1E' as
the line terminator. Just don't expect the sniffer to recognize
that dialect.

-- HansM


 
Reply With Quote
 
Dennis Lee Bieber
Guest
Posts: n/a
 
      07-20-2012
On Fri, 20 Jul 2012 18:59:24 +0200, Hans Mulder <(E-Mail Removed)>
declaimed the following in gmane.comp.python.general:


> The sniffer will always guess '\r\n' as the line terminator.
>
> That should not stop you from creating a dialect with '\x1E' as
> the line terminator. Just don't expect the sniffer to recognize
> that dialect.
>

{devil's advocate}: Maybe it's time to expand the CSV module... Of
course, if we set it to recognize x1E as a record separator, we should
be fair and also incorporate the other two ASCII "separator" codes.

x1D (group separator) could be used to signal a "new table" -- ie; a
change in record structure (number of columns, header labels). And then
x1C (file separator) could represent a new "worksheet" (in Excel terms).

We'd need some sort of flag/query method to detect these changes, of
course.

while not csv.EndOfSheet():
while not csv.EndOfTable():
...

And then there is the potential of using <VT> and <FF> as
equivalents for x1D and x1C (for those files using <TAB> and <CR><LF> as
field/record separators).

--
Wulfraed Dennis Lee Bieber AF6VN
(E-Mail Removed) HTTP://wlfraed.home.netcom.com/

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: Odd csv column-name truncation with only one column Tim Chase Python 0 07-19-2012 01:04 PM
Re: Odd csv column-name truncation with only one column Peter Otten Python 0 07-19-2012 11:49 AM
MySql Data Truncation Marcelo Java 3 12-21-2005 01:29 AM
Text Truncation manas ASP .Net 1 07-01-2005 10:41 PM
Truncation of long filename plxjb Computer Information 0 09-28-2004 06:40 AM



Advertisments