Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Re: Eliminate "extra" variable

Reply
Thread Tools

Re: Eliminate "extra" variable

 
 
Mark Lawrence
Guest
Posts: n/a
 
      12-08-2013
On 08/12/2013 18:58, Tim Chase wrote:
> On 2013-12-07 23:14, Igor Korot wrote:


[big snip]

>


Whenever I need date manipulations I always reach out to this
http://labix.org/python-dateutil

--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

 
Reply With Quote
 
 
 
 
Roy Smith
Guest
Posts: n/a
 
      12-08-2013
In article <(E-Mail Removed)>,
Mark Lawrence <(E-Mail Removed)> wrote:

> On 08/12/2013 18:58, Tim Chase wrote:
> > On 2013-12-07 23:14, Igor Korot wrote:

>
> [big snip]
>
> >

>
> Whenever I need date manipulations I always reach out to this
> http://labix.org/python-dateutil


The problem with dateutil is it's dog slow. Sure, I use it too, when
convenience is more important than performance, but have you ever looked
at the code for dateutil.parser.parse()? It's worth reading, just for
fun.
 
Reply With Quote
 
 
 
 
Igor Korot
Guest
Posts: n/a
 
      12-08-2013
---------- Forwarded message ----------
From: Igor Korot <(E-Mail Removed)>
Date: Sun, Dec 8, 2013 at 12:57 PM
Subject: Re: Eliminate "extra" variable
To: Roy Smith <(E-Mail Removed)>


Hi, guys,
Thank you for all those valuable suggestions.
2Tim Chase:
I guess you missed this: "My originalData comes from the database
query" and so the checking of the data quality is a DB burden.
As to the function: the function purpose is to process the data and
give out the list of dates and the dictionary of date:frequency.
It's input is the query result, so there is no looping when the
function is called. It is called only once.

Also, the data comes from either SQLite or mySQL and so to eliminate
the difference between those engines dates are processed as strings
and converted to dates for the calculation purposes only.
Maybe I will need to refactor SQLite processing to get the dates as
dates and not a string, but that's probably for the future. so that
dates will be kept as the datetime type until the end of the function.
As I wrote the dates will be used as the text for the plotting window
axis labels and as the labels they should come out as strings, hence
the conversion.

Thank you.


On Sun, Dec 8, 2013 at 12:07 PM, Roy Smith <(E-Mail Removed)> wrote:
> In article <(E-Mail Removed)>,
> Mark Lawrence <(E-Mail Removed)> wrote:
>
>> On 08/12/2013 18:58, Tim Chase wrote:
>> > On 2013-12-07 23:14, Igor Korot wrote:

>>
>> [big snip]
>>
>> >

>>
>> Whenever I need date manipulations I always reach out to this
>> http://labix.org/python-dateutil

>
> The problem with dateutil is it's dog slow. Sure, I use it too, when
> convenience is more important than performance, but have you ever looked
> at the code for dateutil.parser.parse()? It's worth reading, just for
> fun.
> --
> https://mail.python.org/mailman/listinfo/python-list

 
Reply With Quote
 
Tim Chase
Guest
Posts: n/a
 
      12-08-2013
On 2013-12-08 12:58, Igor Korot wrote:
> Also, the data comes from either SQLite or mySQL and so to eliminate
> the difference between those engines dates are processed as strings
> and converted to dates for the calculation purposes only.
> Maybe I will need to refactor SQLite processing to get the dates as
> dates and not a string, but that's probably for the future. so that
> dates will be kept as the datetime type until the end of the
> function. As I wrote the dates will be used as the text for the
> plotting window axis labels and as the labels they should come out
> as strings, hence the conversion.


Sqlite can do this automatically if you tell it to upon connecting:

>>> import sqlite3
>>> conn = sqlite3.connect('x.sqlite',

.... detect_types=sqlite3.PARSE_DECLTYPES|sqlite3.PARSE _COLNAMES)
>>> cur.execute("create table foo (s date);")

<sqlite3.Cursor object at 0x7f7f31665570>
>>> import datetime
>>> today = datetime.date.today()
>>> cur.execute("insert into foo(s) values (?)", (today,))

<sqlite3.Cursor object at 0x7f7f31665570>
>>> cur.execute("select * from foo")

<sqlite3.Cursor object at 0x7f7f31665570>
>>> r = cur.fetchone()
>>> print r

(datetime.date(2013, 12, ,)


Note that it returns a datetime.date, the same as it was defined.

-tkc




 
Reply With Quote
 
Dave Angel
Guest
Posts: n/a
 
      12-08-2013
On Sun, 8 Dec 2013 12:58:18 -0800, Igor Korot <(E-Mail Removed)>
wrote:
> It's input is the query result, so there is no looping when the
> function is called. It is called only once.


Then why save part of the result in an instance attribute? Just
return all of the results as a tuple.

--
DaveA

 
Reply With Quote
 
Igor Korot
Guest
Posts: n/a
 
      12-15-2013
Tim,

On Sun, Dec 8, 2013 at 2:18 PM, Tim Chase <(E-Mail Removed)> wrote:
> On 2013-12-08 12:58, Igor Korot wrote:
>> Also, the data comes from either SQLite or mySQL and so to eliminate
>> the difference between those engines dates are processed as strings
>> and converted to dates for the calculation purposes only.
>> Maybe I will need to refactor SQLite processing to get the dates as
>> dates and not a string, but that's probably for the future. so that
>> dates will be kept as the datetime type until the end of the
>> function. As I wrote the dates will be used as the text for the
>> plotting window axis labels and as the labels they should come out
>> as strings, hence the conversion.

>
> Sqlite can do this automatically if you tell it to upon connecting:
>
>>>> import sqlite3
>>>> conn = sqlite3.connect('x.sqlite',

> ... detect_types=sqlite3.PARSE_DECLTYPES|sqlite3.PARSE _COLNAMES)
>>>> cur.execute("create table foo (s date);")

> <sqlite3.Cursor object at 0x7f7f31665570>
>>>> import datetime
>>>> today = datetime.date.today()
>>>> cur.execute("insert into foo(s) values (?)", (today,))

> <sqlite3.Cursor object at 0x7f7f31665570>
>>>> cur.execute("select * from foo")

> <sqlite3.Cursor object at 0x7f7f31665570>
>>>> r = cur.fetchone()
>>>> print r

> (datetime.date(2013, 12, ,)


Interesting.
I'm using datetime rather than the date type for CREATE TABLE() command.
And when running SELECT I still see the b'1998-08-05 23:12:12' string
representation when running my program under debugger.

And it is confirmed by running this:

>>> cur.execute("CREATE TABLE foo(bar datetime);")

<sqlite3.Cursor object at 0x00E1E6A0>
>>> import datetime
>>> today = datetime.date.today()
>>> cur.execute("insert into foo(bar) values (?)", (today,))

<sqlite3.Cursor object at 0x00E1E6A0>
>>> cur.execute("select * from foo")

<sqlite3.Cursor object at 0x00E1E6A0>
>>> res = cur.fetchall()
>>> print res

[(u'2013-12-14',)]
>>>


So, I guess this is a bug in the sqlite3 python module as datetime is
legal data type on the DB engine.

Thank you.

>
>
> Note that it returns a datetime.date, the same as it was defined.
>
> -tkc
>
>
>
>
> --
> https://mail.python.org/mailman/listinfo/python-list

 
Reply With Quote
 
Tim Chase
Guest
Posts: n/a
 
      12-15-2013
On 2013-12-14 23:49, Igor Korot wrote:
> Tim,
>
> On Sun, Dec 8, 2013 at 2:18 PM, Tim Chase wrote:
>>>>> conn = sqlite3.connect('x.sqlite',

>>... detect_types=sqlite3.PARSE_DECLTYPES|sqlite3.PARSE _COLNAMES)


Your example code omitted this one crucial line. Do you specify the
detect_types parameter to connect()?

-tkc



 
Reply With Quote
 
Tim Chase
Guest
Posts: n/a
 
      12-15-2013
On 2013-12-15 06:17, Tim Chase wrote:
>>>>>> conn = sqlite3.connect('x.sqlite',
>>>... detect_types=sqlite3.PARSE_DECLTYPES|sqlite3.PARSE _COLNAMES)

>
> Your example code omitted this one crucial line. Do you specify the
> detect_types parameter to connect()?


It's really the PARSE_DECLTYPES that is important.

http://docs.python.org/2/library/sql...ARSE_DECLTYPES

-tkc


 
Reply With Quote
 
Dennis Lee Bieber
Guest
Posts: n/a
 
      12-15-2013
On Sat, 14 Dec 2013 23:49:58 -0800, Igor Korot <(E-Mail Removed)>
declaimed the following:


>
>So, I guess this is a bug in the sqlite3 python module as datetime is
>legal data type on the DB engine.
>

Note though, SQLite3 only has four native data types: integer, float,
character, blob. It accepts almost anything for a type declaration, and
will map that to the closest internal format (moreover, it will accept any
actual datatype in any actual field -- you can store strings in an integer
field unless the string content IS a representation of an integer in which
case it is turned to binary).

Regardless of how Python stores a date/datetime/time (well, except for
the floating point C date/time types), it will mostly be kept as a string
representation within SQLite3.

It then becomes the responsibility of the converter modules invoked by
the parsing option to identify and format the data. I suspect date and
datetime both parse into the same converter. I'm not sure how it behaves if
one defined converters for date and datetime separately, as I think it uses
substring matching to invoke the converter: a type of date and a type of
datetime would both match a converter for "date" if the date converter is
first in the lookup list.


--
Wulfraed Dennis Lee Bieber AF6VN
http://www.velocityreviews.com/forums/(E-Mail Removed) HTTP://wlfraed.home.netcom.com/

 
Reply With Quote
 
Igor Korot
Guest
Posts: n/a
 
      12-15-2013
Tim,

On Sun, Dec 15, 2013 at 4:29 AM, Tim Chase
<(E-Mail Removed)> wrote:
> On 2013-12-15 06:17, Tim Chase wrote:
>>>>>>> conn = sqlite3.connect('x.sqlite',
>>>>... detect_types=sqlite3.PARSE_DECLTYPES|sqlite3.PARSE _COLNAMES)

>>
>> Your example code omitted this one crucial line. Do you specify the
>> detect_types parameter to connect()?


Yes, I did.
This is the beginning of the session:

>>> import sqlite3
>>> conn = sqlite3.connect('c:\Documents and Settings\Igor.FORDANWORK\Desktop\mydb.db', detect_types = sqlite3.PARSE_DECLTYPES|sqlite3.PARSE_COLNAMES)


Also please note that you session was missing the cursor creation command.

Thank you.

>
> It's really the PARSE_DECLTYPES that is important.
>
> http://docs.python.org/2/library/sql...ARSE_DECLTYPES
>
> -tkc
>
>
> --
> https://mail.python.org/mailman/listinfo/python-list

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Eliminate "extra" variable Igor Korot Python 0 12-06-2013 07:37 PM



Advertisments