Velocity Reviews

Velocity Reviews (http://www.velocityreviews.com/forums/index.php)
-   Python (http://www.velocityreviews.com/forums/f43-python.html)
-   -   Re: best way to handle this in Python (http://www.velocityreviews.com/forums/t948315-re-best-way-to-handle-this-in-python.html)

Ian Kelly 07-20-2012 06:14 PM

Re: best way to handle this in Python
 
On Fri, Jul 20, 2012 at 4:34 AM, Rita <rmorgan466@gmail.com> wrote:
> Thats an interesting data structure Dennis. I will actually be running this
> type of query many times preferable in an ad-hoc environment. That makes it
> tough for sqlite3 since there will be several hundred thousand tuples.


Several hundred thousand is not an enormous number. I think you're
underestimating sqlite3. I just tried a test with one million tuples,
six colors per tuple (six million rows altogether). Each row contains
a primary key, a timestamp, a color, and a count, with an index on the
timestamp column. Building the database from scratch took about a
minute; adding the index took about another minute. Incremental
updates would of course be much faster. Queries like "select * from
data where timestamp between 500000 and 600000" return instantly (from
a user perspective).

Cheers,
Ian

Steven D'Aprano 07-21-2012 12:43 AM

Re: best way to handle this in Python
 
On Fri, 20 Jul 2012 12:14:30 -0600, Ian Kelly wrote:

> On Fri, Jul 20, 2012 at 4:34 AM, Rita <rmorgan466@gmail.com> wrote:
>> Thats an interesting data structure Dennis. I will actually be running
>> this type of query many times preferable in an ad-hoc environment. That
>> makes it tough for sqlite3 since there will be several hundred thousand
>> tuples.

>
> Several hundred thousand is not an enormous number. I think you're
> underestimating sqlite3.



A common trap, and not just for sqlite. I frequently have to remind
people -- including myself -- that what is a lot of data for you may not
be a lot of data for your computer.




--
Steven


All times are GMT. The time now is 04:32 PM.

Powered by vBulletin®. Copyright ©2000 - 2014, vBulletin Solutions, Inc.
SEO by vBSEO ©2010, Crawlability, Inc.