Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Re: Async client for PostgreSQL?

Reply
Thread Tools

Re: Async client for PostgreSQL?

 
 
Laszlo Nagy
Guest
Posts: n/a
 
      09-01-2012

> Hi
>
> does running on tornado imply that you would not consider twisted
> http://twistedmatrix.com ?
>
> If not, twisted has exactly this capability hiding long running
> queries on whatever db's behind deferToThread().

All right, I was reading its documentation

http://twistedmatrix.com/documents/1...rToThread.html

It doesn't tell too much about it: "Run a function in a thread and
return the result as a Deferred.".

Run a function but in what thread? Does it create a new thread for every
invocation? In that case, I don't want to use this. My example case: 10%
from 100 requests/second deal with a database. But it does not mean that
one db-related request will do a single db API call only. They will
almost always do more: start transaction, parse and open query, fetch
with cursor, close query, open another query etc. then commit
transaction. 8 API calls to do a quick fetch + update (usually under
100msec, but it might be blocked by another transaction for a while...)
So we are talking about 80 database API calls per seconds at least. It
would be insane to initialize a new thread for each invocation. And
wrapping these API calls into a single closure function is not useful
either, because that function would not be able to safely access the
state that is stored in the main thread. Unless you protet it with
locks. But it is whole point of async I/O server: to avoid using slow
locks, expensive threads and context switching.

Maybe, deferToThread uses a thread pool? But it doesn't say much about
it. (Am I reading the wrong documentation?) BTW I could try a version
that uses a thread pool.

It is sad, by the way. We have async I/O servers for Python that can be
used for large number of clients, but most external modules/extensions
do not support their I/O loops. Including the extension modules of the
most popular databases. So yes, you can use Twisted or torandoweb until
you do not want to call *some* API functions that are blocking. (By
*some* I mean: much less blocking than non-blocking, but quite a few.)
We also have synchronous Python servers, but we cannot get rid of the
GIL, Python threads are expensive and slow, so they cannot be used for a
large number of clients. And finally, we have messaging services/IPC
like zeromq. They are probably the most expensive, but they scale very
well. But you need more money to operate the underlying hardware. I'm
starting to think that I did not get a quick answer because my use case
(100 clients) fall into to the "heavy weight" category, and the solution
is to invest more in the hardware.

Thanks,

Laszlo

 
Reply With Quote
 
 
 
 
calderone.jeanpaul@gmail.com
Guest
Posts: n/a
 
      09-03-2012
On Saturday, September 1, 2012 3:28:52 PM UTC-4, Laszlo Nagy wrote:
> > Hi

>
> >

>
> > does running on tornado imply that you would not consider twisted

>
> > http://twistedmatrix.com ?

>
> >

>
> > If not, twisted has exactly this capability hiding long running

>
> > queries on whatever db's behind deferToThread().

>
> All right, I was reading its documentation
>
>
>
> http://twistedmatrix.com/documents/1...rToThread.html
>
>
>
> It doesn't tell too much about it: "Run a function in a thread and
>
> return the result as a Deferred.".
>


You can find more documentation here:

http://twistedmatrix.com/documents/c...threading.html

Also, Twisted has dedicated APIs for interacting with databases asynchronously:

http://twistedmatrix.com/documents/c...wto/rdbms.html

Additionally, there is a non-blocking (rather than thread-based) implementation of the above API available for PostgreSQL:

http://pypi.python.org/pypi/txpostgres

>
>
> Run a function but in what thread? Does it create a new thread for every
>
> invocation? In that case, I don't want to use this. My example case: 10%
>
> from 100 requests/second deal with a database. But it does not mean that
>
> one db-related request will do a single db API call only. They will
>
> almost always do more: start transaction, parse and open query, fetch
>
> with cursor, close query, open another query etc. then commit
>
> transaction. 8 API calls to do a quick fetch + update (usually under
>
> 100msec, but it might be blocked by another transaction for a while...)
>
> So we are talking about 80 database API calls per seconds at least. It
>
> would be insane to initialize a new thread for each invocation. And
>
> wrapping these API calls into a single closure function is not useful
>
> either, because that function would not be able to safely access the
>
> state that is stored in the main thread. Unless you protet it with
>
> locks. But it is whole point of async I/O server: to avoid using slow
>
> locks, expensive threads and context switching.
>
>
>
> Maybe, deferToThread uses a thread pool? But it doesn't say much about
>
> it. (Am I reading the wrong documentation?) BTW I could try a version
>
> that uses a thread pool.
>
>
>
> It is sad, by the way. We have async I/O servers for Python that can be
>
> used for large number of clients, but most external modules/extensions
>
> do not support their I/O loops. Including the extension modules of the
>
> most popular databases. So yes, you can use Twisted or torandoweb until
>
> you do not want to call *some* API functions that are blocking. (By
>
> *some* I mean: much less blocking than non-blocking, but quite a few.)
>
> We also have synchronous Python servers, but we cannot get rid of the
>
> GIL, Python threads are expensive and slow, so they cannot be used for a
>
> large number of clients. And finally, we have messaging services/IPC
>
> like zeromq. They are probably the most expensive, but they scale very
>
> well. But you need more money to operate the underlying hardware. I'm
>
> starting to think that I did not get a quick answer because my use case
>
> (100 clients) fall into to the "heavy weight" category, and the solution
>
> is to invest more in the hardware.
>
>
>
> Thanks,
>
>
>
> Laszlo

 
Reply With Quote
 
 
 
 
calderone.jeanpaul@gmail.com
Guest
Posts: n/a
 
      09-03-2012
On Saturday, September 1, 2012 3:28:52 PM UTC-4, Laszlo Nagy wrote:
> > Hi

>
> >

>
> > does running on tornado imply that you would not consider twisted

>
> > http://twistedmatrix.com ?

>
> >

>
> > If not, twisted has exactly this capability hiding long running

>
> > queries on whatever db's behind deferToThread().

>
> All right, I was reading its documentation
>
>
>
> http://twistedmatrix.com/documents/1...rToThread.html
>
>
>
> It doesn't tell too much about it: "Run a function in a thread and
>
> return the result as a Deferred.".
>


You can find more documentation here:

http://twistedmatrix.com/documents/c...threading.html

Also, Twisted has dedicated APIs for interacting with databases asynchronously:

http://twistedmatrix.com/documents/c...wto/rdbms.html

Additionally, there is a non-blocking (rather than thread-based) implementation of the above API available for PostgreSQL:

http://pypi.python.org/pypi/txpostgres

>
>
> Run a function but in what thread? Does it create a new thread for every
>
> invocation? In that case, I don't want to use this. My example case: 10%
>
> from 100 requests/second deal with a database. But it does not mean that
>
> one db-related request will do a single db API call only. They will
>
> almost always do more: start transaction, parse and open query, fetch
>
> with cursor, close query, open another query etc. then commit
>
> transaction. 8 API calls to do a quick fetch + update (usually under
>
> 100msec, but it might be blocked by another transaction for a while...)
>
> So we are talking about 80 database API calls per seconds at least. It
>
> would be insane to initialize a new thread for each invocation. And
>
> wrapping these API calls into a single closure function is not useful
>
> either, because that function would not be able to safely access the
>
> state that is stored in the main thread. Unless you protet it with
>
> locks. But it is whole point of async I/O server: to avoid using slow
>
> locks, expensive threads and context switching.
>
>
>
> Maybe, deferToThread uses a thread pool? But it doesn't say much about
>
> it. (Am I reading the wrong documentation?) BTW I could try a version
>
> that uses a thread pool.
>
>
>
> It is sad, by the way. We have async I/O servers for Python that can be
>
> used for large number of clients, but most external modules/extensions
>
> do not support their I/O loops. Including the extension modules of the
>
> most popular databases. So yes, you can use Twisted or torandoweb until
>
> you do not want to call *some* API functions that are blocking. (By
>
> *some* I mean: much less blocking than non-blocking, but quite a few.)
>
> We also have synchronous Python servers, but we cannot get rid of the
>
> GIL, Python threads are expensive and slow, so they cannot be used for a
>
> large number of clients. And finally, we have messaging services/IPC
>
> like zeromq. They are probably the most expensive, but they scale very
>
> well. But you need more money to operate the underlying hardware. I'm
>
> starting to think that I did not get a quick answer because my use case
>
> (100 clients) fall into to the "heavy weight" category, and the solution
>
> is to invest more in the hardware.
>
>
>
> Thanks,
>
>
>
> Laszlo

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Async client for PostgreSQL? Laszlo Nagy Python 2 09-01-2012 11:57 PM
Newbie: async mode dedicated versus async mode interactive!! Pink_Floyd Cisco 4 06-16-2006 12:16 AM
Async webservice call in async webpage (.Net 2.0) does not return Steven ASP .Net Web Services 0 11-30-2005 01:06 AM
Async Client with 1K connections? William Chang Python 12 02-13-2004 08:33 AM
problem with async chat client in windows Jonas Python 3 01-05-2004 07:24 PM



Advertisments