Introduction to Twisted Enterprise


Twisted is an asynchronous networking framework, but most database API implementations unfortunately have blocking interfaces -- for this reason, twisted.enterprise.adbapi was created. It is a non-blocking interface to the standardized DB-API 2.0 API, which allows you to access a number of different RDBMSes.

What you should already know

Quick Overview

Twisted is an asynchronous framework. This means standard database modules cannot be used directly, as they typically work something like:

# Create connection... 
db = dbmodule.connect('mydb', 'andrew', 'password') 
# ...which blocks for an unknown amount of time 
# Create a cursor 
cursor = db.cursor() 
# Do a query... 
resultset = cursor.query('SELECT * FROM table WHERE ...') 
# ...which could take a long time, perhaps even minutes. 

Those delays are unacceptable when using an asynchronous framework such as Twisted. For this reason, twisted provides twisted.enterprise.adbapi, an asynchronous wrapper for any DB-API 2.0-compliant module. It is currently best tested with the pyPgSQL module for PostgreSQL.

enterprise.adbapi will do blocking database operations in seperate threads, which trigger callbacks in the originating thread when they complete. In the meantime, the original thread can continue doing normal work, like servicing other requests.

How do I use adbapi?

Rather than creating a database connection directly, use the adbapi.ConnectionPool class to manage a connections for you. This allows enterprise.adbapi to use multiple connections, one per thread. This is easy:

# Using the "dbmodule" from the previous example, create a ConnectionPool 
from twisted.enterprise import adbapi 
dbpool = adbapi.ConnectionPool("dbmodule", 'mydb', 'andrew', 'password') 

Things to note about doing this:

So, now you need to be able to dispatch queries to your ConnectionPool. We do this by subclassing adbapi.Augmentation. Here's an example:

class AgeDatabase(adbapi.Augmentation): 
    """A simple example that can retrieve an age from the database""" 
    def getAge(self, name): 
        # Define the query 
        sql = """SELECT Age FROM People WHERE name = ?""" 
        # Run the query, and return a Deferred to the caller to add 
        # callbacks to. 
        return self.runQuery(sql, name) 
def gotAge(resultlist, name): 
    """Callback for handling the result of the query""" 
    age = resultlist[0][0]          # First field of first record 
    print "%s is %d years old" % (name, age) 
db = AgeDatabase(dbpool) 
# These will *not* block.  Hooray! 
db.getAge("Andrew").addCallbacks(gotAge, db.operationError, 
db.getAge("Glyph").addCallbacks(gotAge, db.operationError, 

# Of course, nothing will happen until the reactor is started
from twisted.internet import reactor

This is straightforward, except perhaps for the return value of getAge. It returns a twisted.internet.defer.Deferred, which allows arbitrary callbacks to be called upon completion (or upon failure). More documentation on Deferred is available here.

Also worth noting is that this example assumes that dbmodule uses the qmarks paramstyle (see the DB-API specification). If your dbmodule uses a different paramstyle (e.g. pyformat) then use that. Twisted doesn't attempt to offer any sort of magic paramater munging -- runQuery(query, params, ...) maps directly onto cursor.execute(query, params, ...).

And that's it!

That's all you need to know to use a database from within Twisted. You probably should read the adbapi module's documentation to get an idea of the other functions it has, but hopefully this document presents the core ideas.