Tuesday, January 25, 2011

AppEngine Task Queue +Transactions behavior

If you are building an app to scale up on AppEngine, I contend that a fundamental programming pattern is the use of a transaction, wherein you do exactly two things:

  1. modify the state of data in the data store
  2. queue a task to act on that new state.

This is fundamental because it permits you to implement any state machine, permitting internal and external events to advance the state of the machine. You can be a Turing Machine.

However, my experience of implementing such a pattern on AppEngine has been unsuccessful.  The task queued in step 2 does not find data committed from step 1 yet, even though these two steps were performed in a transaction. I have produced some evidence to show this is happening and I have submitted a bug report. It is possible that I am doing something wrong too.

To be fair, the documentation states that the task is guaranteed to be enqueued if and only if the transaction is committed successfully.

You can enqueue a task as part of a datastore transaction, such that the task is only enqueued—and guaranteed to be enqueued—if the transaction is committed successfully. If the transaction does get committed, the task is guaranteed to be enqueued. Once enqueued, the task is not guaranteed to execute immediately and any operations performed within the task execute independent of the original transaction. The task retries until it succeeds. This applies to any task enqueued in the context of a transaction.


It also has that phrase about executing “independently”…   So does all this mean once the task is in the queue, it is free to execute, even if the commit is not yet “done”?

But consider this: if the commit fails and there is a rollback, then we are guaranteed that the task will not be enqueued, right? so it can’t run.  This implies that the task might be present in the queue, but not permitted to execute yet until the commit. (Perhaps this explains the limit of five transactional queue adds, to limit the overhead of unlocking these. Hmmm, I wonder what underlying queue system is, if it is an open source JMS, or home-rolled queue…?)

Meanwhile I plan to recreate this problem in a separate project, and try to isolate the problem. As I investigate, I intend to share code and information in this blog, and update the comments in the bug report.

In any case, I must determine how to successfully implement this basic pattern on AppEngine. I find it truly essential to build reliable and scalable apps on AppEngine.