Friday, December 31, 2010

Java AppEngine Queue API semantics are dangerous!

Here is the API documentation for the Queue interface in AppEngine:

com.google.appengine.api.taskqueue
Interface Queue


public interface Queue

Queue is used to manage a task queue.

Implementations of this interface must be threadsafe.

Queues are transactional. If a datastore transaction is in progress when add() or add(TaskOptions) is invoked, the task will only be added to the queue if the datastore transaction successfully commits. If you want to add a task to a queue and have that operation succeed or fail independently of an existing datastore transaction you can invoke add(Transaction, TaskOptions) with a null transaction argument. Note that while the addition of the task to the queue can participate in an existing transaction, the execution of the task cannot participate in this transaction. In other words, when the transaction commits you are guaranteed that your task will run, not that your task executed successfully.

 

This API design is WRONG.

I make the argument that if you want your add() to the queue to be part of a transaction, then you should explicitly call the add(Transaction txn, TaskOptions taskOptions) method rather than it just “magically” work. Please don’t do magic behind my back.

If I were forced to be explicit, it would be a much clearer API design. This way you would know exactly what behavior to expect of your code without having to examine the “context” of the rest of your code where your call to add(TaskOptions taskOptions) resides. And heaven help you if your add to a queue resides in another function where an unsuspecting developer cannot immediately “see” there is a transaction context around it. Don’t make this “magically” work with transactions – make us be explicit.

A solid API design should keep developers out of trouble. Didn’t you read, or watch Joshua Bloch on this topic? Google, let’s please nip this one in the bud, even if it means painfully breaking some code living out there. :-)

Tuesday, December 28, 2010

AppEngine taskName limitations

Are you seeing an exception similar to this one?

java.lang.IllegalArgumentException: Task name does not match expression [a-zA-Z\d-]{1,500}; given taskname: 'agxlbXBvd2VyLXRleHRyLwsSBU9yZGVyIhRWNzdQWDM2UEtOQkRGQ1pSVjRWNQwLEglPcmRlckl0ZW0Y_ToM'

Task names are apparently limited to letters and digits, according to the error message, although this limitation is not explained in the API documentation. (A bug report was filed for the missing documentation; you can vote it up.)

But I ask, why the restriction? I want to have a task name containing a DataStore Key, e.g. the string produced by KeyFactory.keyToString(). It contains underscores. I wonder what is wrong with having underscores in task names?

Tuesday, December 21, 2010

using POST with a TaskQueue on AppEngine (Java)

The AppEngine docs for Task Queues (in Java) lack a sufficient example using a POST method.

It is simple to see how using a GET method will simply encode all information into the URL itself, and the servlet that accepts the task must simply deconstruct the URL (and maybe some query parameters) in order to perform its duty.

But how about receiving a POST? How does this match up with how you filled the payload? Yes it is straightforward, but the docs never explicitly tell you to pair your call to payload() with a call to getReader() in the receiving servlet. (Or you might have a need for getInputStream() instead.)

So here’s my example: I need to query for objects in the DataStore that each need processing. In the same query I need to mark those objects so I know they have been queued for processing.

A transaction is needed to make this atomic. But you can only add up to 5 things to a queue during a transaction. So the strategy is to only add 1 thing to a queue. Accumulate those object keys into a StringBuilder, and POST them all at once to a task queue.

  static public int queueObjectsNeedingProcessing() {
    int count = 0;
    Queue queue = QueueFactory.getQueue("queue-for-doing-processing");
    PersistenceManager pm = PMF.getForTransactions().getPersistenceManager();
    try {
      pm.currentTransaction().begin();
      
      Query query = pm.newQuery(Foo.class);
      // query.setFilter( ... );
      // and other query setup...
      StringBuilder buf = new StringBuilder();
      List<Foo> results = (List<Foo>) query.execute();
      for ( Foo f : results ) {
        String key = KeyFactory.keyToString(f.getId());
        buf.append(key);
        buf.append("\n");
        f.markAsQueued();
        count++;
      }
      TaskOptions taskOptions = TaskOptions.Builder.withUrl("/tasks/process-group").method(Method.POST);
      taskOptions.payload(buf.toString());
      queue.add(taskOptions);
      pm.currentTransaction().commit();
      return count;
    }
    finally {
      if (pm.currentTransaction().isActive()) {
        pm.currentTransaction().rollback();
      }
      pm.close();
    }
  }

 

On the other end, the servlet that receives the POST at /context/tasks/process-group simply recovers the posted String from the InputStream found via the HTTP request object. But rather than using the actual low level InputStream, life is better using the BufferedReader provided by request.getReader().

public class TaskWorkerServlet extends HttpServlet {
  
  final static private int SC_INTERNAL_SERVER_ERROR = 500;
  final static private String PATH_GROUP = "/process-group";
  
  @Override
  protected void doPost(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
    String pathInfo = req.getPathInfo();
    if ( pathInfo.equals(PATH_GROUP) ) {
      BufferedReader buf = req.getReader();
      String line = buf.readLine();
      while ( line != null ) {
        handleOneLine(line);
        // if you want this task to "fail" and retry the whole thing,
        // then return a HTTP error not in 2xx range. I am glossing over this.
        // resp.setStatus(SC_INTERNAL_SERVER_ERROR);
        line = buf.readLine();        
      }
      
    }
  }
}

Saturday, December 18, 2010

Unpacking the “Mario” ChromeOS netbook

Last week I was fortunate to receive one of the unbranded netbooks from Google running their ChromeOS. In fact it sat among a few other unopened holiday parcels before I realized I didn’t know what was in that particular box.

In fact, I was further confused when inside the parcel I found the box with the crazy “hamster rocket” artwork. I was fully certain that Amazon shipped some kid’s Christmas toy to the wrong address.

IMG_5332.CR2

But then I remembered I might be seeing a netbook from Google! Happy Holidays indeed! So here’s a few pics as I open the box, starting with the single page “getting started” sheet.

IMG_5334.CR2 IMG_5335.CR2

Putting the battery in:

IMG_5339.CR2

Looks like the model is called “Mario”.

IMG_5341

This device looks quite nice. Slim, lightweight – very nice size, feels good to hold/carry. On right hand side it has a 19.5VDC power jack (smaller sized, looks about like 3.5mm to me…), a USB port, and what looks like a standard audio jack. On left hand side it has a VGA out port.

IMG_5344.CR2 IMG_5345.CR2

Time to turn it on!

IMG_5347.CR2 IMG_5348.CR2

It was a snap to set up – just sign in with your google account. I was up an running in a minute. This has significant appeal as an “appliance”.

In my limited use so far, I really like how “instant” it comes to life (or goes to sleep) when you open (close) the lid. With my laptop, if I am interrupted to do something, I carefully set it aside, lid open, so I can resume later. With this device I can just shut the lid knowing I can pick up where I left off by just opening the lid. This is how it should be.

My one complaint so far is that the touchpad (or software listening to it) is not great. Trying to “drag” anything is near impossible. I also get inconsistent results when I want to do the two finger scroll. Sometimes it “jumps”. I think it sometimes interprets the two finger touch as the beginning of a click-drag move. Maybe I am doing something wrong…

The other part that is lacking is the rich variety of “cloud” apps available. Yes, this is a chicken and egg problem, and I am grateful for Google’s initiative to plant some seeds in the ecosystem in the way of Android, ChromeOS, and championing the HTML5/CSS3 efforts on the planet. But there simply aren’t much for apps yet. It would be nice if this would change with the same rate that apps have shown up for mobile devices – but I won’t hold my breath.

Ironically, the first app I wanted was a SSH client, since I conduct a lot of my life signed into unix machines. And I realize I will not likely be able to get a SSH client running in Chrome connecting directly to my unix box – getting a direct socket just isn’t possible.

So far I am liking this device and the possibilities it represents.

Monday, December 6, 2010

“Always On” fixes AppEngine timeout issues

I have anxiously awaited this “Always On” feature, previously discussed in the issue forums, and was finally announced on Dec 2 with release of App Engine version 1.4! I was getting seriously screwed by JAXB in the GoogleCheckout java library, which I blogged about previously.

So I am now paying $9 a month to keep some instances “Always On”. My control panel has an extra spot on the Instances page now:

appengine_always_on

I am still awaiting Google App Engine for Business, so that I can have a SLA for my apps. Surely this “Always On” feature will be standard since GAE for Biz will be a paid service.  I certainly think any app using JAXB has the potential to require an “Always On” approach to make it function on AppEngine without timing out.