Friday, December 31, 2010

Java AppEngine Queue API semantics are dangerous!

Here is the API documentation for the Queue interface in AppEngine:

com.google.appengine.api.taskqueue
Interface Queue


public interface Queue

Queue is used to manage a task queue.

Implementations of this interface must be threadsafe.

Queues are transactional. If a datastore transaction is in progress when add() or add(TaskOptions) is invoked, the task will only be added to the queue if the datastore transaction successfully commits. If you want to add a task to a queue and have that operation succeed or fail independently of an existing datastore transaction you can invoke add(Transaction, TaskOptions) with a null transaction argument. Note that while the addition of the task to the queue can participate in an existing transaction, the execution of the task cannot participate in this transaction. In other words, when the transaction commits you are guaranteed that your task will run, not that your task executed successfully.

 

This API design is WRONG.

I make the argument that if you want your add() to the queue to be part of a transaction, then you should explicitly call the add(Transaction txn, TaskOptions taskOptions) method rather than it just “magically” work. Please don’t do magic behind my back.

If I were forced to be explicit, it would be a much clearer API design. This way you would know exactly what behavior to expect of your code without having to examine the “context” of the rest of your code where your call to add(TaskOptions taskOptions) resides. And heaven help you if your add to a queue resides in another function where an unsuspecting developer cannot immediately “see” there is a transaction context around it. Don’t make this “magically” work with transactions – make us be explicit.

A solid API design should keep developers out of trouble. Didn’t you read, or watch Joshua Bloch on this topic? Google, let’s please nip this one in the bud, even if it means painfully breaking some code living out there. :-)

Tuesday, December 28, 2010

AppEngine taskName limitations

Are you seeing an exception similar to this one?

java.lang.IllegalArgumentException: Task name does not match expression [a-zA-Z\d-]{1,500}; given taskname: 'agxlbXBvd2VyLXRleHRyLwsSBU9yZGVyIhRWNzdQWDM2UEtOQkRGQ1pSVjRWNQwLEglPcmRlckl0ZW0Y_ToM'

Task names are apparently limited to letters and digits, according to the error message, although this limitation is not explained in the API documentation. (A bug report was filed for the missing documentation; you can vote it up.)

But I ask, why the restriction? I want to have a task name containing a DataStore Key, e.g. the string produced by KeyFactory.keyToString(). It contains underscores. I wonder what is wrong with having underscores in task names?

Tuesday, December 21, 2010

using POST with a TaskQueue on AppEngine (Java)

The AppEngine docs for Task Queues (in Java) lack a sufficient example using a POST method.

It is simple to see how using a GET method will simply encode all information into the URL itself, and the servlet that accepts the task must simply deconstruct the URL (and maybe some query parameters) in order to perform its duty.

But how about receiving a POST? How does this match up with how you filled the payload? Yes it is straightforward, but the docs never explicitly tell you to pair your call to payload() with a call to getReader() in the receiving servlet. (Or you might have a need for getInputStream() instead.)

So here’s my example: I need to query for objects in the DataStore that each need processing. In the same query I need to mark those objects so I know they have been queued for processing.

A transaction is needed to make this atomic. But you can only add up to 5 things to a queue during a transaction. So the strategy is to only add 1 thing to a queue. Accumulate those object keys into a StringBuilder, and POST them all at once to a task queue.

  static public int queueObjectsNeedingProcessing() {
    int count = 0;
    Queue queue = QueueFactory.getQueue("queue-for-doing-processing");
    PersistenceManager pm = PMF.getForTransactions().getPersistenceManager();
    try {
      pm.currentTransaction().begin();
      
      Query query = pm.newQuery(Foo.class);
      // query.setFilter( ... );
      // and other query setup...
      StringBuilder buf = new StringBuilder();
      List<Foo> results = (List<Foo>) query.execute();
      for ( Foo f : results ) {
        String key = KeyFactory.keyToString(f.getId());
        buf.append(key);
        buf.append("\n");
        f.markAsQueued();
        count++;
      }
      TaskOptions taskOptions = TaskOptions.Builder.withUrl("/tasks/process-group").method(Method.POST);
      taskOptions.payload(buf.toString());
      queue.add(taskOptions);
      pm.currentTransaction().commit();
      return count;
    }
    finally {
      if (pm.currentTransaction().isActive()) {
        pm.currentTransaction().rollback();
      }
      pm.close();
    }
  }

 

On the other end, the servlet that receives the POST at /context/tasks/process-group simply recovers the posted String from the InputStream found via the HTTP request object. But rather than using the actual low level InputStream, life is better using the BufferedReader provided by request.getReader().

public class TaskWorkerServlet extends HttpServlet {
  
  final static private int SC_INTERNAL_SERVER_ERROR = 500;
  final static private String PATH_GROUP = "/process-group";
  
  @Override
  protected void doPost(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
    String pathInfo = req.getPathInfo();
    if ( pathInfo.equals(PATH_GROUP) ) {
      BufferedReader buf = req.getReader();
      String line = buf.readLine();
      while ( line != null ) {
        handleOneLine(line);
        // if you want this task to "fail" and retry the whole thing,
        // then return a HTTP error not in 2xx range. I am glossing over this.
        // resp.setStatus(SC_INTERNAL_SERVER_ERROR);
        line = buf.readLine();        
      }
      
    }
  }
}

Saturday, December 18, 2010

Unpacking the “Mario” ChromeOS netbook

Last week I was fortunate to receive one of the unbranded netbooks from Google running their ChromeOS. In fact it sat among a few other unopened holiday parcels before I realized I didn’t know what was in that particular box.

In fact, I was further confused when inside the parcel I found the box with the crazy “hamster rocket” artwork. I was fully certain that Amazon shipped some kid’s Christmas toy to the wrong address.

IMG_5332.CR2

But then I remembered I might be seeing a netbook from Google! Happy Holidays indeed! So here’s a few pics as I open the box, starting with the single page “getting started” sheet.

IMG_5334.CR2 IMG_5335.CR2

Putting the battery in:

IMG_5339.CR2

Looks like the model is called “Mario”.

IMG_5341

This device looks quite nice. Slim, lightweight – very nice size, feels good to hold/carry. On right hand side it has a 19.5VDC power jack (smaller sized, looks about like 3.5mm to me…), a USB port, and what looks like a standard audio jack. On left hand side it has a VGA out port.

IMG_5344.CR2 IMG_5345.CR2

Time to turn it on!

IMG_5347.CR2 IMG_5348.CR2

It was a snap to set up – just sign in with your google account. I was up an running in a minute. This has significant appeal as an “appliance”.

In my limited use so far, I really like how “instant” it comes to life (or goes to sleep) when you open (close) the lid. With my laptop, if I am interrupted to do something, I carefully set it aside, lid open, so I can resume later. With this device I can just shut the lid knowing I can pick up where I left off by just opening the lid. This is how it should be.

My one complaint so far is that the touchpad (or software listening to it) is not great. Trying to “drag” anything is near impossible. I also get inconsistent results when I want to do the two finger scroll. Sometimes it “jumps”. I think it sometimes interprets the two finger touch as the beginning of a click-drag move. Maybe I am doing something wrong…

The other part that is lacking is the rich variety of “cloud” apps available. Yes, this is a chicken and egg problem, and I am grateful for Google’s initiative to plant some seeds in the ecosystem in the way of Android, ChromeOS, and championing the HTML5/CSS3 efforts on the planet. But there simply aren’t much for apps yet. It would be nice if this would change with the same rate that apps have shown up for mobile devices – but I won’t hold my breath.

Ironically, the first app I wanted was a SSH client, since I conduct a lot of my life signed into unix machines. And I realize I will not likely be able to get a SSH client running in Chrome connecting directly to my unix box – getting a direct socket just isn’t possible.

So far I am liking this device and the possibilities it represents.

Monday, December 6, 2010

“Always On” fixes AppEngine timeout issues

I have anxiously awaited this “Always On” feature, previously discussed in the issue forums, and was finally announced on Dec 2 with release of App Engine version 1.4! I was getting seriously screwed by JAXB in the GoogleCheckout java library, which I blogged about previously.

So I am now paying $9 a month to keep some instances “Always On”. My control panel has an extra spot on the Instances page now:

appengine_always_on

I am still awaiting Google App Engine for Business, so that I can have a SLA for my apps. Surely this “Always On” feature will be standard since GAE for Biz will be a paid service.  I certainly think any app using JAXB has the potential to require an “Always On” approach to make it function on AppEngine without timing out.

Wednesday, November 10, 2010

Received IOException parsing the input stream for … web.xml

If you use Eclipse with the Google plugin to do development for App Engine, you might one day encounter the following error when you try to deploy your app:

  Received IOException parsing the input stream for …path/to/your/web.xml

This only happend when I updated my plugin recently, which is now:

  Google Plugin for Eclipse 3.5   1.4.0.v201010280047

The problem was the DTD in the header of the web.xml file, which was out of date. If you create a brand new Google Web Application Project, you can steal the proper header from that project.

Here’s what my web.xml header used to look like, causing deployment failure:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE web-app
    PUBLIC "-//Sun Microsystems, Inc.//DTD Web Application 2.3//EN"
    "http://java.sun.com/dtd/web-app_2_3.dtd">
<web-app>
  ...
</web-app>

 

And here’s what the new header looks like:

<?xml version="1.0" encoding="utf-8"?>
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://java.sun.com/xml/ns/javaee"
xmlns:web="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
xsi:schemaLocation="http://java.sun.com/xml/ns/javaee
http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" version="2.5">
  ...
</web-app>

Thursday, November 4, 2010

AppEngine / Google Checkout FAIL

 

I’m dying here…

appengine_googlecheckout_fail

From what I understand, Google Checkout will wait 10 seconds for a response from your callback URL.

But if your callback URL lives on App Engine, and if you are using the Google Checkout Java SDK – you’re in trouble. It will take longer than 10 seconds for your SDK to simply “warm up”.

I have tracked down the source of the problem, which I’ve blogged about earlier. It is a call to

   JAXBContext.newInstance(“com.google.checkout.sdk.domain”);

But I began wondering why this call takes *so* long on AppEngine, but only a couple seconds on my wimpy little desktop machine?

Just by examining the syntax of the above JAXB function call, one would suspect there is some Java reflection being employed to scan for all the classes present in the com.google.checkout.sdk.domain namespace. And doesn’t reflect come with some costs, like interacting with the Security Manager?  The GoogleCheckout Java SDK has 100 classes in that namespace, and I suspect Google’s JVM for AppEngine has it’s own Security Manager…   

I’m totally guessing here at a possible culprit, but I’m thinking it has to be something unique to the AppEngine environment since the same call on my wimpy dev machine is only a couple seconds vs. 20 +/- seconds on AppEngine.

I’ve already filed a bug report. I’m not sure what I am going to do next, but I have to act soon because the above errors are not going to stop. I’d really prefer to keep using the Google Checkout SDK, else I’d have to re-invent a bunch of code to deal with all the possible XML that can arrive at my callback URL – this seems a bad strategy for a system already in production – what could go wrong?! I could move that function to Rackspace, or somewhere else…  but then I’d really consider moving it all off AppEngine, and that would be too bad…

Well, something’s gotta give, and quickly.

Monday, October 25, 2010

More carbon friendly computing to come…

The new Apple MacBook Air laptops give us that “instant on” that we expect of an appliance. And when we close the lid, we get standby, letting us quickly pick up where we left off. No hard drive, just flash memory.

Yes, I want this on my desktop machine, and I’d turn it off when I’m done for the day.

http://www.apple.com/macbookair/

Sunday, October 17, 2010

Integrating Google Checkout with Analytics (and GWT)

 

I use Google Analytics to track how my site is used, so it is natural to want to track sales made via Google Checkout. There’s information on how to link these in the docs at the Google Checkout site.

Here’s how it works: First you grab the analytics tracking data that was generated for that web page. This data is represented as one big string that is Base64 encoded. The Google Checkout folks created some javascript code to help extract the analytics data and encode it for you. Once you have this encoded analytics data, you place it into a field of the XML structure that is sent to Google Checkout when the order is placed.

However, I have a couple problems with the solution presented in that document at the Google Checkout site. First, it uses an “old” version of analytics code, which at first was disconcerting. Second, it presumes you are using a form to POST, either directly to Google, or to yourself if you are doing a server-to-server order placement. But I am not using a FORM. I am using GWT and taking action in a onClick() function.

So here is what I did to adapt.

It turns out that it doesn’t matter if you are using the “old” or “new” style of analytics javascript code. The important thing is that both have a function _gat._getTracker("UA-XXXXXXX-X"), and they both do. Here’s how you can tell which you have. The “new” flavor of analytics javascript code looks something like this that follows, and is recommended to be put into your <head> section:

<script type="text/javascript">
  var _gaq = _gaq || [];
  _gaq.push(['_setAccount', 'UA-XXXXXXX-X']);
  _gaq.push(['_trackPageview']);
  (function() {
    var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
    ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
    var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
  })();
</script>

 

The “old” style, (the one used in the Google Checkout docs) looks something like this that follows, and this style of analytics was recommended to be added just before the closing <body> tag:

<script type="text/javascript">
  var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");
  document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));
</script>
<script type="text/javascript">
  try {
    var pageTracker = _gat._getTracker("UA-XXXXXXX-X");
    pageTracker._trackPageview();
  } catch(err) {
  }
</script>

 

Just following the analytics code, I added the ga_post.js code snippet from the Google Checkout folks:

<script src="http://checkout.google.com/files/digital/ga_post.js"
  type="text/javascript">
</script>

 

Let’s have a look at that code:

var URCHINFIELD = "analyticsdata";
var uacct, userv, uwv, ufsc, utitle, uflash, ugifpath;
function encode64(input) {
  var k = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/=";
  var out = "",c1,c2,c3,e1,e2,e3,e4,i = 0;
  do {
    c1 = input.charCodeAt(i++);
    c2 = input.charCodeAt(i++);
    c3 = input.charCodeAt(i++);
    e1 = c1 >> 2;
    e2 = ((c1 & 3) << 4) | (c2 >> 4);
    e3 = ((c2 & 15) << 2) | (c3 >> 6);
    e4 = c3 & 63;
    if (isNaN(c2)) {
      e3 = e4 = 64;
    } else if (isNaN(c3)) {
      e4 = 64;
    }
    out = out + k.charAt(e1) + k.charAt(e2) + k.charAt(e3) + k.charAt(e4);
  } while (i < input.length);
  return out;
}
function _uGC(l, n) {
  if (!l || l == "" || !n || n == "") return "";
  var i = l.indexOf(n);
  if (i > -1) {
    var i2 = l.indexOf(";", i);
    if (i2 < 0) {
      i2 = l.length;
    }
    var i3 = n.indexOf("=") + 1;
    return l.substring((i + i3), i2);
  }
  return "";
}
function getUrchinFieldValue() {
  var s = "__uacct=" + uacct + ";";
  s += "__userv=" + userv + ";";
  if (userv == 0 || userv == 2) {
    var i,l = document.location;
    s += "__ugifpath=" + l.protocol + "//" + l.host + ugifpath + ";";
  }
  if (typeof uwv != "undefined") s += "__uwv=" + uwv + ";";
  if (typeof ufsc != "undefined") s += "__ufsc=" + ufsc + ";";
  if (typeof utitle != "undefined") s += "__utitle=" + utitle + ";";
  if (typeof uflash != "undefined") s += "__uflash=" + uflash + ";";
  var dc = document.cookie,umcval = "";
  umcval += "__utma=" + _uGC(dc, "__utma=") + ";";
  umcval += "__utmb=" + _uGC(dc, "__utmb=") + ";";
  umcval += "__utmc=" + _uGC(dc, "__utmc=") + ";";
  umcval += "__utmz=" + _uGC(dc, "__utmz=") + ";";
  umcval += "__utmv=" + _uGC(dc, "__utmv=") + ";";
  umcval += "__utmx=" + _uGC(dc, "__utmx=") + ";";
  umcval += "__utmxx=" + _uGC(dc, "__utmxx=") + ";";
  s += "__umcval=" + escape(umcval) + ";";
  return encode64(s);
}
function getUrchinInputCode() {
  return "<input type=\"hidden\" name=\"" + URCHINFIELD + "\" value=\""
      + getUrchinFieldValue() + "\">";
}
function setUrchinInputCodeOld() {
  if (typeof uacct == "undefined") return;
  for (var i = 0; i < document.forms.length; i++) {
    if (document.forms[i].urchindata)
      document.forms[i].urchindata.value = getUrchinFieldValue();
    if (document.forms[i].analyticsdata)
      document.forms[i].analyticsdata.value = getUrchinFieldValue();
  }
}
function setUrchinInputCode(obj) {
  if (typeof _gat == "undefined") return;
  uacct = obj._getAccount();
  userv = obj._getServiceMode();
  uwv = obj._getVersion();
  ufsc = obj._getClientInfo();
  utitle = obj._getDetectTitle();
  uflash = obj._getDetectFlash();
  ugifpath = obj._getLocalGifPath();
  setUrchinInputCodeOld();
}

 

It is very straightforward – here’s a quick tour: The function setUrchinInputCode() accepts a tracker object from the analytics world, and hangs on to the variables uacct, userv, uwv, ufsc, utitle, uflash, and ugifpath. The second function of interest is getUrchinFieldValue() which takes those seven analytics fields, plus all the __utm* fields from the cookies, and a couple of others fields, and assembles them all into a single semicolon delimited string, which is then encoded Base64 and returned. The first function also saves the encoded version of all this data into a form field named “analyticsdata”, if it exists.

This is all great, except I don’t want to save the encoded data into a form field. In my case I don’t have that form field, and it turns out the code executes gracefully without it being present. I just want to grab the encoded analytics data directly in GWT and pass it on to my Java code that places my order with Google Checkout. So in my GWT code, I wrote a JSNI snippet that looks like:

  private static native String getEncodedUrchinCode() /*-{
    $wnd.setUrchinInputCode($wnd._gat._getTracker("UA-XXXXXXX-X"));
    return $wnd.getUrchinFieldValue();
  }-*/;

 

To be clear, the _gat.getTracker() function is defined in the Google Analytics code. As described above, the other two calls, i.e., setUrchinInputCode() and getUrchinFieldValue() are defined in the ga_post.js code from the Google Checkout folks. Furthermore, the use of $wnd. is needed, and was a stumbling point for me. GWT code runs in an iframe and so those javascript functions do not exist there, and you will see “undefined" errors if you do not use $wnd. to specify the main window.

In my app, I have a GWT Image defined which has a click handler, which is where I can grab the Analytics data and pass it on to my code that makes the purchase.

imgCheckoutButton.addClickHandler(new ClickHandler() {
      @Override
      public void onClick(ClickEvent event) {
        String urchinData = getEncodedUrchinCode();
        doGoogleCheckoutOrder(urchinData, orderData);
      }
    });

 

I pass along my urchinData until I eventually arrive at my server side java code that actually places the order with Google Checkout. Here, I am using the Google Checkout Java SDK, which uses JAXB to map java objects into XML structures appropriate for Google Checkout. So it is a matter of finding the correct Java class to instantiate and inject into the XML structure. The answer is you have to create a CheckoutFlowSupport object, and a MerchantCheckoutFlowSupport object, and set your urchinData there. It looks approximately like this:

// This function returns the redirect URL dictated by Google Checkout
// after sending an order. This is the URL to take the user to the proper
// Google Checkout page.
  public String placeOrder(CartOrder order, String urchinData) {
    // CartOrder and orderItems below are objects in my app that I use to build a cart.
    // ApiContext is something the Google Checkout Java SDK created. It has your merchant keys, etc.
    CheckoutShoppingCartBuilder cart = apiContext.cartPoster().makeCart();
    for ( CartOrderItem item : orderItems ) {
      cart.addItem(item.getProductDescription(), item.getRecipientDescription(), item.getPrice(), 1, dc);
    }
    // build cart and add analytics data
    CheckoutShoppingCart checkoutShoppingCart = cart.build();
    CheckoutFlowSupport checkoutFlowSupport = checkoutShoppingCart.getCheckoutFlowSupport();
    if ( checkoutFlowSupport == null ) {
      checkoutFlowSupport = new CheckoutFlowSupport();
    }
    MerchantCheckoutFlowSupport merchantCheckoutFlowSupport = checkoutFlowSupport.getMerchantCheckoutFlowSupport();
    if ( merchantCheckoutFlowSupport == null ) {
      merchantCheckoutFlowSupport = new MerchantCheckoutFlowSupport();
    }
    merchantCheckoutFlowSupport.setAnalyticsData(urchinData);
    checkoutFlowSupport.setMerchantCheckoutFlowSupport(merchantCheckoutFlowSupport);
    checkoutShoppingCart.setCheckoutFlowSupport(checkoutFlowSupport);
    
    CheckoutRedirect checkoutRedirect = apiContext.cartPoster().postCart(checkoutShoppingCart);
    return checkoutRedirect.getRedirectUrl();
  }

 

 

.

Wednesday, October 13, 2010

Google Checkout and JAXBContext vs. Google AppEngine

 

I am using the Google Checkout Java SDK v2.5 with my app running on Google App Engine. When sending a new order to Google Checkout, the 30 second threshold has been frequently exceeded and stopping people from being able to buy anything.

I spent a number of hours debugging this problem, eventually pinpointing the cause in the Google Checkout code:   a call to JAXBContext.newInstance("com.google.checkout.sdk.domain") in com.google.checkout.sdk.util.Utils.

Sure enough after pinpointing this, googling for JAXBContext and App Engine revealed that this is not an unusual problem for App Engine users. The killer for me is that I want to use the Google Checkout Java SDK, out of the box.

Here is my workaround.  I checked out the source code for the Google Checkout SDK and changed the Utils.java class. First I commented out the context variable, and the code that initialized it. For my strategy, I plan to initialize the JAXB context variable “in advance”, so that it is ready prior to sending out a new order to google checkout.

//  private static final JAXBContext context = makeJaxbContext();
//  private static JAXBContext makeJaxbContext() {
//    try {
//      return JAXBContext.newInstance("com.google.checkout.sdk.domain");
//    } catch (JAXBException e) {
//      throw new RuntimeException(e);
//    }
//  }

 

Now, for the places in Utils.java where “context” is now broken, I replaced it with a “getter” to a singleton that has the JAXB context prepared for me. For example, the fromXML() method now looks like:

public static <T> JAXBElement<T> fromXML(InputStream is) throws JAXBException {
  JAXBContext context = GaeGoogleCheckoutJaxbHelper.INSTANCE.getJaxbContext();
  return (JAXBElement<T>) context.createUnmarshaller().unmarshal(is);
}

 

And here’s the singleton that prepares the JAXB context:

import java.util.logging.Logger;
import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBException;
public enum GaeGoogleCheckoutJaxbHelper {
  INSTANCE;
  
  private JAXBContext jaxbContext;
  private GaeGoogleCheckoutJaxbHelper() {
    try {
      Logger logger = Logger.getLogger(GaeGoogleCheckoutJaxbHelper.class.getSimpleName());
      logger.info("creating new JAXB context.");
      this.jaxbContext = JAXBContext.newInstance("com.google.checkout.sdk.domain");
      logger.info("done creating new JAXB context.");
    }
    catch (JAXBException e) {
      throw new RuntimeException(e);
    }
  }
  
  public JAXBContext getJaxbContext() {
    return this.jaxbContext;
  }
  
}

 

In my case, I have three steps leading up to a purchase, with the third page being a review page with the Google Checkout button present. When I reach my “step 2” page, I queue a task that calls GaeGoogleCheckoutJaxbHelper.INSTANCE.getJaxbContext() and ignores the result. By the time the user gets to the end of step 3, the JAXB context is awaiting in memory.

This by no means is foolproof. If the user waits *too* long, then App Engine will kick out our app, and we are no better off.

So this will get me by for now while I seek a better solution. I am sad that I may have to roll my own non-JAXB google checkout client. This would be really bad.

Furthermore, I cannot understand why loading that JAXB context takes so long on appengine?! It only took 4 seconds on my wimpy 2GHz machine, yet App Engine consistently takes 20 seconds…!?!?!

This topic relates to being able to control what parts of your app “stay warm” in the JVM, which is a topic circulated elsewhere. I would clearly put my JAXB context stuff there. Google, can we have a “startup” hook for our app’s lifecycle, and keep a few things “warm”?  And can we have a Checkout SDK that integrates with App Engine nicely?

UPDATE 11/2/2010:  I am seeing timeouts on AppEngine when GoogleCheckout is calling me. Unlike the case where the client is about to contact GoogleCheckout, I can’t anticipate when to “warm up” JAXB. I have filed a bug report to the AppEngine folks, though I don’t know what they can do without giving “special treatment” to some classes or libs. I suppose they could just increase the timeout. On the other hand, it really should not take so long for JAXBContext.newInstance() to execute.

Monday, May 3, 2010

Microsoft is responding to HTML5

Last Thursday, Steve Jobs posted a public letter “Thoughts on Flash” addressing the criticisms of Apple choosing not to use Adobe Flash on their mobile devices. He makes very good points, and I agree with him on those points, one of which was the use of HTML5/CSS3/JavaScript as open tools to create rich user experience.

So Apple is another big player who has clearly bought in to the HTML5/CSS/JavaScript future of web clients. I think this adds more pressure to Microsoft for their HTML5 browser, who contrary to my bias, is not asleep at the wheel. Just lagging behind perhaps. And already fully aware that the world is heading this direction, and they better get on board.

Nearly the same time as Steve’s post last week, Microsoft’s Dean Hachamovitch (IE general manager) spoke of the importance of HTML5 in bringing richer user experience in the browser. There was also some evidence of HTML5 support in IE9 at a session at Mix10. So I am beginning to believe we can expect some progress on this front in IE9.

Friday, April 30, 2010

Will Chromium OS improve carbon emissions?

With Google’s Chromium operating system, it is a goal to not have to wait so long to have a usable machine once you power it on. In fact the goal is seven seconds from the time you push the “on” button, to being able to access the internet (which is where Google wishes all your apps to live).

Hmmm. Seven seconds to being in a usable state. Can I have all my apps and windows where I left them too? Can I run all my developer tools yet? Ok, this is going to take some time to get there. But when we do, I might actually start turning off my computer(s) and emitting less carbon.

Setting this kind of standard, making the PC more of an “appliance” in a sense, could have a small impact on carbon emissions in the world. Cool.

Hey Microsoft, where’s your HTML5 browser?

Ok, this is just a rant.

I am really getting annoyed with IE being such a “special case” to deal with. I long for the days when moving an application to the web was the ultimate cross platform solution. Mac, PC, Unix – sure, just point your web browser at it! It runs everywhere!

I’m not telling you anything you don’t already know. Yes, we are sadly returning to wasted time and energy dealing with browser variations, as we once did with PC vs. Mac apps 20 years ago. And dealing with this cross platform issue will become more and more pronounced as richer features on the client side are employed by more and more developers. These richer web apps are only going to grow; I have a new Comet application employing server push, and the client side *really* stinks, just because of IE.

So I appeal to you developers: don’t do it. Don’t include IE! Save your time and energy. Instead, write an awesome app and tell your users to install a browser that keeps up with modern specs. Ok, I know that is insane to ask some users to actually install anything on their machine – there’s no chance. (The EU was right…) But today I stumbled upon someone who told IE to jump in a lake. Clarity Accounting. Bravo, Clarity Accounting!

Hypocritical? I mean, I hate it when a website requires me to use IE for some special thing that only IE does. However, there is a difference. I do not find it terribly unreasonable to expect browsers to comply with a modern specification, like HTML5. So Microsoft, where’s your HTML5 compliant browser? Evidence would suggest that it is not even on your radar.

A year ago at Google I/O, Google really pitched hard developers about all the great rich client stuff you can accomplish in an HTML5 browser. Granted, Google is placing their bets on cloud computing, and having proper platforms to access cloud apps is critical. Thus we see Google’s hands in action with the Chrome Browser, the upcoming Chromium OS (basically everything in a browser), and Android (the other place where we interact with our online life).

So I guess Microsoft is in a real spot. Comply with HTML5 spec, and developers can create rich client apps with easy cross browser compatibility. And those open-spec savvy developers won’t necessarily be using Silverlight.  Now broad user acceptance of cloud applications could reach a tipping point more easily with a better IE.   All those people who know only IE would begin to see new rich client apps developed which run from the cloud. Perhaps that’s a threat worth dragging your feet?

Yes, I am drawing a line connecting the quality of your client experience with cloud computing. And frankly I am ignorant of how well MS Azure stands up to other cloud computing platforms; I am inferring that perhaps Microsoft execs are not ready for (or don’t know if they want to be in) the cloud biz. I just don’t see Microsoft’s rationale to drag their feet on having an HTML5 browser. Surely they have the talent to accomplish it. Or maybe that kind of talent would rather work elsewhere where doing the next obvious cool internet thing is permitted by their employer…?

End rant, back to reality. Now I have to really decide if I can really tell my users to install a better browser for my app. Damn you Microsoft.

Wednesday, April 14, 2010

Public/Private Exposure of your GeoServer

When you setup your GeoServer, you may wish to expose only a portion of it to the public. Namely, in the case of serving “tiles” for an overlay in a google map, you need to permit the public to reach the GeoWebCache, but not permit them to reach the other functionality of GeoServer.

One way to do this is by fronting your Tomcat server with an Apache server and use mod_jk to “mount” only the URLs needed to support tile serving.

The URL used by google maps to grab tiles has the format of:

var url = "http://private:8080/geoserver/gwc/service/gmaps?layers=layer_name&" +
  "zoom=" + zoom + "&x=" + coord.x + "&y=" + coord.y + "&format=image/png";

 

All the GeoServer functionality exists under  /geoserver/*, so we would like to limit exposure only to /geoserver/gwc/service/gmaps/*.

(A litmus test is loading /geoserver/gwc/demo and seeing if it exists or not. We don’t want this page to load for the public.)

 

So how to hookup mod_jk between Apache and Tomcat? I will first assume you have a working Apache installation. I am using version 2.x. I will also presume you have a working Tomcat installation with GeoServer installed and functioning.

First, Apache must have the mod_jk.so module loaded. (You may have to build mod_jk, which is beyond the scope of this post.) In the httpd.conf file, check for the following line, and add it if it doesn’t exist. There should be a whole list of LoadModule entries together in the httpd.conf.

LoadModule jk_module libexec/apache22/mod_jk.so

 

At the end of the httpd.conf, add the following:

<IfModule jk_module>
# Where to find workers.properties
# Update this path to match your conf directory location (put
# workers.properties next to httpd.conf)
JkWorkersFile /your/website/path/etc/apache22/workers.properties
# Where to put jk shared memory
# Update this path to match your local state directory or logs directory
JkShmFile     /your/website/path/log/mod_jk.shm
# Where to put jk logs
# Update this path to match your logs directory location (put mod_jk.log
# next to access_log)
JkLogFile     /your/website/path/log/mod_jk.log
# Set the jk log level [debug/error/info]
JkLogLevel    info
# Select the timestamp log format
JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
</IfModule>
## my mod_jk mounts, same for SSL section too
Include etc/apache22/mod_jk_mounts.conf

 

This sets up apache to use mod_jk and its log files. It also refers to two more important files that you must create. Those two files are mod_jk_mounts.conf and workers.properties.

The mod_jk_mounts.conf file is used to tell Apache what URL patterns to watch for, and if it has a match, it will delegate the work to render the page to a tomcat worker. The file looks like this:

<IfModule jk_module>
# this maps to /geoserver tile server at localhost tomcat
JkMount  /geoserver/gwc/service/gmaps* worker1
</IfModule>

 

Above you see that the URL is the precise URL we need to serve our tiles, i.e. /geoserver/gwc/service/gmaps/*.  So traffic at this URL will get directed to Tomcat’s worker1, which is defined in the next file, workers.properties, which looks like:

# Define workers using ajp13
worker.list=worker1
# Set properties for worker1 (ajp13)
worker.worker1.type=ajp13
worker.worker1.host=localhost
worker.worker1.port=8009

 

Apache and Tomcat speak a custom protocol on a custom port. This is the meaning of ajp13 and port 8009. It is also presumed that Tomcat is listening on a localhost address. If not, supply your local (and presumably private!) address above.

Now you should be able to serve tiles from the GeoWebCache via the front-end apache, but not be able to see the other GeoServer functionality.  So now you can use a public URL like this:

var url = "http://publicserver/geoserver/gwc/service/gmaps?layers=layer_name&" +
  "zoom=" + zoom + "&x=" + coord.x + "&y=" + coord.y + "&format=image/png";

 

 

Postscript:

It would be nice to also have a super clean URL that obscures the URL that indicates you are using GeoServer. Bu you cannot use the Apache “Alias” configuration to remap the URL because it expects the target path to be a physical path and not a virtual one. I would like to hear if anyone out there has a way to Alias mounted JK paths.