Intent Based Logging

Does your log message tell a story? Does it have enough context to help you understand the use case and code flow that caused the execution to flow at the given point? In one of our projects, we are experimenting with something I like to call "logging with intent". So what I mean by intent is that, at critical points in the code (entry point, significant fork points, significant intermediate points, etc.), you record the intent of what your code is doing. The implementation of this can choose to use any programming construct that makes sense (we ended up using Thread Local variables for our Google App Engine based project). But the basic concept is that you keep adding "intent breadcrumbs" along the way. Now when it is time to log a critical event (such as a warning or severe or error log), in addition to printing the specific log message, you also print the "intent trace log". So you see the log message, you also get additional context as to what the code was trying to do. Now you could use AOP or other byte code manipulation technique to add intent for every entry and exit point for every method call... but I think then it becomes too fine grained and mechanical (and there may be cases where that is useful, but there are other ways to achieve it rather than making it so pervasive). I am going to track how useful this is in real world in our application and make tweaks as necessary. I will blog about it if I end up making significant changes to the current approach.

Appengine JDO Issue: JDOFatalUserException

I recently ran into a weird appengine JDO issue (as far as I understand). Here is the background.

I got the following exception:

javax.jdo.JDOFatalUserException: Illegal argument
NestedThrowables:
java.lang.IllegalArgumentException: can't operate on multiple entity groups in a single transaction. found both Element

It is expected to see this exception when you do any DB operations (fetch, save, delete, etc.) within the context of the same JDO transaction on 2 entities that are not in the same entity-group.

I do not get this exception when there is NO transaction active on the entire execution thread of the request. This is expected and working fine, if I read the specs correctly.

Now the weird part of this. If I have another transaction in progress (of course, on another PersistenceManager, but active in the same request), I get the above exception. Note that the PersistenceManager on which I get this error is not a part of any transaction. Is this a appengine JDO implementation bug? I don't know - for now, we will need to work around this issue.

Programming against a Performance Interface

We have been experimenting with the concept of coding against performance as part of a normal developer's activities versus doing it at the end of the product release cycle. Here are some things we are trying, maybe some of them are applicable to the general development community. I will explain them in the context of our development environment but they can be applied to almost all programming environments:

When a developer codes one or more operations that taken together constitute towards a user gesture, we want the developer to think about what server's response duration is acceptable for the given user gesture. User gesture can be something like, log in or creating a business object, search, and so on. This performance number is then recorded as a contract (Application Response Interface - an ARI as we like to call it) for that user gesture. The framework will then measure the actual response time as seen by the client and any time the contract for performance is broken, will trigger a performance exception. The performance exception report is then archived/sent to the instrumentation server with other relevant contextual information (user gesture, client properties, client location, and so on). The performance exception report can be further modified to do different things when running in development mode or production mode. In addition, it can be normalized to reflect such things as network latency between the client and server (which we also measure on a periodic basis - and gets sent to the instrumentation server as a latency report every few minutes).

We have other plans to correlate this to other server internal performance, but I will write about that in a later blog post.

We have implemented this using GWT on a Java based Google App Engine (GAE) application, but can be extended to any other programming environment.

Making GWT UncaughtExceptionHandler send problem reports to server

If you are using GWT to write any level of real world application, you will most likely implement the com.google.gwt.core.client.GWT.UncaughtExceptionHandler. This is setup via the API provided via com.google.gwt.core.client.GWT (setUncaughtExceptionHandler()).

We have added some additional error reporting code in the exception handler, that will report the exception to the server. Along with the exception trace we are also sending client context and operation state information (and of course user-agent information is paired with that on the server). This entire set of information is logged in to the database for analysis. Given the differences in cross-browser implementation of some of these technologies, we hope to be able to make the quality of code better via this reporting mechanism. We simply don't have the bandwidth to test every user action on every browser (that of course is the very reason we use GWT).

We will report on our findings after letting this code run for a few weeks.

Google App Engine ArrayList JDO object serialization with GWT RPC

We recently had to make a change which made us use ArrayList as a @Persistent object for JDO persistence with GAE. In our code we are following a pattern where the Business Object, Data Access Object (DAO) and the Data Transfer Object (DTO) are implemented by the same class. For reasons specific to our code, we will be using this pattern for at least some objects in the system. We have some other adapter type pattern which does normalization/filtering of the data, but in essence we follow this pattern. What we observed is that when the DTO (marshalled via GWT RPC mechanism) comes to the server and gets adapted and saved to the Google Data Store, things work fine. That is, the ArrayList is saved correctly. When we try to access the same object later by retrieving from the Data Store, it is made available as org.datanucleus.sco.backed.ArrayList. Now things work perfectly fine, until you need to send the adapted form of that over to the client via GWT RPC. org.datanucleus.sco.backed.ArrayList is not a supported type for GWT RPC. We get the following exception:

SEVERE: [1271592597625000] javax.servlet.ServletContext log: Exception while dispatching incoming RPC call

com.google.gwt.user.client.rpc.SerializationException: Type 'org.datanucleus.sco.backed.ArrayList' was not included in the set of types which can be serialized by this SerializationPolicy or its Class object could not be loaded. For security purposes, this type will not be serialized.: instance = []

at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamWriter.serialize(ServerSerializationStreamWriter.java:610)

Since we do not want to move away from the our single BO/DAO/DTO pattern, we were forced to do a post DB fetch operation which removes the org.datanucleus.sco.backed.ArrayList and replaces that with something that is acceptable to GWT RPC.

Based on some other posts we read, we think that there may be others who are trying to use GWT RPC and JDO in this fashion on GAE. Hence we wanted to share this experience. We will be posting about this and some other GAE patterns we have established soon.

Using Captcha with Google App Engine

While developing a new registration form for web application, developers want to include captcha to reduce the possibility of accounts being signed up by spammers. Captcha is a visual / audible form of text encryption which is meant to be deciphered only by humans. A spammer would have to write a heuristic program to hack the captcha.

There are many open sourced / freeware utilities available for generating Captchas.

At Cloudglow, we evaluated the following for one of our GAE application:

1. JCAPTCHA (available for download at http://jcaptcha.sourceforge.net/)
Most of the links provided on this site were not working at the time of our evaluation. We were too reluctant to test something which didn't have main links on this site working. We will get back to it sometime when the site is back in operation.

2. SimpleCaptcha (available for download at http://simplecaptcha.sourceforge.net/)
Simple captcha is a very simple and powerful utility to generate captchas. We followed SimpleCpatcha's installation and test instructions ( available at http://simplecaptcha.sourceforge.net/installing.html) and tried it first on Mac. All this was done using eclipse as IDE and it worked in GAE development environment. However when we tested the same on Windows, it immediately returned an error for java.awt.image.BufferedImage class being not supported. Since it is not on GAE's whitelist, it is not recommended to use this class. Please track the following on GAE forum for the latest updates:http://code.google.com/p/googleappengine/issues/detail?id=1423. We have not filed a bug since if a class is not white-listed by GAE, Google is not responsible for support. We can expect enhancement from google sometime later in future. We did find differences in Google's Mac GAE development server version and Windows version when it comes to complaining about classes that are not in the GAE whitelist. We are working to file a bug on that.

3. ICPATCHA (available for download at http://code.google.com/p/icaptcha/)
ICAPTCHA uses I/OStreaming which is again not listed on GAE's whitelisted classes. Hence there was no point going any further.

4. RECAPTCHA (available for download at http://recaptcha.net/)
Recaptcha is another free and popular captcha service. This will add a dependency on their service which we do not want to have. You can refer to http://gaejexperiments.wordpress.com/2010/02/22/episode-15-using-a-captcha-in-your-google-app-engine-application/ for details.

In summary, we are back to the drawing table and may end up doing a simple mathematical captcha. Guess it is good enough for now.

Making GWT RPC endpoint independent of GWT Module path

The default GWT RPC service (Servlet) endpoint is @RemoteServiceRelativePath("some_name"), which resolves to /module_base/some_name at runtime on the client. The issue with this approach is that your RPC endpoint is now tied to GWT Module. While this may be fine for some cases, it was not for our situation. Hence this post.

We ended up creating a RPC services factory class that would create a static instance of the service endpoint and also seed it with the right endpoint; something like this:

public class ServicesFactory

{

public static final RPCServiceAsync RPCService = GWT.create(RPCService.class);

static

{

((ServiceDefTarget) RPCService).setServiceEntryPoint(GWT.getHostPageBaseURL() + RPCService.END_POINT);

}

}

Note that END_POINT is defined in the service interface itself.