Extrema::blog

SimpleDS 1.0_RC1 for AppEngine has been released

Editorial -

For those that know me from somewhere else, SimpleDS is our open source framework for persistence in AppEngine. It has been compared with Objectify and Twig, and has been mentioned in the Google App Engine Blog a couple of months ago.

This is our first feature-complete release of SimpleDS. Lots of things have been included in so little time.

Getting up-to-date with AppEngine

It's hard to keep up the pace with these guys. This release includes:

  • Unindexed attributes.
  • Cursors support.
  • IN and != clauses.

Cache

We have included a great Level 1 and Level 2 cache. If you come from JDO/JPA, you may already know what this means:

  • Level 1 cache: This is basically a Map bound to the current thread. Until the end of the current request, any get() invocation will check this cache first. If a match is found, no invocation will be propagated to GAE.
  • Level 2 cache: Datastore entities are also stored in memcache, which is a second chance to get a positive match.
    Cacheable entities must be marked with @Cacheable, with an optional expiration time. This feature will work with single and batch get(), and the cache entries are updated with put() and delete() invocations.
// Invoke memcache or the datastore
List data = entityManager.get(key1, key2, key3);

// this does not invoke anything (resolved by the Level 1 cache)
MyData d2 = entityManager.get(key1);

Functions

We are going extremely functional these days. This release includes a package with functions to transform collections and PagedList instances, which can be combined with batch get() to get even better performance results. This is a simple example, equivalent to a situation quite common when using relations:

// n + 1 requests to the datastore
List data = entityManager.find(query);
Collection parents = Lists.newArrayListwithCapacity(data.size());
for (MyData d : data) {
  parents.add(entityManager.get(d.getKey().getParent());
}

// Transformations: 2 requests 
List data = entityManager.find(query);
Collection parentKeys = Collections2.transform(data, new EntityToParentKeyFunction(MyData.class));
Collection parents = entityManager.get(parentKeys);

That's all it takes to get all entities and their parents, and store them in the Level 1 cache so any request by PK will not hit memcache or the datastore. This release includes functions to retrieve parent and foreign keys, and also works with PagedList. We have taken two real-world snapshots with just this optimization (transformation + cacheable), applied to a single loop:

Before: http://www.flickr.com/photos/koliseocom/4575062969/in/set-72157623904289518/
After: http://www.flickr.com/photos/koliseocom/4575696456/in/set-72157623904289518/

Notice that "before" are also requests to the datastore, while most of the "after" are requests to memcache.

Background tasks

We are aware that background tasks are in the roadmap for AppEngine, but we needed these today. This started as an exercise to upgrade the datastore schema (add properties, delete entities etc) and ended up as a full reusable implementation of tasks that I expect to deprecate once that AppEngine includes its own, probably better, implementation.

Background tasks require adding a servlet to web.xml (and optionally appengine-web.xml as an admin-console entry) and configuring the tasks on application start using plain Java. Tasks can be invoked directly by cron triggers, queues or by POST requests.

public class WriteBehindCacheTask extends IterableTask {

 protected WriteBehindCacheTask() {
  super("my-task-id");
 }

 @Override
 protected SimpleQuery createQuery(TaskRequest request) {
  return entityManager.createQuery(MyClass.class);
 }

 @Override
 protected void process(MyClass entity, TaskRequest request) {
  // ...modify entity...
  entityManager.put(entity);
 }

}

Query cursors and execution deferral will be done transparently (no need to limit or handle cursors). There are some implementation superclasses depending on what you need to do, and some of them just use the raw AppEngine datastore and don't even require SimpleDS to work.

Other features can be checked out at the SimpleDS home page and the changelog. Any feedback is welcome :)