EclecticGuy

As I make progress on Shrugh, I plan to capture the experience as I go. Last week I gave a status update on the Shrugh concept demo and feedback I’ve received. I’m still getting feedback (a good thing) but I have enough already on the concept and usability (collected on this Google Form) that I am ready to start exploring implementation.

As I mentioned in the last status update, I had done some prototyping in both Ruby on Rails and Google App Engine to determine which direction I want to head. To be sure, this was not an apple-to-apple comparison. RoR is really a programming language and Web framework. You still have to choose an RDBMS (most likely MySQL since I have a lot of experience with it), find a hosting service, and deal with deployment and other issues. Google App Engine, on the other hand, is a complete Cloud Computing environment ready to go. It currently uses Python for code development, the Django Web framework, and Google’s BigTable object store. This all runs on Google’s App Engine environment – which provides automatic scalability, reliability, robustness, automatic application versioning and a host of other capabilities. In addition, it is free to get started (pricing model is usage based so it scales with the application). Finally, Google provides a really nice freestanding development environment that includes a simulated BigTable datastore along with the other services. It is very easy to download and start using this environment.

So, with the demo Mockups and user feedback “on screen”, I began by designing the “model” for Shrughs and implemented them iteratively in Python. Similar to Django, App Engine uses a Model class for managing data storage.  You write your model as a Python class and App Engine does the rest. This makes it very easy to just jump in and get started.

My goal for this first iteration was to create a reasonable model for Shrugh objects, display them on screen using CSS and a templatized presentation (also part of Django), and deploy all of this on Google App Engine. This would give me experience with the entire cycle of develop and test locally then deploy, test and activate on The Cloud. That’s a lot in a short time!

So, I knew that a Shrugh needed to have these attributes: an owner, a type (yes/no, list, pros vs cons), the actual question, and a few access control attributes. Here is what I came up with:

class Shrugh(db.Model):

	owner = db.UserProperty()

	shrugh_type = db.IntegerProperty(default=0, choices=set([0,1,2]))	

	question = db.StringProperty(multiline=True)

	comment = db.StringProperty()

	share_on_open = db.IntegerProperty(default=1, choices=set([0,1,2]))

	created_datetime = db.DateTimeProperty(auto_now_add=True)

	close_datetime = db.DateTimeProperty()

	share_on_close = db.BooleanProperty(default=True)

	everyone_add = db.BooleanProperty(default=False)

A few things need to be said about App Engine and BigTable that are demonstrated in this class. First, you’ll notice that I derive from db.Model. Model is the superclass for data model definitions whose properties are static. There is another form, db.Expando, which is a superclass for data model definitions whose properties are determined dynamically. I chose Model simply because that seemed easier to get started.

Next, you’ll notice that owner is a UserProperty. App Engine is tied to Google’s user model through UserProperty. This is one area of App Engine that I am still trying to wrap my head around – and apparently so are a lot of other developers based on the number of discussions and blog posts on this subject! Ideally, Shrugh users would not have to have a Google user account. That’s fine for now during the early development stage though. (turns out that Django has its own user model and it is possible to combine it with Google’s to support Google users and non-Google users.)

You’ll also notice that some properties have a default value as well as pre-configured values (choices) for the property. This is all very easy to do.

Probably the biggest difference with BigTable and traditional RDBMs is that queries using GQL (Google’s SQL-like query language) can not perform “joins”. Unlike RDBMs, BigTable uses a distributed architecture to manage scaling to very large data sets. Developers can optimize how data is distributed by describing relationships between data objects, and by defining indexes for queries. Defining a relationship is straight-forward, simply use a ReferenceProperty to another model instance. Defining indexes for query optimization is a little less friendly – both in how it is done and in how it is deployed on the App Engine service. Defined indexes are also required for sorting result sets (like date/time ordering).

Indexes are defined in the index.yaml file (part of an application’s configuration). Once you realize that and understand what needs to be indexed and why, creating the indexes is not too tricky. The nasty part comes during deployment. When you upload your application to App Engine, the service looks at this yaml file and builds the indexes. There are reports of this taking 24-36 hours! My experience creating a single index on created_datetime (so that I can sort query results) was not very positive. I ended up having to create 2 different “applications” on App Engine before I finally got it right. The first time, App Engine completed the index build in about 30 minutes but execution resulted in a terminal failure. I never figured out why! Finally, after some reading and experimenting, I was able to upload my Shrugh prototype and App Engine built the index in about 10 minutes. Here is the current version: Shrugh Development Version.

One quick sidebar, the Google AppEngineLauncher application for Mac OSX is really nice! It manages the local application server as well as uploading to the App Engine service along with log viewing and access to the App Engine Dashboard. Here’s a screenshot:

So, in just a few hours, I had defined a model for a Shrugh and wrote some code to display Shrughs in a decent layout with a little form to let users create a Shrugh and choose a type. Not bad!