Bomber – a Node web framework — Actions

November 29th, 2009

This is another post in my series of posts about designing a web framework using Node. Catch the first two here and here.

A New Website

First a quick update: last night I decided Bomber needed its own place on the web, so I threw together a website for it. It is mostly based on the code for this site, so it was pretty quick. Anyway…

Announcing bomber.obtdev.com! I’m pretty proud of the icon, so make sure to check it out. Now I have a place to put documentation and more details then are appropriate for blog posts.

Actions

Today’s installment is going to be about what I am calling “actions”, by which I mean, the code that is run for each request. The last post talked about routing, which was the business of figuring out from the URL which code should be run. So, this is the second half of that equation. For comparison, Rails calls these “actions” and Django calls them “views”.

Conceptually actions are pretty simple. Given a request, it is their role to generate a response. If you requested an HTML list of blog posts, it is the job of the action to generate that HTML document. In an MVC framework, the action plays the role of the glue between all the different parts of application: the database, the templates, the cookie or session variables, etc. Here is an example of an action from Rails, generated by its scaffold command:

def index
  @photos = Photo.all

  respond_to do |format|
    format.html # index.html.erb
    format.xml  { render :xml => @photos }
  end
end

The code is pretty straightforward (even if you aren’t familiar with Rails). It loads a list of photos from the database, and then depending on the requested format it either renders the ‘index.html.erb’ template, or converts the list to an XML document.

Actions have a lot of different needs. The two most basic are dealing with the request and the response:

An action needs to have access to the details of the request: What HTTP method was it? What headers were sent? What were the query parameters? What was the body of the request? What POST variables were sent? What cookies were sent? What session is it associated with? What session variables are set? Where is the request from? What browser is the request from? And so on.
An action needs to be able to modify all the details of the response that it is creating. Primarily, that is setting headers, cookies, or session variables and generating the HTML.

Giving users access to the Response and the Request in existing frameworks

Django is very explicit, here’s an example of a simple view from the docs:

from django.http import HttpResponse
import datetime

def current_datetime(request):
    now = datetime.datetime.now()
    html = "<html><body>It is now %s.</body></html>" % now
    return HttpResponse(html)

You are given a request, and you return a response. This is very intuitive, and powerful but can be a bit of a pain sometimes when you don’t want to be writing all that boilerplate code over and over again. I wish I just didn’t have to type quite so much. (This is a complaint I keep coming back to over and over again with Django, and it really isn’t a strike against Django. Django is this way precisely because they don’t want to pigeon hole you into writing your projects in a specific way. But sometimes I just wish they would pick a way for me)

Rails is a bit more “magical” in the sense that a lot of the code happens behind the scenes. With Rails, an ActionController object is iniated for each request, and then the requested action method is called on that object. And because self is optional in Rails it seems like things just come out of nowhere. Here’s an example action:

def show
  @photo = Photo.find(params[:id])

  # just so you can see what it is like to set session and cookie vars
  log session[:key] # get the session variable from the response
  session[:key] = "value" # set the session variable for the response

  cookies[:key] = "value"

  # and setting headers
  response.headers["Content-Type"] = "application/pdf"

  respond_to do |format|
    format.html # show.html.erb
    format.xml  { render :xml => @photo }
  end
end

What is going on is params, session, cookies, response and respond_to are all actually methods on the ActionController object. To return a response you call a method like render, or redirect_to. I really do feel like writing applications in Rails is like writing in a-whole-nother language. Rails has request and response objects, but you generally don’t use them in favor of the other simpler methods. With Rails there isn’t a very clear distinction between requests and responses like there is in Django.

Sinatra works in a very similar way to Rails. And of the Node frameworks, the majority of them are Sinatra like in spirit.

Evented programming

If we already didn’t have enough choices for how to design an API for a web framework, Node then adds an additional level of complexity due to its evented nature.

What does that mean? Well, Node lives by the philosophy that a program should never just sit there and do nothing while it is waiting for input or output to finish. Here’s an example of what not to do from Ryan’s presentation at JSConf this year:

var result = db.query("SELECT..");

Ryan asks a simple question, “What is the software doing while it queries the database?” Well, with most libraries it is doing nothing, it is just sitting there waiting for the database to do its thing. This is inefficient. An evented library uses callbacks, so instead of waiting for the database to return, it does some other computation, and then when the database query is done, it calls the callback function. Here’s Ryan’s example:

db.query("SELECT..", function (result) { 
  // use result
});

This is pretty straight forward but what if you need to make a second database call depending on the result of the first one, and then use both those results? You’d get something like this:

db.query("SELECT..", function (result1) { 
  db.query("SELECT..", function (result2) {
    // use both results here
  });
});

And what if you need to load in a template file (which would mean reading the file from disk) and use that template to format your results?

db.query("SELECT..", function (result1) { 
  db.query("SELECT..", function (result2) {
    templates.load("file-name", function(template) {
      // use both results and the template here
    });
  });
});

There is an additional problem here. How do you keep track of state between all these different functions? In the previous example it is easy because a function is aware of all the local variables that exist at the time it is created, so the last function has access to result1 and result2 but this isn’t always the case. Especially if you are trying to reuse functions in different parts of the app.

This is very quickly getting unwieldy. And thus far none of the Node frameworks have addressed this issue head on. There have been some talks on the mailing list of adding deferred objects to Node and I think many people hope that this is going to solve the problem of writing evented web frameworks. While I think deferreds would be awesome, I think the solution needs to be a little more integrated then that.

Bomber Actions

(Note, this is the API I plan to implement in Bomber, but not all of it is completely written yet. Some of it though!)

In Bomber, the solution is Action objects, which have deferred functionality, but are a little more aware of the problem they are trying to address. Specifically, an Action has a list of tasks that should be run to complete the action. And just like with deferreds, the result of a previous task is passed to the next task. This way it is easy chain functions to be run after one another. However, (and this is the real bread and butter of Actions) unlike with deferreds (at least as far as I can tell) if the return result of a previous task is itself a deferred (or a Promise in Node-speak), the Action will wait for that deferred to finish its task before calling the next task. Now you can chain up all your tasks, and not worry about the asynchronicity of the different parts.

Action objects have a couple other tricks up their sleeves.

They will make sure that all tasks are bound to the action object itself, so you can keep track of state between tasks by setting variables on this.
Every task will be called with the request and the response for that Action as the first two arguments. This makes it easy to access those objects so you can get/set cookies, sessions or headers.
If any task returns a certain kind of object (like a ‘404 error’ object – I haven’t decided what to call it, yet) Node will stop running the tasks for that Action and send that as the response for this request.
If the result of the last task is a string or an object, Node assumes you want to use this value for the body of the request. If it is a string it assumes it is HTML and if it is any thing other than either a string or one of the objects in the previous bullet point, it converts it to JSON and sends that. This allows you to easily construct simple responses, but if you want more control you can just return null and manage it yourself.

Here is how the Server initializes an Action:

var action = new Action(request, response, view_function);
action.start();

Then a possible view_function could look like this (to tackle the example from the section on evented programming):

function view_function(request, response) {
  // in this case the db.query function is assumed to return a deferred object
  this.addTask(function(request, response) {
    return db.query("SELECT...")); 
  });

  // the next task is called with the result from the previous deferred object
  this.addTask(function(request, response, result1) {
    this.result1 = result1;
    return db.query("SELECT...")); 
  });

  this.addTask(function(request, response, result2) {
    this.result2 = result2;
    // I'm assuming templates.load is a deferred object as well...
    return templates.load('file-name');
  });

  this.addTask(function(request, response, template) {
    // use both results and the template here
  });
}

Now, I admit that at this point this looks like it might have been more work. But I guess I am going to have to try it out some to really see. I think this potentially could allow for better use of DRY programming because now it is trivial to use predeclared javascript functions as opposed to anonymous ones. That with the additional benefit of being able to return 404 errors or 3xx redirects at any point in that process will make this pretty handy.

I have one more post planned about this Bomber design stuff but it might be a little while before I get to it. (I need to figure out how I want templating to work before I can write it! But I have some ideas…)