I decided to investigate AuthKit, part of Pylons, to service my authentication needs, and struggled through a lack of documentation and fairly large code base, all for your pleasure.

Authentication with AuthKit

AuthKit assumes a lot of the setup for your middleware follows Pylons conventions. It was a struggle for me to make heads or tails of the examples, not being familiar with Pylons application configuration and how requests were routed. The secret sauce to actually make AuthKit work with bottle is to realize that there are actually multiple levels of AuthKit middleware that you have to invoke to get the authorization chain to even start up. Here is how you go about it in Bottle:

from authkit import authenticate, authorize
from authkit.permissions import RemoteUser

from bottle import *

# bottle exposed function
@default() # maps to root URL
def hello():
    return "hello"

# get the default bottle application
app = default_app()

# set up an authorization permission for
# basic authentication of a remote user
app = authorize.middleware(app, RemoteUser())

# A simple authentication function
def basic_auth(environ, username, password):
    return username ==  password

# now activate the authentication
auth_config = {
    'authkit.setup.method':'basic',
    'authkit.basic.realm':'Test Realm',
    'authkit.basic.authenticate.function':basic_auth,
    'authkit.setup.enable':'True'
}
app = authenticate.middleware(app,app_conf=auth_config)

# run the application
run(app=app)

To make this work for App Engine, you need to include the AuthKit sources and account for deploying Bottle applicatios on GAE, covered in the Bottle docs and other posts.

WSGI Middleware

WSGI middleware is a handy way to add functionality to an application by adding layers to the request/response chain in between the client request and your application. Incidently, Ruby’s Rack project took inspiration from WSGI middleware.

Method Overriding

A while back I bemoaned the fact that CherryPy’s RoutedDispatcher could not handle PUT and DELETE requests from forms submitted by your typical web browser, which most often only does GET and POST requests. I submitted a patch to the CherryPy project, but I now believe that this is the wrong approach, and that middleware can handle altering the REQUEST_METHOD header in response to a submitted form with a hidden “_method” parameter, as is the accepted convention. Here is some middleware for the server side application which will resige on GAE:

# method_overide.py
# WSGI middleware to set the HTTP REQUEST_METHOD header from a submitted form
# that contains a "_method" hidden variable.

class MethodOverride(object):
  def __init__(self, app):
    self.app = app

  def __call__(self, environ, start_response):
    method = webapp.Request(environ).get('_method')
    if method:
      environ['REQUEST_METHOD'] = method.upper()
    return self.app(environ, start_response)

And here is how you would insert it in between your bottle application:

from bottle import *
from google.appengine.ext.webapp import util
from method_override import MethodOverride

@route("/test_put",method="PUT")
def testput():
  return "PUT success"

@route("/test_delete",method="DELETE")
def testdelete():
  return "DELETE success"

# run in GAE
def main():
  app= default_app()
  # insert the method override middleware
  app = MethodOverride(default_app())

  util.run_wsgi_app(app)

if __name__ == '__main__':
  main()

Now forms that define the REQUEST_METHOD as a hidden param “_method” will be routed to the correct function.

from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app

class HelloWorld(webapp.RequestHandler):
	def get(self):
		self.response.out.write("Hello World!")

app = webapp.WSGIApplication([("/",HellowWorld)])

def main():
	run_wsgi_application(app)

if __name__ == "main":
	main()

Now here is the version in Sinatra:

require "rubygems" # RubyGems is optional, depending on your setup
require "sinatra"
get '/' do
  "Hello World!"
end

Much more readable and concise. Here is a version of HellWorld in CherryPy, using the default MethodDispatcher, for comparison:

import cherrypy

class HelloWorld:
    exposed = True
    def GET(self):
        return "Hello World!"

app = HelloWorld()

d = cherrypy.dispatch.MethodDispatcher()
conf = {'/': {'request.dispatch': d}}
cherrypy.tree.mount(root, "/", conf)

Better, but you are still separating the the mapping of the root URL “/” to the HelloWorld class outside of the class. I started digging around the interwebz for Python frameworks that would work closer to the lightweight Sinatra and found Bottle. Bottle is small, self-contained and uses python decorators to map a function to a route. Brilliant. Here is the example using Bottle:

from bottle import route, run

@route("/") # assumes GET method
def hello():
	return "Hello world!"

run() # This starts the HTTP server

Now that’s what I’m talking about! Deploying a Bottle application to GAE is covered in their documentation. We’ll be using Bottle for cu3w0rx to create the master and slave daemons in later posts.

In this series we will be looking to implement a simple Map-Reduce framework that closely models the design and implementation of the CloudCrowd, which is written in Ruby. CloudCrowd  has some nice design choices. Specifically its small size (~1,800 LOC), use of JSON for message transport, and emphasis on HTTP as the protocol is quite nice. It is also pretty to look at, and the interface is entirely AJAX driven, so relies on the same service calls as the rest of the suite.  I think it is worth our while to set the stage for the project. Here are a list of things we will be taking directly from CloudCrowd:

1. The App Engine application will serve as the central master resource.
2. It will serve as the master work queue. All jobs will be submitted to it for processing.
3. All communication will be via JSON messages. As in CC, the web site will make use of the JSON returned from AJAX requests to the resource handlers.
4. There will be a clear specification for the top-level properties of a Job message, but handlers will be responsible for vetting the provided options to handle the request.
5. Operations on inputs are assumed to happen on local disk. E.g. jobs will be staged onto the worker nodes’ scratch space.
6. Jobs will have the option of a callback URL on success.
7. For simplicity’s sake, authentication to the master, and between master and slaves, will be the same basic HTTP authentication credentials.
8. A worker node will accept work item requests based on the machine’s load.

There will be several points that will stray from CloudCrowds implementation as well:

1. Nodes will publish their capabilities to the master. It will not be assumed that all nodes have all the same capabilities. In this respect cu3w0rx will more resemble the nanite project (Ruby + ERLang).
2. Worker nodes will not share the same code base as the master. The major reason for this is that worker nodes will not run on GAE, hence it makes no sense to hamstring them with the restrains that GAE imposes on python.
3. Workers will maintain their own state in a local database. This will server to keep track of capabilities, number of jobs processed, monitoring statistics, and results from previous jobs.
4. Map and reduce are implemented as two separate jobs, as far as the worker nodes are concerned.
5. Since this is a demonstration project, we will not implement a scheme to save result files to non-volitile storage. Instead we will provide a way to give authenticated access to result files from worker nodes. Result files from Reduce phases will live only at the final destination host (e.g. the hosts that have checked back into the master as having succeeded a particular job.
6. The queue is not necessarily FIFO. I am actually not sure if this is also true about CloudCrowd, but its worth mentioning here.

I would also like to implement a way to provision and configure cloud VMs using the excellent libcloud library, but I think that is outside the scope of a demonstration project. If you see anything missing (or think some stuff can be left out) leave a comment!

Yeah you can map to domain name, but it makes writing tutorials that much more of a pain

Having said that, I plan on doing one more series for Pythonistas, in order to implement a simple Map-Reduce work queue system using GAE as the master node. This comes from a direct need for us to support MapReduce type workflows on both Windows and Linux machines and an existing Ruby project that I would have loved to use (CloudCrowd) does not work on windows. In general any Ruby project that assumes fork() is available on the system tends to have problems in a Windows environment. The project is small enough that it will not be too much work to port over the concepts to Python.

“But Angel, doesn’t disco already fit the MapReduce void for Python?”

Technically, yes it does, but it relies on Erlang for communication between master and nodes, which is obviously a no-go for GAE.

“But Angel, if you are a Ruby guy, why don’t you just fork CloudCrowd and make it run on JRuby + GAE?”

Maybe one day I will, but Windows worker node compatibility is a must-have for us and as I researched CloudCrowd, the code base kept getting more and more Unix-centric. I did make a branch of the codebase that uses Ruby threads to overcome the use of fork() but the solution is non-optimal and broke when I tried to merge it back into the master branch which added even more into Unix dependencies for node CPU and memory statistics. Then other priorities took over and I have not looked back on that project.

Plus it will be an interesting project to cover on AppMecha!

Google App Engine Backup and Restore

http://github.com/aral/gaebar/tree/master

Check it out! Also GitHub.com is probably the most useful site I have ever used.

Don

Those white spaces are really pretty annoying, but there is not much I can do about it now, unless my name is Hiro, and its not. As bad as it is, though, it is only going to get worse, as there are a lot of projects that need to get out the door by end-of-year. Admittedly this is a self-imposed deadline, but a deadline none the less.

I think what really bugs me is that the white spaces don’t have an obvious pattern. I think I would not mind it as much if it was more checker-board type, or stripes:

Much better. The message is a little annoying, but I can deal with that.  Which is to say that I am trying to balance the day-jobs work-load with family and hobbies, so the posts will keep coming, but at a slower pace.

Personally I don’t think it will be Java for a few reasons. First and foremost because even though GWT is a Java based framework, the output is Java script, not compiled code. Second, the whole reason Python is so ill-disposed to C extensions is to restrict access to the file system. I would assume that arbitrary compiled Java packages carry the same problem. Possibly you can get around this by stringent sandboxing of the JVM or providing a custom compiler for GAE deployments. I don’t know enough about it to make any intelligent sort of comment (may have already proven that) but both of these solutions seem like heavy investment into an immature product. Especially when there are viable alternatives, like PHP and Ruby, that share the same sort of constraints as the current Python implementation.

If I had a vote, it would be Ruby, since I work in that every day. I am biased,  yes, but it’s my blog, so I am entitled I would not be surprised if it was PHP, though. After all, it pretty much still runs the Web.

Mako is a template library written in Python. HTML markup is intermixed with Python varaibles, control structures and statements. As such it uses similar approaches to JSP, ASP and PHP, but given the origins in MVC style Web applications, it is closer to ERB templates in Ruby land. Strictly speaking, Mako can produce any text format, not just HTML, but that is it’s main purpose.

Sidebar: Mako templates are compiled to Python modules at runtime for maximum performance. This has implications on memory and CPU usage quotas for GAE, but only if you have a tremendous amount of templates. In any case, you can configure Mako to only hold a certain amount of pre-compiled templates in memory, and the engine will retire the oldest (base on last access time) templates to make room for the new.

Let’s take a look at a basic example:

<%!
  import re
  def censor(text):
    return re.sub(r'ass', 'butt', text)
%>
<%
  msg = "<b>Hello World!</b>"
%>
<html>
<body>
  <p>Hello World!</p>
  <p>${ msg }</p>
  <p>${ msg | h }</p>
  <p>
    "Classic mistake" == ${ censor("Classic mistake") }
  </p>
</body>
</html>

OK, that’s a lot to take in, so let’s review each piece in turn. First and foremost are the directives for arbitrary Python blocks, denoted by <% … %> and module level blocks, denoted by <%! … %>. The difference between the two is that module level blocks are only evaluated on loading the template on the first time into memory, which may only be once per application. Ideally these module level blocks should be used for imports and some method definitions. The blocks follow all of the syntax rules for Python, including proper indentation.

Next is a simple expression substitution, denoted by ${ … }. The contents within the ${} tag are evaluated by Python directly and evaluated directly into a string before handing the result back to the template. You can also call any imported function or functions defined within the scope of the template, such as the censor() function defined in the module block.

If the content for the expression is user-supplied (e.g. user generated content from the data store or forms) you often want to further process that result before presenting it to the client for security reasons (e.g. preventing malicious Javascript). This is done by using filters to escape the expression’s output string to something that is non-threatening to a browser. These escapes can be added to an expression substituion using the | operator:

  ${ "<script> alert(\"danger will robinson\") ; </script> " | h }

The above expression produces

  <script> alert("danger will robinson"); </script>

instead of the possibly malicious javascript within the content of the expression. Mako includes a number of built-in escaping mechanisms, including HTML, URI and XML escaping, as well as a “trim” function.

Control Structures

Mako templates provides the basic set of control structures – conditionals (i.e. if/elif/else), loops ( while and for), and exception handling ( try/except). These are denoted by a plain “%” sign as the first non-whitespace character on a line. Since Python indentation does not apply to them, must be explicitly ended by the corresponding end tag. Here are two examples copied from the Mako docs:

% if x==5:
    this is some output
% endif

% for a in ['one', 'two', 'three', 'four', 'five']:
    % if a[0] == 't':
     its two or three
    % elif a[0] == 'f':
    four/five
    % else:
    one
    %endif
% endfor

As you can see, the conditions and iterators from Python work just fine.

Tags

In addition to these Python-embedding structures, Mako provide a few XML tags for adding functionality specific to the Mako templates. The tag names all begin with “%” but are not like the Python block demarkations since the tag end does not have a corresponding “%”. Confusing, I know, but here is the basic syntax format:

<%include file="foo.txt"/>

<%def name="foo" buffered="True">
    this is a def
</%def>

Notice how the closing “>” is not prefixed with “%” as the Python blocks are. Also notice that when the tag has content, the closing tag follows XML syntax. The following table lists the available tags and attributes:

All of the tags are very well documented in the Mako documentation, but it is worth discussing at the tag, which defines functions like the Python def, but the content of the tag follows the syntax rules of Mako. These are compiled into Python bytecode and can be evaluated in expression statements like so:

<%def name="int_square(x)">
  Integer square of ${x} is ${ int(x) * int(x) }
</%def>

${ int_square(33) }

Next post, we’ll cover how templates can call and include each other using the inherit tag.