Caching Django views with nginx and memcache

–
Tags: django, performance, nginx, memcache, scaling

Table of contents

Summary

Update: Steve Jalim has kindly continued my work on this over at the torchbox GitHub.

I wanted a way to increase performance for some of our static pages. One way to do this is to cache Django pages somewhere, and serve that content from earlier on the request pipe.

A simple approach is having nginx grab the page directly from memcache, without touching Django at all. Enter nginx_memcache.

Motivation

Performance increase. Avoiding database queries when they're unnecessary. Make your static pages load faster.

Pros and cons

What can you do or cannot do with this approach?

Pros:

  • Easy to set which views are cached from Django
  • Flag for turning on and off
  • Page versioning, so you can cache multiple versions for the same page, based on your own criteria
  • User will see the same page version based on a cookie
  • Significant speedup versus loading with Django (more below).

Cons:

  • Can only cache entire page at a time (though more research may show otherwise)
  • No auto-invalidate from Django, until user visits some non-cached page (though possible to clear this cache when updating your codebase with a commit hook).
  • Can no longer embed forms on that page unless CSRF is obtained through JS.

Results

Our tests showed a 10x increase in kbps transferred with ApacheBench tests against our landing page (run with ab -k -c 20 -n 2000, before and after).

Deployment was very straightforward. You can read the docs for details.

Alternatives

Varnish can also be placed in front of or behind nginx, and there are several articles showing how to do this.

Here's my 2 cents on django + varnish + nginx VS django + memcache + nginx:

  • (+1 for memcache) Easier to configure nginx with memcache than Varnish. Configuring Varnish implied learning a whole new language (VCL), and adding one more piece of software to our technology stack.
  • (+1 for varnish) Some perf tests out there say Varnish can be quite a bit faster. However, the difference is not orders of magnitude, so I was not compelled.
  • (+1 for memcache) Django and Memcache are very well integrated. By contrast, python-varnish and django-varnish are not actively developed and have several known issues. Didn't seem a good idea for production.