• Setting up Pinax on DreamHost Private Server
• An Improved WSGI Script for use with Django

Disable Passenger if Migrating from Shared Hosting

If you set up your shared site for Django recently, you likely enabled Phusion Passenger to serve your Django projects. While it works great on shared hosting, you’re better off using the more conventional mod_wsgi approach on a PS. If you don’t disable it, it will interfere with mod_wsgi AND use up a lot of the memory you are paying for on your PS. In the graph below, memory use had been steady at about 250 MB. After disabling Passenger, memory use leveled off at around 50 MB. This chart is from the DreamHost control panel.

Disable DreamHost Management of Web Server

To enable mod_wsgi and make a few directories accessible by Apache, you will need to modify httpd.conf. The DreamHost control panel does a great job of managing httpd.conf for you. But, unless you want to be constantly merging in your changes, you need to manage httpd.conf yourself.

Under Private Servers, select Configure Server. Then in the Web Server Configuration section, uncheck DreamHost Managed and click the Save psNNNNN settings… button.

To edit the config and restart Apache, you need to setup a user with sudo privileges. I created a special user account solely for this purpose. After creating the new user account through the control panel, give that user sudo privileges via the Manage Admin Servers entry in the Private Servers section of the DreamHost control panel.

Important Directories

In addition to knowing about /usr/local/dh/apache2, there are a couple of other important directories, some of which need to be added. Replace YOU with your username. This directory layout isn’t required, but you’ll have to adjust the rest of the instructions if you save things elsewhere.

• /home/YOU/src – Convenient place for downloading libraries for building and installing
• /home/YOU/wsgi-scripts – Will contain a wsgi-script for each Django project
• /home/YOU/projects – Will contain all Django projects

Since Apache will be accessing these directories as a different user, make sure the directories are accessible (i.e., chmod 755). I got bit by this as part of my shared hosting migration. On shared hosting, scripts get run under your user, so my project didn’t need to be group or other executable.

Inside of a Django project, I use the following layout. In this example, I’m serving up static files with the same web server. In the instructions below, I’ll be setting up an alias to the media directory from Apache.

__init.py__
manage.py
settings.py
templates
  |---- app1
     |---- base.html
     |---- index.html
     |---- ...
  |---- app2
     ...
media
  |---- css
  |---- js
  ...
...

To keep people from accessing directory listings of your media directory, you can add a .htaccess file to that directory with the following contents.

Options -Indexes

Setup Python

I ran into issues with Python 2.6 (No module named _md5), so I recommend sticking with Python 2.5. In case the default Python version on your server is 2.4, make 2.5 the default:

$ sudo rm /usr/bin/python
$ sudo ln -s /usr/bin/python2.5 /usr/bin/python

Create a directory in which to download and install libraries.

$ mkdir ~/src
$ cd ~/src

Download and install setuptools for Python 2.5.

wget http://pypi.python.org/packages/2.5/s/setuptools/setuptools-0.6c11-py2.5.egg#md5=64c94f3bf7a72a13ec83e0b24f2749b2
sudo sh setuptools-0.6c11-py2.5.egg

Upgrade Django, if desired.

$ sudo easy_install pip
$ pip install -U django

Download and build MySQLdb library. I prefer oursql, but it is easier to get started with MySQLdb.

$ tar xzf MySQL-python-1.2.3c1.tar.gz
$ cd MySQL-python-1.2.3c1
$ sudo python setup.py install

Preston’s instructions also cover the Python Imaging Library and virtualenv. I’ve skipped them here to keep the instructions shorter. You can easily install them later.

Before preceding with mod_wsgi, make sure your Django app runs on the Django dev server. It’s a lot easier to sort out issues with settings.py at this stage.

Create your Django project with django-admin.py or copy an existing project into the projects directory mentioned above.
In settings.py, add a variable that determines the location of your project directory. You’ll also need to add import os at the top of the file.

PROJECT_ROOT = os.path.realpath(os.path.dirname(__file__))

And then set up your media and templates:

MEDIA_ROOT = os.path.join(PROJECT_ROOT, 'media')
TEMPLATE_DIRS = (os.path.join(PROJECT_ROOT, 'templates'),)

In the instructions below, I set up aliases for Apache to the project media and admin media directoris, so I set both media URLs to /media/.

MEDIA_URL = '/media/'
ADMIN_MEDIA_PREFIX = '/media/'

Running with the Devserver

From your project directory, create the database if your app needs one (this assumes you’ve already set up a MySQL database through the control panel or manually).

$ chmod u+x manage.py
$ ./manage.py syncdb

Start the dev server and make sure your app loads.

./manage.py runserver 0.0.0.0:8000

If not, check the output that is being logged to the console. You may also want to temporarily change settings.py so that DEBUG = True. Kill the dev server with ctrl-c once you’ve got things sorted out.

Install mod_wsgi

Download, build and install mod_wsgi.

$ cd ~/src
$ wget http://modwsgi.googlecode.com/files/mod_wsgi-3.0c5.tar.gz
$ tar xzf mod_wsgi-3.0c5.tar.gz
$ cd mod_wsgi-3.0c5
$ ./configure --with-apxs=/usr/local/dh/apache2/template/sbin/apxs --with-python=/usr/bin/python
$ make
$ sudo make install

Create a WSGI Script

In the /home/YOU/wsgi-scripts directory create an PROJECT.wsgi file, where you replace PROJECT with your project’s name. Here’s the contents of mine . You just need to replace YOU and PROJECT with appropriate values.

import os
import sys
 
# setup_tools will try to cache egg files in surprising locations where the
# apache user likely doesn't have write access unless you do this
os.environ['PYTHON_EGG_CACHE'] = '/tmp'
 
# Ensure that parent directory of project is on PYTHONPATH
sys.path.insert(0,'/home/YOU/projects')
 
os.environ['DJANGO_SETTINGS_MODULE'] = 'PROJECT.settings'
 
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()
 
def test_wsgi(environ, start_response):
    status = '200 OK'
    output = 'Hello World! wsgi py \n' + sys.version + '\n' + '\n'.join(sys.path)
    response_headers = [('Content-type', 'text/plain'),
                        ('Content-Length', str(len(output)))]
    start_response(status, response_headers)
    return [output]
 
# Uncomment next line to test that basic mod_wsgi setup works
# and to see the contents of PYTHONPATH
application = test_wsgi

Note, that the last line is uncommented. That will allow us to first make sure that mod_wsgi is working, in general. Later, we’ll comment it out.

Test mod_wsgi

Next you need to edit the Apache config file. Change psNNNNN to your actual private server ID in each of the following steps.

To keep some of the following commands short, cd into the apache2 directory:

$ cd /usr/local/dh/apache2

To edit the Apache config (of course, you can use another editor besides vi):

$ sudo vi apache2-psNNNNN/etc/httpd.conf

To restart Apache:

$ sudo /etc/init.d/httpd2 restart apache2-psNNNNN

To view the Apache error log:

$ sudo less apache2-psNNNNN/logs/error.log

Open httpd.conf in an editor, search for the modules section and add the following line:

LoadModule wsgi_module /dh/apache2/template/lib/modules/mod_wsgi.so

Then, search for the VirtualHost section for your domain. At the end before , add the following while replacing YOU and PROJECT as appropriate:

# wsgi
<Directory /home/YOU/wsgi-scripts>
Order allow,deny
Allow from all
</Directory>

# absolute path
WSGIScriptAlias / /home/YOU/wsgi-scripts/PROJECT.wsgi

<Directory "/home/YOU/projects/PROJECT/media">
Allow from all
Order allow,deny
</Directory>

<Directory "/usr/lib/python2.5/site-packages/django/contrib/admin/media">
Allow from all
Order allow,deny
</Directory>

Alias "/media/" "/home/YOU/projects/PROJECT/media/"

# note some use hyphen - others _
Alias "/media/" "/usr/lib/python2.5/site-packages/django/contrib/admin/media/"

Only the first two sections are needed for a basic mod_wsgi test. If you run into any problems, try it with just the part up to and including the WSGIScriptAlias.

Restart Apache and access what would be the normal URL for your Django app. You should see something like:

Hello World! wsgi py
2.5 (release25-maint, Jan 24 2010, 14:37:54)
[GCC 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)]
/home/YOU/projects
/usr/lib/python2.5/site-packages/setuptools-0.6c11-py2.5.egg
/usr/lib/python2.5/site-packages/MySQL_python-1.2.3c1-py2.5-linux-x86_64.egg
/usr/lib/python2.5/site-packages/pip-0.7.1-py2.5.egg
/usr/lib/python25.zip
/usr/lib/python2.5
/usr/lib/python2.5/plat-linux2
/usr/lib/python2.5/lib-tk
/usr/lib/python2.5/lib-dynload
/usr/local/lib/python2.5/site-packages
/usr/lib/python2.5/site-packages
/usr/lib/python2.5/site-packages/PIL
/usr/lib/site-python

Test Django with mod_wsgi

Once that’s working, comment out the last line in your WSGI script and hopefully your app will load.

Troubleshooting

I found it helpful to open a couple of console windows where I could:

• Log in as admin user and tail the Apache error log (sudo tail -f apache2-psNNNNN/logs/error.log)
• Log in as admin user and restart Apache as necessary
• Log in as regular user and tail the HTTP error log (tail -f /home/YOU/DOMAIN/http/error.log)
• Log in as regular user and edit my wsgi script, settings.py, etc.

The output in the error logs was key to solving most of the problems I ran into. The two web pages I mentioned at the beginning of this post were also crucial. Graham’s post includes a lot of useful detail on WSGI. Reading it enabled me to better understand what the error messages I was seeing really meant.

I’ve hosted this site on DreamHost since 2002 and have been very happy with their performance, attitude and sense of humor. I’ve added two more domains since then.

A few months ago I developed and deployed a simple Django app for calculating magic numbers for soccer leagues on shared hosting. The app scrapes league standings off websites and displays a matrix of teams and the magic numbers. DreamHost did a fantastic job of making it really easy to deploy a Django app on shared hosting with Phusion Passenger. I only had to make a few config changes from running it locally on the Django development server.

Recently I moved onto a Private Server and ported my app to run on mod_wsgi. Although mod_wsgi is the preferred way (I think) of deploying Django apps, it required a lot more debugging and config work on my part. I’m working on a blog post to cover everything I had to do to get it working. Part of the problem came from me having previously had it running on shared hosting. Someone starting with a private server would have had fewer issues.

I’m also planning to port the app to run on Google App Engine. Since there is no database backend, it should be very straightforward.

This is all leading up to a much more complicated web app that I plan to run on my DreamHost PS, as well as GAE. The data it will be storing is a natural fit for a relational database, so it will be interesting getting the app working on the App Engine datastore. I’m also going to try MongoDB as the datastore.

• Add an environment variable for the location where you installed Django. For example, add the following line to ~/.bash_profile:
  export DJANGO_HOME=/Users/rstewart/code/Django-1.1.1
• Add a line after this one in ~/.bash_profile to read the script, e.g.,
  source $DJANGO_HOME/extras/django_bash_completion
• After saving .bash_profile, open a new terminal or run source ~/.bash_profile

Now, you can press tab to complete commands. Below, I pressed tab after entering “manage.py”.

$ python manage.py
compilemessages   diffsettings      loaddata          runserver         sqlclear          sqlreset          syncdb
createcachetable  dumpdata          makemessages      shell             sqlcustom         sqlsequencereset  test
createsuperuser   flush             reset             sql               sqlflush          startapp          validate
dbshell           inspectdb         runfcgi           sqlall            sqlindexes        startproject

Ideally, you’ll first upgrade to a 64-bit version of MySQL 5.1 or later, unless you already have. Then, follow Geert’s simple instructions for building the MySQL Python connector.

Apparently, the US Census team was so uncertain of the post Katrina population in Bay St. Louis they used a “Z” in place of a number for 2006 and 2007. They did the same for Pass Christian, Long Beach and Waveland. So then I decided to look at their estimates for nearby cities.

The population decrease in New Orleans has been written about quite a bit, but it is still startling to see the numbers next to each other. I don’t remember seeing population data for the Mississippi coastal cities.

I had been looking for an excuse to play around with the Google Charts API. Here’s a graph of the population decreases for some of the cities.

If you’re using Pydev in Eclipse, be sure /Library/Python/2.5/site-packages is in the System PYTHONPATH. Bring up the Preferences dialog, open Pydev in the tree, click on Interpreter – Python, click New Folder to the right of System PYTHONPATH, and then add the directory, if necessary.

Update 12/27/2009: With MySQL 5.1.41 and Snow Leopard (OS X 10.6), you no longer need to create the lib/mysql directory with symbolic links as described in the above article. Also, since Snow Leopard comes with Python 2.6, the path above for Eclipse should be /Library/Python/2.6/site-packages. I found Geert’s instructions to be quite helpful for Snow Leopard. There’s also a lot of useful info here.

One problem I ran into is that I initially built the MySQL Python connector with a 32-bit version of MySQL 5.1 installed. After upgrading, I forgot to delete the build directory to force the connector to completely rebuild. When I then tried to run “import MySQLdb” in the interpreter, I got the cryptic message “ImportError: dynamic module does not define init function (init_mysql)”. If I had read all of the comments in that last article before figuring this out, I might have gotten a hint from the comment mentioning that not even “setup.py clean” would delete the build directory if you left off the 64-bit architecture flags.