Fast forward nearly ten years and not only have the tools and libraries come a long way, but there are many more free or inexpensive hosted services that simplify building a tool/service like PhoneBlogger. Instead of hosting the application code on a shared hosting site, I can now build and deploy on Google App Engine. Though scalability is not an issue for my personal use of PhoneBlogger, if it were turned into a public service, App Engine would make scaling much simpler and more economical. App Engine also makes deployment a snap, though with a small amount of work, so would Fabric. For my PhoneBlogger rewrite, I decided to use App Engine.

In the original version of PhoneBlogger, I coded a bunch of static VoiceXML and JavaScript for managing the telephone interaction with a caller. At the time, three of the most prominent services for VoiceXML developers were Tellme (now owned by Microsoft), BeVocal (now owned by Nuance) and Voxeo (still independent). I had to write slightly different code for Tellme and BeVocal, but the differences weren’t that significant. I think it would have been pretty simple to port to Voxeo, as well. Improved support of VoiceXML 2 would now likely allow me to use the same code on each platform.

While VoiceXML is still a great option for building speech apps, a couple of new services bring you simple APIs for building speech or DTMF (touchtone) applications, at the cost of portability. This time around I’ve started with Twilio. I very quickly turned a Python/GAE example from the Twilio website into a DTMF app for tweeting by phone. Although speech recognition allows you to build much more complex and natural applications, many simple applications can be built quickly and easily with just support for pressing keys to provide input. PhoneBlogger falls into that category, for now.

One very convenient thing about Twilio is that I can use their platform to capture and host recordings in a format that is simple to play back in a web browser. If I were really concerned about longevity of the recordings I could easily retrieve them and store them elsewhere, but I’m okay with keeping them on Twilio servers for now. That’s an easy enhancement to add later. The biggest downside for tweeting the Twilio links is that the Twilio recording URLs are ginormous. Fortunately, the goo.gl URL shortener made quick work of that problem.

I’m also going to take a look at porting my code to Tropo, which is a service offered by Voxeo. Tropo is built on Voxeo’s Prophecy platform and offers speech recognition as an option.

I decided to begin the rewrite by first supporting tweeting by phone. Twitter offers a great API, which is made even simpler by libraries like Tweepy. I highly recommend first checking out the OAuth support in any library for Twitter you might consider using. OAuth can be a complex beast, but libraries like Tweepy make it almost trivial.

The original PhoneBlogger source code and a couple iterations of it are available on SourceForge. I wasn’t particularly interested in learning about CVS at the time, so I just uploaded tarballs of all the code. While SourceForge has improved a lot, I’ve become more of a fan of GitHub. Google Code, LaunchPad and BitBucket are also great options. I started using LaunchPad when working on a Java library for Gearman, but then set up a couple of repos on GitHub when I started working on Log4mongo-Java. I’m much happier with Git, Bazaar and Mercurial than Subversion and CVS (Caveman Versioning System). I’ve already started posting code for the new phoneblogger project on GitHub.

As of now, the new version of PhoneBlogger supports tweeting by phone. All the code is on GitHub, along with a README file with the basic steps to set it up for yourself. In an upcoming blog post I’ll walk through those steps in a little more detail.

I’ve very excited to be speaking on Logging Application Behavior to MongoDB at Mongo SV 2010 in Mountain View on December 3rd. Interest in the conference has soared and 10Gen is now expecting 500 attendees. Just a week ago, they were at 300 attendees. They had to add a fourth track of talks to handle the interest.

If you’re going to be at Mongo SV, please let me know or look for me at the conference. I’ll obviously be easy to find between 2:45 and 3:30 pm.

In my talk I’m planning to cover:

• Why MongoDB is useful for application logging
• Strategies for logging to MongoDB
• Commonalities in logging libraries
• General purpose and MongoDB-specific logging libraries for:
  • Java
  • Python
  • Ruby
  • PHP
  • C#/.NET
• Analyzing logs in MongoDB

• Setting up Pinax on DreamHost Private Server
• An Improved WSGI Script for use with Django

Disable Passenger if Migrating from Shared Hosting

If you set up your shared site for Django recently, you likely enabled Phusion Passenger to serve your Django projects. While it works great on shared hosting, you’re better off using the more conventional mod_wsgi approach on a PS. If you don’t disable it, it will interfere with mod_wsgi AND use up a lot of the memory you are paying for on your PS. In the graph below, memory use had been steady at about 250 MB. After disabling Passenger, memory use leveled off at around 50 MB. This chart is from the DreamHost control panel.

Disable DreamHost Management of Web Server

To enable mod_wsgi and make a few directories accessible by Apache, you will need to modify httpd.conf. The DreamHost control panel does a great job of managing httpd.conf for you. But, unless you want to be constantly merging in your changes, you need to manage httpd.conf yourself.

Under Private Servers, select Configure Server. Then in the Web Server Configuration section, uncheck DreamHost Managed and click the Save psNNNNN settings… button.

To edit the config and restart Apache, you need to setup a user with sudo privileges. I created a special user account solely for this purpose. After creating the new user account through the control panel, give that user sudo privileges via the Manage Admin Servers entry in the Private Servers section of the DreamHost control panel.

Important Directories

In addition to knowing about /usr/local/dh/apache2, there are a couple of other important directories, some of which need to be added. Replace YOU with your username. This directory layout isn’t required, but you’ll have to adjust the rest of the instructions if you save things elsewhere.

• /home/YOU/src – Convenient place for downloading libraries for building and installing
• /home/YOU/wsgi-scripts – Will contain a wsgi-script for each Django project
• /home/YOU/projects – Will contain all Django projects

Since Apache will be accessing these directories as a different user, make sure the directories are accessible (i.e., chmod 755). I got bit by this as part of my shared hosting migration. On shared hosting, scripts get run under your user, so my project didn’t need to be group or other executable.

Inside of a Django project, I use the following layout. In this example, I’m serving up static files with the same web server. In the instructions below, I’ll be setting up an alias to the media directory from Apache.

__init.py__
manage.py
settings.py
templates
  |---- app1
     |---- base.html
     |---- index.html
     |---- ...
  |---- app2
     ...
media
  |---- css
  |---- js
  ...
...

Options -Indexes

Setup Python

I ran into issues with Python 2.6 (No module named _md5), so I recommend sticking with Python 2.5. In case the default Python version on your server is 2.4, make 2.5 the default:

$ sudo rm /usr/bin/python
$ sudo ln -s /usr/bin/python2.5 /usr/bin/python

$ mkdir ~/src
$ cd ~/src

wget http://pypi.python.org/packages/2.5/s/setuptools/setuptools-0.6c11-py2.5.egg#md5=64c94f3bf7a72a13ec83e0b24f2749b2
sudo sh setuptools-0.6c11-py2.5.egg

$ sudo easy_install pip
$ pip install -U django

$ tar xzf MySQL-python-1.2.3c1.tar.gz
$ cd MySQL-python-1.2.3c1
$ sudo python setup.py install

Before preceding with mod_wsgi, make sure your Django app runs on the Django dev server. It’s a lot easier to sort out issues with settings.py at this stage.

Create your Django project with django-admin.py or copy an existing project into the projects directory mentioned above. In settings.py, add a variable that determines the location of your project directory. You’ll also need to add import os at the top of the file.

PROJECT_ROOT = os.path.realpath(os.path.dirname(file))

MEDIA_ROOT = os.path.join(PROJECT_ROOT, 'media')
TEMPLATE_DIRS = (os.path.join(PROJECT_ROOT, 'templates'),)

MEDIA_URL = '/media/'
ADMIN_MEDIA_PREFIX = '/media/'

Running with the Devserver

From your project directory, create the database if your app needs one (this assumes you’ve already set up a MySQL database through the control panel or manually).

$ chmod u+x manage.py
$ ./manage.py syncdb

./manage.py runserver 0.0.0.0:8000

Install mod_wsgi

Download, build and install mod_wsgi.

$ cd ~/src
$ wget http://modwsgi.googlecode.com/files/mod_wsgi-3.0c5.tar.gz
$ tar xzf mod_wsgi-3.0c5.tar.gz
$ cd mod_wsgi-3.0c5
$ ./configure --with-apxs=/usr/local/dh/apache2/template/sbin/apxs --with-python=/usr/bin/python
$ make
$ sudo make install

Create a WSGI Script

In the /home/YOU/wsgi-scripts directory create a PROJECT.wsgi file, where you replace PROJECT with your project’s name. Below are the contents of mine. Replace YOU and PROJECT with appropriate values. See the comments for this post for an explanation of why you should use /var/tmp instead of /tmp for PYTHON_EGG_CACHE.

import os
import sys
 
# setup_tools will try to cache egg files in surprising locations where the
# apache user likely doesn't have write access unless you do this
os.environ['PYTHON_EGG_CACHE'] = '/var/tmp'
 
# Ensure that parent directory of project is on PYTHONPATH
sys.path.insert(0,'/home/YOU/projects')
 
os.environ['DJANGO_SETTINGS_MODULE'] = 'PROJECT.settings'
 
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()
 
def test_wsgi(environ, start_response):
    status = '200 OK'
    output = 'Hello World! wsgi py \n' + sys.version + '\n' + '\n'.join(sys.path)
    response_headers = [('Content-type', 'text/plain'),
                        ('Content-Length', str(len(output)))]
    start_response(status, response_headers)
    return [output]
 
# Uncomment next line to test that basic mod_wsgi setup works
# and to see the contents of PYTHONPATH
application = test_wsgi

Note, that the last line is uncommented. That will allow us to first make sure that mod_wsgi is working, in general. Later, we’ll comment it out.

Test mod_wsgi

Next you need to edit the Apache config file. Change psNNNNN to your actual private server ID in each of the following steps.

To keep some of the following commands short, cd into the apache2 directory:

$ cd /usr/local/dh/apache2

$ sudo vi apache2-psNNNNN/etc/httpd.conf

$ sudo /etc/init.d/httpd2 restart apache2-psNNNNN

$ sudo less apache2-psNNNNN/logs/error.log

LoadModule wsgi_module /dh/apache2/template/lib/modules/mod_wsgi.so

# wsgi
<Directory /home/YOU/wsgi-scripts>
Order allow,deny
Allow from all
</Directory>

absolute path

WSGIScriptAlias / /home/YOU/wsgi-scripts/PROJECT.wsgi

<Directory "/home/YOU/projects/PROJECT/media"> Allow from all Order allow,deny </Directory>

<Directory "/usr/lib/python2.5/site-packages/django/contrib/admin/media"> Allow from all Order allow,deny </Directory>

Alias "/media/" "/home/YOU/projects/PROJECT/media/"

note some use hyphen - others _

Alias "/media/" "/usr/lib/python2.5/site-packages/django/contrib/admin/media/" Only the first two sections are needed for a basic mod_wsgi test. If you run into any problems, try it with just the part up to and including the WSGIScriptAlias.

Restart Apache and access what would be the normal URL for your Django app. You should see something like:

Hello World! wsgi py
2.5 (release25-maint, Jan 24 2010, 14:37:54)
[GCC 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)]
/home/YOU/projects
/usr/lib/python2.5/site-packages/setuptools-0.6c11-py2.5.egg
/usr/lib/python2.5/site-packages/MySQL_python-1.2.3c1-py2.5-linux-x86_64.egg
/usr/lib/python2.5/site-packages/pip-0.7.1-py2.5.egg
/usr/lib/python25.zip
/usr/lib/python2.5
/usr/lib/python2.5/plat-linux2
/usr/lib/python2.5/lib-tk
/usr/lib/python2.5/lib-dynload
/usr/local/lib/python2.5/site-packages
/usr/lib/python2.5/site-packages
/usr/lib/python2.5/site-packages/PIL
/usr/lib/site-python

Test Django with mod_wsgi

Once that’s working, comment out the last line in your WSGI script and hopefully your app will load.

Troubleshooting

I found it helpful to open a couple of console windows where I could:

• Log in as admin user and tail the Apache error log (sudo tail -f apache2-psNNNNN/logs/error.log)
• Log in as admin user and restart Apache as necessary
• Log in as regular user and tail the HTTP error log (tail -f /home/YOU/DOMAIN/http/error.log)
• Log in as regular user and edit my wsgi script, settings.py, etc.

The output in the error logs was key to solving most of the problems I ran into. The two web pages I mentioned at the beginning of this post were also crucial. Graham’s post includes a lot of useful detail on WSGI. Reading it enabled me to better understand what the error messages I was seeing really meant.

I’ve hosted this site on DreamHost since 2002 and have been very happy with their performance, attitude and sense of humor. I’ve added two more domains since then.

A few months ago I developed and deployed a simple Django app for calculating magic numbers for soccer leagues on shared hosting. The app scrapes league standings off websites and displays a matrix of teams and the magic numbers. DreamHost did a fantastic job of making it really easy to deploy a Django app on shared hosting with Phusion Passenger. I only had to make a few config changes from running it locally on the Django development server.

Recently I moved onto a Private Server and ported my app to run on mod_wsgi. Although mod_wsgi is the preferred way (I think) of deploying Django apps, it required a lot more debugging and config work on my part. I’m working on a blog post to cover everything I had to do to get it working. Part of the problem came from me having previously had it running on shared hosting. Someone starting with a private server would have had fewer issues.

I’m also planning to port the app to run on Google App Engine. Since there is no database backend, it should be very straightforward.

This is all leading up to a much more complicated web app that I plan to run on my DreamHost PS, as well as GAE. The data it will be storing is a natural fit for a relational database, so it will be interesting getting the app working on the App Engine datastore. I’m also going to try MongoDB as the datastore.

• Add an environment variable for the location where you installed Django. For example, add the following line to ~/.bash_profile: export DJANGO_HOME=/Users/rstewart/code/Django-1.1.1
• Add a line after this one in ~/.bash_profile to read the script, e.g., source $DJANGO_HOME/extras/django_bash_completion
• After saving .bash_profile, open a new terminal or run source ~/.bash_profile

Now, you can press tab to complete commands. Below, I pressed tab after entering “manage.py”.

$ python manage.py
compilemessages   diffsettings      loaddata          runserver         sqlclear          sqlreset          syncdb
createcachetable  dumpdata          makemessages      shell             sqlcustom         sqlsequencereset  test
createsuperuser   flush             reset             sql               sqlflush          startapp          validate
dbshell           inspectdb         runfcgi           sqlall            sqlindexes        startproject

Ideally, you’ll first upgrade to a 64-bit version of MySQL 5.1 or later, unless you already have. Then, follow Geert’s simple instructions for building the MySQL Python connector.

Apparently, the US Census team was so uncertain of the post Katrina population in Bay St. Louis they used a “Z” in place of a number for 2006 and 2007. They did the same for Pass Christian, Long Beach and Waveland. So then I decided to look at their estimates for nearby cities.

The population decrease in New Orleans has been written about quite a bit, but it is still startling to see the numbers next to each other. I don’t remember seeing population data for the Mississippi coastal cities.

I had been looking for an excuse to play around with the Google Charts API. Here’s a graph of the population decreases for some of the cities.

If you’re using Pydev in Eclipse, be sure /Library/Python/2.5/site-packages is in the System PYTHONPATH. Bring up the Preferences dialog, open Pydev in the tree, click on Interpreter – Python, click New Folder to the right of System PYTHONPATH, and then add the directory, if necessary.

Update 12/27/2009: With MySQL 5.1.41 and Snow Leopard (OS X 10.6), you no longer need to create the lib/mysql directory with symbolic links as described in the above article. Also, since Snow Leopard comes with Python 2.6, the path above for Eclipse should be /Library/Python/2.6/site-packages. I found Geert’s instructions to be quite helpful for Snow Leopard. There’s also a lot of useful info here.

One problem I ran into is that I initially built the MySQL Python connector with a 32-bit version of MySQL 5.1 installed. After upgrading, I forgot to delete the build directory to force the connector to completely rebuild. When I then tried to run “import MySQLdb” in the interpreter, I got the cryptic message “ImportError: dynamic module does not define init function (init_mysql)”. If I had read all of the comments in that last article before figuring this out, I might have gotten a hint from the comment mentioning that not even “setup.py clean” would delete the build directory if you left off the 64-bit architecture flags.