WombatNation

SoccerPhone is a speech application I wrote about five years ago so I could get live updates on Major League Soccer scores whenever I was away from an Internet connection. I wrote the application in VoiceXML, JavaScript, and Python. Since SoccerPhone gathers the live data by scraping information from the HTML scoreboard page on the MLS website, I often have to update my code when the MLS website changes each year. Fortunately, this year’s change was fairly minor.

1. Call 1-877-33-VOCAL (877-338-6225)
2. When asked for your PIN, enter or say 5425 (5425 = KICK)
3. When asked for your userid, enter or say 6575425 (6575425 = MLSKICK)
4. After you hear me say “Welcome to SoccerPhone”, you can say an MLS
   team name, such as “Houston Dynamo”, or say “all teams”.

Since SoccerPhone lets you request this week’s match results for a team by saying the team’s name, I also have to update the GRXML grammars when new teams are added. While I could dynamically generate grammars from the team names that the app extracts from the MLS website, it’s not that big of a deal to manually maintain the grammars. Also, manually coding them allows me to tune the gramars for better recognition rates.

If you’ve call the app before, you’ll be disappointed to hear that I am using the same lame voice talent, i.e., me. On the good side, though, I re-recorded a bunch of the prompts using Audacity. I also eliminated a little more of the TTS by adding additional recordings. I really should take advantage of the great recording studio we have at Voxify, but then I would feel obligated to use a real voice talent.

I haven’t updated the code at the SoccerPhone SourceForge project site, yet. But I will get to that soon. I got sidetracked by looking into converting the CVS repository to Subversion.

It’s been ages since I’ve written about Soccerphone, or even about anything at all. The last few weeks have just been too hectic. But, I did find time this weekend to make a few updates to Soccerphone, which is an automated speech application I built a few years ago so I could receive live Major League Soccer scores by phone.

One update of questionable merit was to use audio recordings made by me to replace some of the prompts that are currently being synthesized by a text-to-speech engine. Not only is the use of myself as a voice talent a rather dodgy decision, but also, there is still quite a bit of TTS. I’m not sure the recording effort really improved the quality of the app that much, if at all. It was fun to do the recordings, though.

Speaking of TTS, I switched from a female voice to one of the male voices that BeVocal supports. I’m now using Reed, which is a Nuance Vocalizer voice. Not only does the app sound better due to no longer switching back and forth between genders, but the TTS engine used to synthesize the Reed voice does a much better job of pronouncing names than the TTS engine used to synthesize the Jennifer voice.

I also finally got around to adding Chivas USA and Real Salt Lake to the grammar, so you can now say them at the Team Name prompt. I added FC Dallas to the grammar, but also left in their old name, the Dallas Burn.

Another minor update was to add a dummy recognition block just before the backend query. Without this, the confirmation prompt from the previous dialog wasn’t being played until the HTTP fetch completed. Since it sometimes takes more than five seconds to get the response back, the confirmation had sounded sort of odd when it was played so late.

Two bad things happened at the MLS website this week, at least with respect to SoccerPhone (it’s a program I wrote that you can use to hear live soccer scores over the telephone). One change is pure unadulterated evil, and the other could be good or evil depending on your aesthetics.

The first pure evil change was to change the URL for the live scores page. What the heck was wrong with /scores.html? I’m sure I’m not the only MLS fan who has that page bookmarked. The new URL is /MLS/scoreboard/index.jsp, or equivalently, /MLS/scoreboard/. While I’m very happy they have gone to a Java-based solution for their website (the performance is now much, much better and the generated HTML quite a bit cleaner), I wish they could have at least redirected scores.html to the appropriate JSP page. While this change also broke SoccerPhone, at least it required only a one line code change.

So how could they have avoided changing the URL that most people used?

One approach would have been to have put the original scores page at /scores/index.html. Then, when they upgraded their site to use JSPs, they could have set up their servlet engine to look for index.jsp in that same directory if someone requested the page http://www.mlsnet.com/scores/. The people who bookmarked http://www.mlsnet.com/scores/ instead of http://www.mlsnet.com/scores/index.html would have never noticed the change. Worst case, the webmaster could have always used Apache’s mod-rewrite module to redirect the requests. Looking at the HTTP headers (thanks to the cool Live HTTP Headers plug-in for Mozilla browsers), they seem to be using Sun ONE Web Server for at least part of the site, which I assume has at least the basic URL rewriting capabilities that would have been needed. Some of the files on the MLS website are coming from an Apache 1.3.26 server.

Another approach would have been to hide the file extension. I’ve read a bit about how to do this using Apache. That allows you to still use reasonable names for files, without the person browing your site having to know if the resources are implemented as static HTML, JSPs, PHP, CGI scripts, etc. The advantage of this approach over the previous one is that you don’t have to create as many directories.

The bigger change to the MLS website was to revamp the UI for the entire website, including the live scores page. That change completely broke SopccerPhone. I grab the team names, scores, etc. by parsing the HTML using regular expressions in a Python CGI script. The new HTML isn’t even remotely close to the old HTML.

I’ve now ported the code for grabbing the team names and the scores to deal with the new HTML layout. Fortunately, the live scores page has a control that lets you easily and quickly see scores from previous weeks. That, in and of itself, is awesome. I had wanted to enhance SoccerPhone so you cold retrieve previous weeks’ scores. The other reason it is cool is that I don’t have to wait until the games start tomorrow to do most of the work on grabbing the game time and the scoring details.

No, I’m not asking you to vote for SoccerPhone for President in 2004. I’m just letting anyone who cares know that SoccerPhone is working this year without me having to make any changes to the code. Fortunately, the people running the MLS website didn’t make any significant changes to the HTML code on the live scores page. In case you are wondering what any of this means:

SoccerPhone is a free, automated service that provides live Major League Soccer scores by phone.

I wrote this application because I wanted to have remote access to updated MLS scores, I wanted to learn how to create VoiceXML applications, and I wanted to learn how to code in Python.

I just uploaded the 0.3 release of SoccerPhone to the SourceForge project site. This is a minor release. The only changes I made were to accomodate recent changes to the MLSNet.com live scores page. Unfortunately, they have been changing fundamental aspects of their HTML markup nearly every week. Sometimes I get lucky and their changes don’t break my code, but too often they do.

While I would love it if MLSNet.com offered an XML feed, perhaps as SportsML, I would be happy if they used CSS more extensively to separate content from presentation. Removing the presentation markup and using meaningful tags to indicate structure would make my life a lot easier. Although they do this in a couple places, in many places they are still using a class called smtext to present unrelated content as small text. Also, the table-based page structure is a nightmare to parse and to understand.

While porting SoccerPhone from TellMe to BeVocal, I ran into a couple differences between the two as a development environment and a deployment environment.

Porting Code from TellMe to BeVocal

The porting process went pretty quickly. Fortunately, the Python CGI scripts didn’t have to change. Three cheers for standards and for application communication via XML over HTTP.

VoiceXML Changes

• Add DTD DOCTYPE to all vxml files so that VoiceXML syntax checker can check for well-formedness
• Must use the BeVocal DTD if using any BeVocal extensions, like the data tag or bevocal:foreach tag
• audio tag must have an attribute like src.
• break tags must be inside prompt tags

Although TellMe also offers a VXML syntax checker, it assumes you are using their DTD. I haven’t tried it with a different DTD, yet, to see if it would actually use it. I like the fact that BeVocal requires it, since it forced me to identify which parts of my code were non-standard.

TellMe also supports the data tag and the foreach tag. The data tag is really cool, as it allows you to return an XML document from a CGI script or a servlet (anything on the other side of an HTTP GET). I hope it makes it into the VoiceXML 2.1 spec.

Tellme allows you to treat an audio tag like a prompt tag and does not require that break tags be inside a prompt. I think BeVocal’s stricter interpretations of the spec are correct.

Grammar File Changes

Both TellMe and BeVocal support Nuance grammar files. I had already decided that I would switch over to the standard SRGS XML format as part of the move to BeVocal. My grammar file wasn’t that complicated, but the lack of good examples for a simple SRGS XML grammar made it far too arduous. I have posted on the SoccerPhone SourceForge project site the source code for both the TellMe code (GSL grammar) and the BeVocal code (SRGS XML grammar) as part of release 0.2

BeVocal requires an xml:lang attribute for a grammar, even if it is a dtmf grammar for which that tag is ignored. I haven’t read the SRGS spec closely enough to know if this is an error in their implementation or an oddity of the spec. Also, if I had stuck with Nuance grammar files, I would have needed to specify the grammar type as type=”application/x-nuance-gsl” instead of “application/x-gsl”.

JavaScript Changes

Fortunately, the change was simple. The BeVocal JavaScript interpreter didn’t allow the DOM function getElementsByTagname() to take two arguments. The TellMe interpreter let me pass in a second argument, even though I’m pretty sure that was just a mistake on my part. I assume their interpreter just ignored the extra argument. My experience with both SoccerPhone and PhoneBlogger has been that the most painful part of development has been writing JavaScript code to parse XML files.

Development Tools

Both sites have really nice on-line development and debugging tools. Right now, I can’t say that I have a clear favorite. The TellMe seems a little more cohesive, but the BeVocal site seems more up to date. The TellMe development tools (at least the free, online ones) have improved in only a few, minor ways in over a year.

The BeVocal Vocal Debugger looked pretty cool, but I didn’t spend much time with it, as the Trace Tool was sufficient for me to find all the problems.

Text To Speech

TellMe is the big winner here for using AT&T Natural Voices. It is far superior to whatever BeVocal is using. In addition to the superior sound quality and accuracy, the TTS engine on Tellme is better at guessing context. The best example is “minute”. Let’s say a game is in progress in the 47th minute. After reading the score, I have SoccerPhone say “minute 47″. BeVocal’s TTS engine pronounces it as “my-nyewt”, as if it were something small. TellMe’s TTS engine pronounces it correctly.

I just published the 0.2 release of SoccerPhone to the SourceForge project site. The main two features of this release are:

1. Support for the 2003 version of the MLS live scores page
2. Support for BeVocal as well as TellMe as a VoiceXML gateway

I had always wanted to port SoccerPhone to another VoiceXML Gateway, but never had a strong enough need to prioritize that activity over other critical activities, like going to actual soccer matches. Well, that changed when TellMe dropped support for application extensions. I tried out Voxeo as well, but ran into a lot of problems just trying to get a simple VoiceXML application working.

After TellMe dropped support for free VoiceXML application extensions on their public service, I didn’t have an easy way to make SoccerPhone available to others. The only way to reach it on TellMe now is to call in with my developer ID and password.

Fortunately, BeVocal allows you to make an application available without having to give out your password. So, to once again hear live Major League Soccer scores:

1. Dial 1-877-33-VOCAL (877-338-6225)
2. When prompted for a PIN, enter or say 7818
3. When prompted for a userid, enter or say 5301860

Unfortunately, that’s a lot of digits to enter, but that’s the cost of free.

Bad news for my free, public SoccerPhone service, which ran as a TellMe Extension. I received the following email from TellMe today:

VoiceXML Developer,
Tellme has made many investments in VoiceXML over the past four years.
One of these investments was in the Extensions program, with the goal
of making VoiceXML a more utilized public standard. Now with VoiceXML
well on its way to standardization in the W3C and with hundreds of
thousands of VoiceXML applications in production, it is clear that
investment has paid off. It is time for us to retire the Extensions
program and invest in other areas. As of Wednesday, April 9th we will
no longer host Extensions on 1-800-555-TELL or
http://studio.tellme.com. Developers can continue to build VoiceXML
applications on Tellme Studio.
Thank you for your individual contribution in making VoiceXML the most
widely-used and successful voice standard in the world.
The Tellme Development Team

Fortunately, it looks like TellMe will still support developer level access (i.e., you need the admin password) to a VoiceXML application, which should be sufficient for most deployments of PhoneBlogger. I’ll now have to look into BeVocal and HeyAnita, although a quick scan of their websites doesn’t suggest that they provide a service similar to TellMe Extensions.

Although I will miss it, this was one of the last remaining relics of the dotcom era. While Extensions got TellMe a decent amount of good PR, I imagine it cost them quite a bit of money to host it, especially when you consider the time that employees were putting into administering a free, hosted service as opposed to one of their services that generates revenue.

I just wish they would have kept it, but without a toll-free number. A lot of people with cellphones have nationwide long distance included in their plan, so TellMe was paying toll charges for nothing. Or, at least I think most people choose the long distance plans. If they don’t, they should. I very rarely make a long distance call from my house anymore.

Eric Snowdeal indicates on his ex machina that he has run into the same problem.

Okay, so I’ve now updated SoccerPhone to work with the new HTML layout for the live scores page at the MLS website. The cool thing is that I only had to update a few methods in the Python CGI scripts, mostly the regular expressions that I use to locate the game time, scores, elapsed time, and goal scorers. I also updated and enhanced the PyUnit automated tests that I use to convince myself that subsequent changes haven’t broken anything.

I didn’t make any changes to the VoiceXML code. The format of the XML document that the Python code sends back stayed the same, even though the source of the data changed quite a bit.

Although I like the new look of the MLS scores page and I like the fact that it updates every 3 minutes instead of 5, there are two changes from last year that I really don’t like. And not just because they are a pain in the ass for my program.

First, the end of a game is not indicated by ‘Final’. Some games ended with the time at 90:00, but one ended at 91:00. When I first checked the scores page, I thought I was catching a couple games in injury time. After they didn’t update for a couple minutes, I realized that they must be over.

Second, the scoring details are littered with the cumulative number of goals or assists for that person. If I want to know season-long stats, I’ll go look at the statistics page.