The extra time at the end of my talk was also nicely filled in by a lot of good questions. I ended up having to rush off the stage for the next talk.

It was also cool that I was speaking to about 100 or so people in the Gold Room at the beautiful Palace Hotel. The Castlight Xmas party was in the same room two years ago.

As always, 10gen put on another excellent show. Meghan and her team are awesome. I’ve attended several of these events and they get better each time.

Here are the slides I presented:

I previously spoke about logging to MongoDB at Mongo SV a few years ago in Mountain View and have attended several other MongoDB conferences and events. The 10Gen team has built a fantastic product and has an amazing events team. Their conferences are very well run, economical, educational and I highly recommend attending them.

Lean In also has a lot to offer to male readers. In addition to advice to men in senior positions on how to ensure women and men have equal opportunities, there’s just a lot of great career advice that applies to most everyone.

I had initially wondered about the amazing detail of the footnotes and how she had time to do that much research. Then I discovered, as I should have expected, that she had a lead researcher. However, Sandberg did provide very detailed acknowledgements giving credit where credit was due. And I think her presentations and Q&A sessions show that she’s not just putting her name on someone else’s work.

One downside of using a 64-bit JVM is that the object pointers used by the JVM to reference objects would normally need to increase from 4 bytes (32 bits) to 8 bytes (64 bits). But, if the heap is under 32 GB, the JVM can take a shortcut because it knows that the offset will always fit in 4 bytes.

The JVM argument -XX:+UseCompressedOops was added to force the JVM to use compressed ordinary object pointers whenever possible. In Java 6 Update 23, the JVM was updated to enable Compressed Oops by default. Compressed Oops works very well for heaps up to about 26 GBs, but can still be advantageous for larger heaps.

However, I found that with Java 6 Update 24, Compressed Oops are not used at times when they could be, even if you specify -XX:+UseCompressedOops.

Specifically, I was using a 31.7 GB heap and my service was unexpectedly running out of memory. On one of our other systems, the heap for the same service reached only 26 GB after its large internal cache was fully loaded. After a lot of investigation that weekend, I discovered that the working system was on Java 6 Update 26. I downgraded it to Update 24 and easily reproduced the problem.

I had previously done a lot of testing to see what our penalty was for crossing the 32 GB heap boundary and found that it was about 8 GB. That’s actually not too bad, as general estimates I had heard from others ranged from 30-50%. This is probably due to the fact the objects we are caching mostly have only primitive data types as fields.

I highly recommend the Everything I Ever Learned About JVM Performance Tuning @Twitter slides and presentation for more info on JVM tuning. Or read Andrew’s summary of Attila’s talk.

By reducing the cache size on the production server enough to not blow out the heap, I confirmed that the heap size when using 1.6.0.24 was about 8 GB higher than when using 1.6.0.26. JVM, meet smoking gun.

The release notes for 1.6.0.26 references a compressedoops related bug fix that may have resulted in this feature now working as described. I read through all of the many bugs fixed in that release and it seems to be the most likely candidate.

I wanted to capture a couple of hours worth of all queries in the slow query log so I could analyze them with pt-query-digest from the excellent Percona Toolkit. So, I used

set global long_query_time=0;

After a couple of hours I had a lot of data. I discovered later that I was actually missing some of the early queries.

Then I set long_query_time back to 1 so only queries longer than 1 second would be logged. To my initial amazement, lots of very short queries continued to be logged to the slow query log.

A little research turned up the fact that the long_query_time session variable for a connection is initialized from the global variable only when a connection is opened. So, any connections that were open when I set long_query_time to 0 continued on as if it were still set to 1. Therefore I missed capturing those queries.

Worse, though, is that some of our code uses connection pools that can keep alive connections for a few hours. Short queries on those connections continued to be written to the slow query log after I set long_query_time = 1. Fortunately, I was tailing the slow query log, so I noticed this before it got too big. At this point, you can temporarily disable slow query logging, let it go and hope those connections don’t last too much longer, or go all club and hammer on the connections and kill them and hope your code handles it gracefully. The last one generally isn’t so great of an idea for a production app.

As I mentioned above, my slow query log capture didn’t include sub-second queries from connections that were open when I set long_query_time to 0. That would obviously affect the results of my analysis by leaving out the queries from long-lived connections.

So, if you’re doing something like this, you should capture data for a long enough time to offset this factor or at least throw out the data from the earlier parts of the log. You should also check out the open connections with show processlist; before changing long_query_time.

Back in the 70′s, subliminal messaging was a controversial topic. I remember running across a book from that same year called Subliminal Persuasion that had a lot of images from advertising and movies with supposedly embedded suggestive words and images, primarily of a sexual nature. Check out the Subliminal Manipulation blog if you don’t sex believe me.

Take a closer look at the hair of the guy on the Mixology cover. Now, think about other parts of a man’s body. Good luck getting this image out of your head (no pun intended) anytime soon. I’m really, really sorry.

Next up we’ve got a dude confidently sporting a pink suit. The previous sentence is the only known sentence on the internet including the words dude and pink suit, but not the word pimp. Ignore the faint yellow polka-dots. I double-checked the pamphlet and they must be a scanning artifact. I was so hoping they weren’t, though. I think my scanner understandably puked on the image.

Almost everyone knows his Zodiac sign today. But few have any real knowledge of astrology.

Intent of astrology data herein is simply to inform, not to advise. Therefore any personal application is the individual’s responsibility.

Check out Mr. quilted pants on the left. What grandmother wouldn’t want to see her granddaughter coming home with a nice boy wearing a handmade quilt? OK, besides any grandmother with something against hobos. Those pants are so appalling that I almost didn’t notice the crazy blue plaid suit in the back. He’s channeling Rodney Dangerfield from Caddyshack, but coming up well short. Powder blue cardigan boy looks positively normal in this photo.

If your party kilt is at the drycleaner, a full tartan suit is always a great substitute. It’s a little hard to see, but, yes, those are matching pants. Fortunately, I don’t think it’s the royal Stewart tartan. Too bad his promiscuous plaid partner up front isn’t in a matching tartan. I can’t identify the tartans for certain due to the cumulative retinal scarring, but I’m suspecting they’re variants of the Montgomery Ward tartan.

If you’re daring and desperate for the full pamphlet in a high enough resolution to actually read it, download the 2 MB zipfile.

When I first glanced at the image of the remote control, I thought the bottom left button said “Lasers”. Now, that would be freaking awesome. Whether as a laser light show to accompany the event or as a modern alternative to the incineration of your contributions, I’m all for it. And surely a couple frickin’ lasers would come in handy when kicking some Neorest butt.

However, if you are returning a very large ResultSet, the client will have to allocate a lot of memory on the heap. If you end up accessing each row to create an object from the data, then you will need enough heap space for the entire ResultSet plus all of the objects you instantiate.

The driver documentation explains how to force the driver to stream the ResultSet row-by-row.

The first catch is that you must be using a regular Statement object, not a PreparedStatement.

The documentation says you need to add the following non-intuitive code before executing the query:

stmt = conn.createStatement(java.sql.ResultSet.TYPE_FORWARD_ONLY,
              java.sql.ResultSet.CONCUR_READ_ONLY);
stmt.setFetchSize(Integer.MIN_VALUE);

though you can actually just use conn.createStatement() since TYPE_FORWARD_ONLY and CONCUR_READ_ONLY are the defaults.

There are a couple caveats in the documentation, though they are fairly obvious. You should process the ResultSet as quickly as possible, since locks will be held as long as the statement (and any transaction it is in) is open.

In addition to being non-intuitive, setting the fetch size to Integer.MIN_VALUE might cause unexpected results if you run your code against a database server other than MySQL.

If you’re willing to go all out in committing to MySQL, you can cast the return value of createStatement() to com.mysql.jdbc.Statement.StatementImpl and then call enableStreamingResults(). That will, at least, make the behavior of your code more obvious.

At work I needed to cache a lot of data from a couple of tables. Using the default behavior caused the heap to grow to over 12.5 GB. That made for trouble when running on my 8 GB laptop. By switching to streaming the ResultSet, the heap maxed out at only 5 GB.

Fast forward nearly ten years and not only have the tools and libraries come a long way, but there are many more free or inexpensive hosted services that simplify building a tool/service like PhoneBlogger. Instead of hosting the application code on a shared hosting site, I can now build and deploy on Google App Engine. Though scalability is not an issue for my personal use of PhoneBlogger, if it were turned into a public service, App Engine would make scaling much simpler and more economical. App Engine also makes deployment a snap, though with a small amount of work, so would Fabric. For my PhoneBlogger rewrite, I decided to use App Engine.

In the original version of PhoneBlogger, I coded a bunch of static VoiceXML and JavaScript for managing the telephone interaction with a caller. At the time, three of the most prominent services for VoiceXML developers were Tellme (now owned by Microsoft), BeVocal (now owned by Nuance) and Voxeo (still independent). I had to write slightly different code for Tellme and BeVocal, but the differences weren’t that significant. I think it would have been pretty simple to port to Voxeo, as well. Improved support of VoiceXML 2 would now likely allow me to use the same code on each platform.

While VoiceXML is still a great option for building speech apps, a couple of new services bring you simple APIs for building speech or DTMF (touchtone) applications, at the cost of portability. This time around I’ve started with Twilio. I very quickly turned a Python/GAE example from the Twilio website into a DTMF app for tweeting by phone. Although speech recognition allows you to build much more complex and natural applications, many simple applications can be built quickly and easily with just support for pressing keys to provide input. PhoneBlogger falls into that category, for now.

One very convenient thing about Twilio is that I can use their platform to capture and host recordings in a format that is simple to play back in a web browser. If I were really concerned about longevity of the recordings I could easily retrieve them and store them elsewhere, but I’m okay with keeping them on Twilio servers for now. That’s an easy enhancement to add later. The biggest downside for tweeting the Twilio links is that the Twilio recording URLs are ginormous. Fortunately, the goo.gl URL shortener made quick work of that problem.

I’m also going to take a look at porting my code to Tropo, which is a service offered by Voxeo. Tropo is built on Voxeo’s Prophecy platform and offers speech recognition as an option.

I decided to begin the rewrite by first supporting tweeting by phone. Twitter offers a great API, which is made even simpler by libraries like Tweepy. I highly recommend first checking out the OAuth support in any library for Twitter you might consider using. OAuth can be a complex beast, but libraries like Tweepy make it almost trivial.

The original PhoneBlogger source code and a couple iterations of it are available on SourceForge. I wasn’t particularly interested in learning about CVS at the time, so I just uploaded tarballs of all the code. While SourceForge has improved a lot, I’ve become more of a fan of GitHub. Google Code, LaunchPad and BitBucket are also great options. I started using LaunchPad when working on a Java library for Gearman, but then set up a couple of repos on GitHub when I started working on Log4mongo-Java. I’m much happier with Git, Bazaar and Mercurial than Subversion and CVS (Caveman Versioning System). I’ve already started posting code for the new phoneblogger project on GitHub.

As of now, the new version of PhoneBlogger supports tweeting by phone. All the code is on GitHub, along with a README file with the basic steps to set it up for yourself. In an upcoming blog post I’ll walk through those steps in a little more detail.