reesesWeb

About 24 hours ago, we launched, bringing to a semi-conclusion a few months of pretty intense work. The site is pretty fast, and we had some excitement at the last minute, so we didn't sit around bored after 2am. We went live at around 1:30AM, and discovered rather quickly after that the application servers were "randomly" crashing. Over the next few hours, we tried to narrow down the issue, based on tying webserver logs from the three load-balanced webservers (pure round-robin, with session affinity handled by the app server tier) to the eight application server processes.

Anyway, at around 7am, we found the issue, which was pretty strange. One of the JSP pages was miscompiled into bytecode, and was corrupt in such a way that, when executed, it would immediately kill the app server. No logs, nothing. At first, we thought it was something in the code, but doing a quick ls -l on the files showed that they differed in size between servers. By a lot. Now, I know there are environment-related issues preventing two files, compiled on separate machines, from being identical, but a 20% difference in size is telling. :-)

This miscompilation took place with the app servers were started up with Windows, rather than manually after the OS stabilised. We were able to bring down the servers, kill the JSP pagebuild tree, and bring the servers back up without incident.

I had brought some DVDs to work in case it became boring. While I didn't get to watch Bring It On, I definitely wasn't bored.

Hungry, though, and it's just as hard to find decent food in the Flatiron area at 4am as it is anywhere else in the US.

Bookmark/Search this post with:

delicious | digg | reddit | technorati