Release Night

 | May 7, 2009 6:34 AM

I was going through my old work email and I found this email I sent on Friday, August 28, 2008, the night we released the new Upromise Online Mall.  (The rumors about metric and the Y2K bug aren’t true but everything else is.)

These fixes were first tested by hot swapping the files in because Perforce was down.  During this time I was mercury poisoned by a broken CFL, there was a car accident, a pizza arrived upside down, Perforce went offline for several hours, Outlook decided to slow down, and there are rumors that everything is switching to metric and that there is a Y2K bug that happens at midnight tonight.

The email was sent at 9:54 PM PDT which does not seem that late until one considers that the team I was working with is located in Newton, MA.  The rest of the team was wide awake and it was almost 1 AM their time.  Some of the team didn’t end up going to sleep until 3 AM.

This is somewhat typical of releases or upgrades at major websites.  We try to test everything but when we go live several issues will suddenly appear.

This particular release though was an unusually event filled night.  Our configuration management system went down and our email server became incredibly slow making communication almost impossible (we moved to AIM).  The manager had to leave because his wife got into a car accident.  Fortunately she was okay and the manager got back online in a few hours.  Even the pizza that was ordered for the team arrived upside down, a bad omen.

To top it off Ji Seon knocked over a lamp in our living room and its CFL bulb broke, sending mercury dust around the living room.  It was like a toxic waste emergency.  We cleared everyone out, I put on a mask and started cleaning up, Ji Seon went on the internet to research what to do and even called the doctor who reassured us the amount of mercury was too small to be a health risk.  After cleaning up the best I could I opened all the windows and kept the kids upstairs.  I vowed that night never to use CFL’s again but I have since revised to that not using CFL’s in table lamps.  I can’t wait until LED lights become more common place.

It’s nice to look back at such a stress filled night with a smile.  I’m glad I am paid by the hour. 🙂

No Responses to “Release Night”

Care to comment?