In the aftermath of Hurricane Sandy last Fall, one story that stood out in the IT community was that of besieged hosting provider Peer 1: both as a story of grit and determination and a canonical example of just how dicey disaster recovery can be.
The problem sounds impossible: Peer 1’s generators on the roof of their 75 Broad Street location in lower Manhattan are normally fed fuel from basement storage tanks. When basement flooding knocked out the pumps to get the fuel up 18 stories to the roof, the only fuel left was five hours’ worth in the generator’s holding tanks.
Faced with barely any fuel and no way to get fuel efficiently to the generators up top, Peer 1 staffers considered the inevitable: cleanly shutting down the servers that hosted services like Trello, FogBugz and Copilot.
But Peer 1 and its datacenter tenants Fog Creek Software and Squarespace would refuse to give up, finally devising a bucket brigade to transport diesel fuel up 18 flights of stairs to the roof. For 72 hours.
A new documentary short details the process of how Peer 1 dealt with the very real and very deadly storm. Sometimes IT can get down to not just what you know, but how much you believe in what you are doing.
Lead image courtesy of Shutterstock.
