In the age of cloud computing, it’s an archaic thought: The livelihood of some popular websites currently rests on bucket brigades carrying diesel fuel up multiple flights of stairs just to keep generators running. But that’s the reality as Lower Manhattan struggles with massive power outages and flooded understructures in the wake of Hurricane Sandy.
It’s a decidedly 17th-century solution to a 21st-century problem. But it’s the only way to get fuel to generators that are located in powerless high-rises when elevators are out of service and basement fuel pumps are incapacitated due to flooding.
For site admins at Fog Creek Software in New York, the solution is clear, albeit arduous: When diesel is delivered to their building at 75 Broad Street, the fuel is hand-delivered to the generators on the 17th floor via bucket brigade, giving the online service provider more hours of uptime while basement flooding is cleared and the building awaits power restoration from ConEd. Meanwhile, users of services like Trello, FogBugz and Copilot are basically in a holding pattern, knowing that one missed fuel delivery or downed generator is all that stands between uptime and downtime right now.
[Disclosure: ReadWrite is a Trello user.]
A Too-Common Tale Of Woe
Fog Creek is not alone in its situation. ISPs Peer1 and Internap both maintain facilities in the same building. Fog Creek and fellow ISP tenant Squarespace are working with Peer1 on the bucket brigade to keep servers going. Internap is reporting that since their fuel pumps were swamped, the facility was shut down after the generators’ fuel ran out.
The story is being repeated across the New York Metro area, as data centers switch to back-up power on diesel generators as expected. But what no one counted on was the massive flooding that would compromise fuel systems and generators across the area hit by Sandy.
ISP Datagram was shoved off the grid Monday night when its 33 Whitehall facility had its basement flooded. This shut down high-volume sites like Gawker, Gizmodo and Buzzfeed. Gawker and Gizmodo are still running “emergency” pared-down sites, but Buzzfeed managed to get itself back up on its feet as early as Tuesday.
Clouds Can Help, Too
How was this done? Probably by the one method that every single Web customer in Lower Manhattan should be thinking about just as soon as the lights come on: BuzzFeed got its website on the cloud.
“In a nut, our engineering team moved everything over to Amazon Web Services, with one developer working overnight despite a tree falling through his roof,” wrote BuzzFeed’s Matt Buchanan.
Granted, it’s not easy to just up and replicate your site to a cloud-based system. BuzzFeed’s team was aided by the fact that much of the site’s content was already replicated on Amazon, so they were halfway there.
“You might wonder why, of course, BuzzFeed, Gawker, and others aren’t already all aboard the cloud train, ready to switch to different servers at the drop of a hat,” Buchanan concluded. “The fact is, Amazon cloud service and other services like it weren’t around when BuzzFeed and sites like Gawker and Huffington Post were architected years ago. If the site was built today, the architecture might look a bit more cloud-like than having a huge data center based in downtown Manhattan.”
Hindsight has the benefit of being 20/20, it’s true. But after the waters from Sandy recede, look for a lot more websites to gain a renewed focus on the cloud.
Image courtesy of Shutterstock.