Julie Bort | Oct. 30, 2012, 4:03 PM
Massive flooding from Hurricane Sandy has taken out a bunch of important data centers in New York and New Jersey.
But its also lead to some incredible stories of brotherly love, including this one from New York startup Squarespace.
As Sandy swept into town, data centers Internap and Peer 1, both located at 75 Broad Street in lower Manhattan, were flooded.
Some customers, like Fog Creek Software, had parts of their websites go down.
Peer 1 customer Anthony Casalena, founder and CEO of Squarespace, was determined not to let that happen. Squarespace has tens of thousands of users who have created 1.4 million websites—so that's a big chunk of the Web Casalena has on his hands.
When he heard about the flooding, he headed over to the building, ready to carry replacement fuel up the stairs took the roof in plastic water bottles if that's what it took. A crew of people have been acting as a manual fuel pipe ever since but working that way "is not sustainable" he says. So far, his websites are safe: As of an hour ago, Squarespace was still online.
UPDATE: Casalena contacted Business Insider to say that credit for keeping the data center up should go to the Peer1 folks, particularly the main engineer onsite, Mike Mazzei. There will be a shutdown at some point but he's hoping it will be for a minimal amount of time.
Here's his story, as posted to the Squarespace status blog:
Anthony here. Just wanted to share a personal update now that I have a moment.
At around 9:00 this morning upon hearing about the fuel situation at Peer 1, I decided to head out and see if I could lend a hand. The streets of Manhattan near where I live (Soho—not in the evacuation zones) are in not-so-bad shape right now, but the damage left from the flooding in the evacuation zones is significant and real.
I’m sitting in our datacenter NOC [network operations center] at 75 Broad St. Not that it’s been pointed out to me, but there are beds set up on the tiled floor here from the great team at Peer 1 who stayed to monitor the situation overnight. These guys have incredible commitment to keeping everything running, and it’s great to see.
Normally, power loss would not be a major problem for our datacenter—Peer1 stayed online during the major Manhattan power outage in 2003 that lasted for days, and we preemptively shifted to backup power around 4:00 pm yesterday predicting that Con Edison would be shutting off power in evacuation zones. Given the nature of the flooding, this situation escalated greatly, submerging our reserve fuel in the basement, shutting off the elevators, and damaging the pumps required to get this fuel to the generator on the 17th floor.
My reasons for coming to the datacenter were twofold: one was simply to help and do whatever I could to help us (and the whole building) stay online. The other was to send our systems team the absolute final signal to perform a clean shutdown of our infrastructure should we be moments away from total power loss. Generally, clean shutdowns are preferable to abrupt halts, since code halting in a known state is better than code halting in an unknown state.
We had an initial warning that 10:45 am was going to be the clean shutdown time. To determine how much time we have remaining, engineers are taking readings at particular time intervals to attempt to determine how quickly we are depleting. The tank readings are behaving somewhat erratically, as there is another mechanism replenishing it from a separate fuel header. Some of our recent readings seem more optimistic, but it is impossible to predict how much fuel remains in this header at this time. As of this writing, we have at least 45 minutes left.
Bridges to the island are open right now, and we currently have a fuel truck en route. We have approval from the building to manually carry fuel up in plastic water bottles, and we have a number of our team on-site to carry fuel up the stairs as needed. I do not know if the manual plan will be successful, but we will certainly try.
Unfortunately, I do not have more information on a final resolution to this issue. You should still expect Squarespace to go offline at some point because of the hurricane’s aftermath, but we will do our best to keep that downtime to a minimum. Once we have a reliable stream of fuel to the building, it will go online independently of any other grid issues related to ConEdison and lower Manhattan in general.
We’ll continue to keep you posted. Thank you for your patience.
A