Updated: When Hurricane Sandy hit 75 Broad Street in Manhattan, forcing Con Ed to cut power and then flooding the basement generators, the most customers could legitimately expect was an orderly shutdown of their equipment. But Peer 1 Hosting’s small local staff — along with friends and customers — went way beyond the call of duty, forming a bucket brigade to deliver gallon upon gallon of diesel fuel up 17 floors to keep the company’s backup generator humming — and their equipment online.
The ongoing effort, flagged for me by Tier1 Research analyst Carl Brooks, is truly super human. “Seventeen flights? These guys don’t just deserve more customers, they deserve a freaking bonus,” Brooks said via email.
The work, which has been ongoing for more than two days now, is led by Mike Mazzei, the company’s head data center manager, and technician Scott Debernardo. They did their due diligence last week as the storm bore down. They arranged for food and water to be brought in and for fuel delivery. All that worked as planned. But, as in any disaster, there were wild cards.
For example, there’s a connection for the fuel trucks to fill the basement tanks, but there is not one to pump diesel up 17 flights of stairs to Peer 1′s other generator. That meant brute force had to be applied. A 55 gallon barrel was set up at the entry point, and another was left at the top of the stairs. The five local Peer 1 staffers, along with friends and customers, schlepped the fuel up by hand in buckets. The generators burn through 40 gallons every hour.
On Monday, there were anywhere from 10 to 15 people working at any given time and this morning there were 35, said Ryan Murphey, VP of data center operations who spoke to me from Peer 1′s San Antonio office.
Anthony Casalena, founder and CEO of Squarespace, a Peer 1 customer, went to the site to ensure a clean shutdown, found the bucket brigade, and ended up helping, as he writes here. As of SquareSpace’s latest update at 10:38 a.m. EDT Tuesday:
“Bucket brigade going strong. We’ve gone through half of our morning fuel delivery and are expecting a truck with 5,000 gallons coming at noon. Potential issues are lack of bucket brigade manpower into the night, and our fuel pumps burning out.”
Update (Nov. 1 at 9:29 a.m. EDT): Fogcreek Software, another customer, is also heavily involved with the diesel brigade effort as outined in its status page. According to Fogcreek fuel pumps are on their way as of Thursday morning but are stuck in gas lines in New Jersey.
also As Tier1′s Brooks pointed out, people who worry about moving workloads to the cloud or third-party provider should take this story to heart. “You should call this story ‘why your data actually is safer in the cloud — cuz your data center people ain’t doing this s**t,’ or ‘Yes, Virginia, you can trust your data center operator.’”
Here’s the other thing. As much as we virtualize and automate and build redundancy, computing — whether it’s “in the cloud” or in your server closet — still depends on such lower-tech things such as a power supply and, just as importantly, human ingenuity.
75 Broad Street photo courtesy of Google Street View.