It was 5 AM, and we were waiting for the end of the world.
I waited in the early morning hours in a set of derelict cubicles with a coworker. Outside, a dozen generators or more ringed the building, with rivers of black cable (two inches across apiece) running between the trailers and enclosures into the building. Just like the aqua OM3 and the yellow singlemode inside, really, but exponentially more deadly and powerful.
The transfer switches, jury-rigged in, were waiting. The fuel trucks were late. The network hummed along in the dark of predawn, grey under power-saving lights, grey and flickering green with the healthy indicators of ports alive with hundreds of gigabits of traffic pulsing as bits of red-lit data down the network.
Engineers paced back and forth between our cubes and the temporary officing, calling for fuel trucks late into arrival. Is it go time? It is five AM. No, we must wait for fuel. Are we good to transfer mechanical? No. The maintenance window is slipping, we only have so long. Everyone else is onsite. Everyone is ready. We must transfer soon, we have hours of diesel left to go. Soon. It must be soon.
We, the NOClings, go outside into the predawn air, walking the riverbeds of inert cable, stepping nonchalantly onto rubber-wrapped copper capable of vaporizing us in an instant. Past the generators. To my coworker, I express the semi-serious hope that something will explode. He laughs. We have seen too much downtime, too many apocalypses to take this, a single building, in any way seriously.
We pace the length of the building, this football-sized, monolithic slab of former factory. Old docks gape like mouths, spilling out noodles of electrical conduit and cable. The trees lean over the fences. On the edge of the sky, it begins to shade to daylight, hinting at it, even as we slip into another world where anything can go wrong.
At the far end of the building, the first of the generators roars to life. Mechanical has been cut over from street power. The engineers are monitoring. The procedure has been tested. Nothing goes wrong. Disappointed, we return to the interior, to our half-sawed, half-disassembled pieces of metal and plastic and shitty upholstery. Behind us, one by one, 5 and 10 megawatt generators, CAT's finest, scream to life. Another day, another dollar.
At 7, 8 AM, street power will drop. Everything must continue. Everything must be transferred away, controlled. Everything must be ready. Engineers to watch all equipment. Technicians to watch all servers. Singapore and San Francisco are online and waiting for the worst case scenario as megawatts drop away and leave us screaming in the dark.
But no, so on and down the line. Data room by data room, the lights flicker for mere seconds, the UPS banks catching the servers but not the fluorescents. Das blinkenlights twinkle serenely like Christmas trees out of season. The drive platters continue to spin out like prayer wheels animated by ghosts in their metal and plastic casings.
Minutes pass. Room one. Room two. All good. The chillers are to manual, not handling the sudden drop in power as well as expected. The air handlers are howling less happily. In hand, the engineers have them in hand. Proceed. Over. Roger, Wilco.
And then, there is a sound like an implosion as they move to the fourth room, a soundless BAM. The office goes dark. The engineers go berserk. Desktops spin down. Networking drops, laptop docks fail, monitors darken: there is no UPS for the front office, and the load has not been taken up by generator. Coworker and I rise slowly, not panicked. A tiny bubble of savage joy blooms in my heart. The facility engineers are louder, yelling. Where is the switch, the switch. When will it be back online. The key, please, the key. Yes, #3. Open the closet. The faithful front office 3750s are spinning back up with the restore of office power.
"Still booting," we report. I crow: while the office switch is plugged into the wall and out of service, my desktop is on a small consumer-grade UPS the size of a small suitcase. My own personal 275 day uptime will not falter. The office network is not nearly so lucky. The 3750s blink once or twice as they begin stack master elections and begin to consider whether or not they'll be functioning sometime in the next hour or so Someone asks about networking elsewhere. They must get back online, update the ticket, this is a disaster. Can't you make it go faster?
No. As the facility engineers rush to find 3G modems at the bottom of their backpacks, to find wireless elsewhere in the building, the switches go green, then orange across ports, and alive, blinking green activity across connected ports. Alive and kicking once again. Elsewhere, the servers sing on, undisturbed by the drop to the office. Uptime is preserved, if only in the production networks.
The sun comes up outside, the generators howl, triumphant. It's 7AM and the building is completely transferred away from street power. It will remain there for ten more hours.