A perfect storm of a nearby car accident, a phased loss of power, a fried air conditioning system motor, a tripped circuit breaker, and a resulting power failure at headquarters made for a challenging Monday on May 15 for our IT Group. IT responded quickly and efficiently to handle all the issues, and a backup generator installed on Saturday, May 13, worked well under pressure despite the challenges.
The issues began shortly after a car crashed into a utility pole on 71st Street not far from our flagship location shortly after 7 a.m. on May 15. A fire blew out a nearby Indianapolis Power and Light transformer, which in turn caused a phased loss of power in multiple buildings near the accident scene, including the 7260 and 7340 buildings, said IT Director Adam Martin. The emergency generator, installed to supply backup power to our main IT server room and IT networking closet, did the job it was designed to do — keep our computing systems running in the event of an unforeseen power outage. This ensures that Indianapolis and our satellite locations stay connected to the network, as well as people working remotely from other locations if a power outage occurs to our main building.
However, the phased power loss fried the motors of the air conditioning unit, which meant we needed to call HVAC repair. The repair team responded quickly; but they neglected to shut off power to an electrical panel connected to the air conditioning that cools the server room. This tripped the main circuit breaker to the generator and the UPS (uninterruptible power supply), which resulted in a power failure that caused the network to go down despite the extra measures in place.
Martin said our IT Group responded well to the situation, getting all systems up and running within about 30 minutes. “Our team did an awesome job in getting us back online,” Martin said. “Our planned shutdown for May 13 actually helped us because it gave us an opportunity to practice our playbook should we need it, which we did just two days later.”
Stay tuned for additional steps that IT will take later this year to ensure that our IT assets remain safe during outages.