AI Data Centers: Fallen Line Exposes a Growing Crisis. Here's the Solution.

A recent power line failure near Washington, DC, led to an unusually prolonged grid recovery, lasting over 10 minutes, significantly longer than the typical few seconds. This delay was attributed to the near-simultaneous disconnection of more than 3 gigawatts of data center load from the power supply.

The incident resulted in a substantial voltage spike across the PJM grid, extending from Northern Virginia to Chicago, according to data compiled by Ting Labs, a startup leveraging an IoT sensor network embedded in electrical sockets.

While a full blackout was averted, lights across the affected region flickered, underscoring the considerable impact data centers can have on grid stability. Experts anticipate such events will become increasingly common.

Northern Virginia, situated within PJM’s operational territory, is recognized globally for its unparalleled concentration of data centers.

“It’s the canary in the coal mine,” stated Ricardo de Azevedo, CTO at ON.Energy, in an interview with TechCrunch. He further noted that these types of occurrences, involving large loads like data centers, are “happening more and more.”

This event mirrors a similar incident on the PJM grid two years prior and could portend more severe disruptions if data centers are not engineered to manage power supply interruptions more gracefully. The PJM Interconnection, which oversees grids from New Jersey to Illinois, serves 67 million customers, establishing it as the largest grid operator in the United States.

Upon the power line failure this week, data centers were triggered to switch to backup power, causing approximately 3.1 gigawatts of load to vanish within about 30 seconds, based on PJM data. Although the grid initially showed signs of recovery, additional loads subsequently dropped off. At its peak, PJM’s grid experienced an excess of 3.49 gigawatts of electricity, requiring another 11 minutes to stabilize. The disconnected data centers represented roughly 3% of PJM’s total demand at that time, as reported by Reuters.

While a small percentage might seem negligible, the electrical grid necessitates operating in a state of near-perfect equilibrium, with supply and demand meticulously balanced. Any significant deviation can lead to voltage sags or spikes. The grid and connected devices can endure minor fluctuations, but excessive variations will activate failsafes within the grid or individual facilities, leading to disconnections.

When data centers in Northern Virginia detected the voltage fluctuation caused by the downed power line, they transitioned to backup power, effectively removing their demand from the grid. As more data centers followed suit, an escalating amount of load was shed. What began as a relatively minor reduction in supply quickly amplified into a significant drop in demand, resulting in a supply surge and the observable flickering of lights.

Most data centers are programmed for instantaneous decision-making, and those that disconnected this week were no exception. Upon sensing the voltage dip, they all initiated disconnection within seconds of each other, explained Ali Zain Banatwala, senior market models specialist at the Independent Electricity System Operator, to TechCrunch.

“We need to figure a way for these loads that are located next to each other to sequentially either disconnect or reconnect,” he emphasized. Implementing a more structured process would empower grid operators to develop more robust pre-emptive procedures.

Alternatively, data centers could be designed to absorb disruptions rather than react by disconnecting. ON.Energy, a pioneering startup, is actively developing a solution to enable data centers — and the broader grid — to withstand such events.

The company has engineered an uninterruptible power supply solution capable of protecting an entire data center campus, encompassing not just servers but also crucial infrastructure like chillers. This system effectively buffers the data center behind a robust bank of batteries integrated with sophisticated power conversion equipment. Consequently, the grid perceives a single, consistent, and stable load, rather than the fluctuating peaks and valleys from individual components within the data center. ON.Energy’s technology also allows data centers to dynamically scale computing workloads, including intensive AI training, without imposing stress on the grid.

Crucially, this system enables data centers to actively absorb power fluctuations originating from the grid. Instead of disconnecting, ON.Energy’s solution can utilize excess grid power to charge its batteries, and conversely, disburse stored power to servers if grid flow diminishes. Furthermore, it can synchronize with grid commands within milliseconds, thereby preventing the sags or surges that contributed to PJM’s recent issues.

ON.Energy is currently in the process of installing systems totaling 3 gigawatts across four distinct data center campuses, according to de Azevedo.

Grid managers globally have also recognized the gravity of this challenge.

For instance, ERCOT is reportedly moving to mandate that large loads, including data centers, must be capable of “riding through” disruptions, de Azevedo confirmed.

However, time is of the essence. This week’s mass disconnection was twice the scale of a similar event in 2024, when 60 data centers simultaneously disconnected, pulling 1.5 gigawatts of load from the grid. At that time, data centers constituted approximately 6% of PJM’s total load, according to Synapse Energy Economics. Projections indicate this figure could rise to 24% by 2040. Without timely intervention, the situation is poised to significantly worsen.