Mock grid, real threats: DARPA borrows an island for a cyberattack drill

Over the last 120 years, Plum Island, a forbidding swath of sand off Long Island, has been at the edge of U.S national security. The island housed gun batteries during the Spanish-American War, a torpedo storage facility during the First World War, and in recent decades it has been the government’s home for studying animal-borne diseases.

In the first week of November, the military found yet another way for Plum Island to serve as a guinea pig. This time, though, it was for a decidedly 21st-century threat: cyberattacks that could hamstring a power grid.

The fictional scenario saw contractors with the Pentagon’s R&D arm — the Defense Advanced Research Projects Agency (DARPA) — team up with engineers from prominent utilities to try to restore power that had been out for weeks following a hypothetical cyberattack.

Their tall task: use a generator to gradually restart the power system, substation by substation — a process known as “black start” — all while coping with renewed threats to the grid. Two mock utilities worked to synchronize their power systems, and DARPA contractors connected cyberdefense tools to the grid to see how they fared in detecting network anomalies.

While Russian hackers have treated the Ukrainian power grid as their laboratory, honing their tradecraft in disruptive attacks in 2015 and 2016, Plum Island is the U.S. government’s own lab for defending against such attacks.

The exercise is “an opportunity for these utilities to get familiar with what a cyberattack would look like before they see one in the real world,” DARPA program manager Walter Weiss told reporters Nov. 6 after a tour of Plum Island, which is beyond the eastern tip of Long Island. Outside the trailer-cum-briefing room, the wind howled past a utility command center as defense teams continued to parry simulated attacks from red-team hackers.

As in the 2015 Ukraine attack, in which a local utility was hit by a denial-of-service attack to keep outages from being reported, the defensive teams in the DARPA drill were thrown curveballs that complicated the recovery process.

Later during that penultimate day of the exercise, the exercise organizers would unleash a data-wiping attack that forced participants to rebuild a substantial portion of network configurations.

“Unless they’re locking things down with those network tools as they make the fixes, they’re inviting even more problems by energizing those links,” Weiss said, relishing his ability to throw digital obstacles in the defenders’ paths.

Searching for ‘ground truth’

The exercise is part of DARPA’s Rapid Attack Detection, Isolation and Characterization Systems (RADICS), a three-year-old program fueled by an initial $77 million in spending. The drill also fell under the Department of Energy’s Liberty Eclipse exercise program, and multiple other organizations were involved. The National Guard provided radio equipment and the Department of Homeland Security facilitated access to Plum Island.

Engineer Gary Seifert explains the layout of the grid exercise. (photo courtesy of DARPA)

Using power assets from Long Island Power Authority, the exercise planners created an isolated grid of rugged substations, voltage and current transformers, and overhead power lines.  The substations were nestled in trailers that dotted the muddy landscape like electric cattle. Inside the trailers, green and red lights told engineers how the compact “substation in a box” was interacting with the utilities.

The experimental grid was built on the back of a testbed — an amalgam of hardware and software — developed at the University of Illinois. The testbed allowed researchers to experiment with networking tools under a variety of grid conditions, explained Tim Yardley, a senior researcher at the University of Illinois.

“It’s not just a bunch of academics that have put something together,” he said aboard a bus that criss-crossed the island. The testbed, Yardley added, goes beyond the actual gear to include “the knowledge that people bring to the table, the issues that need to be solved, and the science that has to pull it altogether.”

The scenario began with emergency orders from the Energy secretary: Utility B must keep a critical power asset within the system on at all costs. The critical asset, a dilapidated building that was a military barrack during the Second World War, had inflatable tubes overhead that flapped in the wind if the power was running.

Utility A had to come to Utility B’s rescue and build a “crank path” of power connectivity via the substations. In these severe blackout conditions, restoring complete power can only be done by gradually synchronizing and stabilizing the two grids. As reporters visited the island, the teams had accomplished the former task but were struggling with the latter.

In the real world, utilities have mutual assistance agreements through which they lend each other personnel and resources in times of crisis, such as destructive hurricanes. That concept is increasingly being applied to preparing for cyberattacks. As power came back online at Plum Island, the exercise’s defense teams continued to find simulated malicious activity on their networks, and a few days into the drill, the two utilities began to share threat reports with each other.

DARPA brought tools it has funded through the RADICS program to test, including hardware and software for network isolation and situational awareness. There was automated-intelligence tech that ingests data from across a power network and produces a model for distinguishing normal and anomalous activity. Another tool allows emergency communication between a substation and a control center.

“We want to find the limits of the tools,” Weiss said, “so we drive them to the point where we see how far they get, and then we beat them back down.”

Gary Seifert, an engineer advising the Plum Island exercise, made the rounds from substation to substation on the island, trekking through the rain to check sensor readings.

“Of the 40 or 50 breakers they had, some of them were compromised and they don’t communicate well,” said Seifert, whose gravelly voice betrays his three decades of experience in the field.

Sometimes there were false alarms — normal grid activity that researchers took for a cyberthreat to the power system. As Weiss put it: “If you take a bunch of researchers [and] stick them on an island like this, they’re going to get pretty paranoid.”

But in other cases, the remote monitoring equipment helped to restore confidence in the power system, giving researchers clear readings of power flow. Seifert called that state of technology-enabled trust “ground truth” — a tranquil equilibrium toward which the grid operators were toiling.

They’ll be back next year

As reporters left Plum Island Nov. 6, there was still another day left in the exercise, with defense teams methodically trying to stabilize the grid. But even after the sun set the following day, the work was far from over.

The exercise, after all, comes as hackers continue to show a strong interest in targeting the industrial control systems that support power grids. Cybersecurity researchers said last month that a subgroup of the broad set of Russian hackers who struck the Ukrainian grid has recently targeted energy companies in Poland and Ukraine.

While U.S. electric infrastructure has not suffered the disruptive attacks that occurred in Ukraine, American utilities continue to fortify their defenses in recognition of the threat. That work is supported by rigorous exercises like the one on Plum Island, which will evolve for its next iteration.

Weiss said he hopes the next RADICS exercise on the island will involve cyber specialists from utilities (rather than just grid operators or engineers) working alongside the DARPA researchers.

“We want to be back here in May or June and make it more complex and more difficult,” Weiss said. “As the tools mature, so does the test range people have to work on.”