DHS cyber tool finds huge amount of ‘shadow IT’ in U.S. agencies

New cybersecurity tools being deployed across the U.S. government found huge numbers of uncatalogued and unmanaged computer devices connected to federal networks — a phenomenon known as “shadow IT” — that necessitated urgent modifications to many hundreds of millions of dollars’ worth of contracts.

Some departments and agencies had “several hundred percent” more devices on their networks than they expected and the average across government was about 44 percent more, Department of Homeland Security official Kevin Cox said last week at the McAfee Security Through Innovation Summit, hosted by CyberScoop.

“There was something of an ‘oh shit’ moment,” said a person familiar with the discovery, made during the recent rollout of phase one of Continuous Diagnostics and Monitoring tools. CDM is a DHS-funded, government-wide acquisition program that buys and installs cybersecurity tools on U.S. departmental and agency networks.

The tools found every kind of device imaginable on federal networks, this person said, from the expected, like printers and PCs, through televisions to thermostats and other environmental sensors and even “stuff like an Xbox.”

It was “a flashlight … being shone into all the dark corners of shadow IT,” said the person, who asked for anonymity because they were not authorized to publicly discuss these aspects of the program.

CDM is touted by officials as representing a new approach to cybersecurity by the federal government. Some have called it a “paradigm shift,” moving from an approach based on annual or periodic checklists, to one based on realtime visibility into government networks.

As part of the bidding process, before awards were made to four large system integrators as prime contractors, Cox explained, each agency set up “reading rooms” containing data about their network. “The vendors came in to read that data and then put their proposals together,” on the basis of it, he said.

But once a proposal was accepted, an award issued and the tools installed, in many agencies, officials and contractors found “a surprising delta between what was planned for and what was discovered,” said the person familiar with CDM.

Cox explained that the existence of all this IT, unknown to the CIO’s offices who had put together the data for vendor reading room, wasn’t in and of itself a sign of poor security. “It might have been an environment that [the CIO’s office] didn’t have eyes on, it might be a program office that didn’t really report to anyone … all those different organizations [within a department] that might have IT out there … They could have been doing a great job keeping it secure, they just didn’t report it.”

Even so, the presence of all those additional devices on the networks meant that the installation and management of the CDM cybersecurity tools became a much larger job in some agencies than the vendors had anticipated — and much larger than the basis on which they had bid.

The contract “was underscoped, in some agencies severely,” said the person familiar with CDM.

Cox said that officials at DHS had to “scramble to figure out how we’re gonna fund” the larger than expected workloads in some agencies and departments. Contracts had to be modified, he told CyberScoop after his panel at the summit. “We had to make some adjustments for funding to find the money we needed to cover that,” he said.

He declined to put an exact dollar figure on the re-scoping that took place, but said it had “impacted hundreds of millions of dollars” worth of contracts. He added that CDM program managers now had discovered the real baseline for every department and agency. “We know what’s out there now.”

He added officials were undergoing a “lessons learned” process before the next round of contracts were issued. “That’s one of the things we’re working on  now … to take a look at that data as we’re moving on [to the next phase] … to get those lessons learned to ensure that the next contract does it better.”

He declined to say which departments had the most shadow IT. “There were certain agencies that were better than others,” he said.

The person familiar with the program said the largest quantities of shadow IT devices tended to be in the “more operationally focused departments, … where you have more opportunities for an attitude that is more focused on getting the job done, achieving the mission, rather being focused on IT change management.”

Nonetheless, the person said, the new information presented a dilemma for the leaderships of those agencies with the largest amounts of shadow IT, because CDM capabilities to kick unauthorized devices off the network or otherwise deal with the risk they present would not be deployed until phase three. That’s not scheduled to be procured until fiscal 2018, starting this October, and probably won’t be deployed until at least another year after that.

The leadership now has visibility of all those additional devices, the person pointed out, “It’s not just a hygiene issue … those devices create a risk for the whole organization.”

Senior department officials are asking themselves “Do I wait [for CDM phase three], or do I take action right away to mitigate or reduce those risks? … What can I do now? … I’m responsible for this now, not in FY18.”