Bilyana Lilly on Western cybersecurity assistance to Ukraine

More than a year into Russia’s invasion of Ukraine, Kyiv has seen remarkable success repelling Russian cyberattacks. That’s due in part to expansive cybersecurity assistance from Western governments and private sector players paired with Ukraine’s remarkable investments in building up its computer response teams after repeated Russian cyberattack on its electrical grid.

CyberScoop Senior Editor Elias Groll recently sat down with Bilyana Lilly, who chairs the cyber track at the Warsaw Security Forum and has studied cybersecurity assistance to Ukraine, for an interview on Safe Mode to discuss what elements of Western aid worked, what didn’t and how Elon Musk’s role in provisioning satellite internet access has played a key — and at times baffling — role in the conflict.

This conversation has been edited for length and clarity.

Nearly a year and a half into the war in Ukraine, Russia’s invasion has failed to achieve its goals. Ukraine’s government has showed remarkable resilience, and in the face of what was expected to be an onslaught of Russian cyberattacks as part of this conflict, Ukraine’s digital infrastructure has largely stayed up and running.

One big reason for this is cyber security assistance from the private sector. I’m wondering if we can begin, perhaps, by describing the scale and nature of private sector cybersecurity assistance to Ukraine. What does that picture look like?

It’s a great question. Dozens of companies have been helping Ukraine since the beginning of this conventional stage of the invasion in February of last year.

But even before that, some of the larger cyber threat intelligence companies have been providing timely intelligence to the Ukrainian government months in advance of the invasion. From our data collection, we identified about 20 to 30 companies that have been providing assistance, but that number is probably higher because no one, not even the Ukrainian government, has a current list of contributions from all companies across the globe that are assisting Ukraine in one way or another.

They vary from the provision of hardware to software to cybersecurity services and with regards to the volume and size of contributions, they also vary from several hundreds of millions of dollars to single instances of incident response assistance. The contributions are rather diverse, and some companies also intentionally don’t disclose them. Some companies have a very clear policy of not disclosing the types of contributions that they’re providing. Others are more willing to talk.

I also want to caveat that whenever we read reports about different contributions, it’s important to consider that there may be companies that are helping massively and are very important to preserving the resilience and enhancing the resilience of Ukraine’s networks or helping with a particular incident response to mitigate a particular attack, but we wouldn’t know about them because they just don’t want to be in the public eye.

It seems like you can break down cybersecurity assistance to Ukraine into two main buckets. Hardware, software, and maybe a third in the form of services.

The one that I find the most fascinating is probably a component of the hardware assistance, and that’s the Starlink assistance, which has been a key tool for the Ukrainian military. They’re running the war in large part via Starlink terminals.

I’m wondering if you can, first off, just describe how the Starlink aid package to Ukraine came together. Additionally, what does the Starlink relationship and their work in Ukraine tell us about the role of companies of this nature and situations of armed conflict?

Starlink is one of the very interesting cases that’s also an example of some lessons that we have to learn and some policies that we may want to improve going forward. And by “we” I mean NATO member states or Western governments that are willing to support a country that’s a victim of aggression or a victim of an armed attack.

In the very beginning of the war, I think that this conventional phase of the war caught a lot of us off guard. I don’t think the Western community really expected the Russians will roll over the border and try to capture Kyiv in three days.

The introduction of Starlink into the war was rather unconventional. The vice prime minister of Ukraine sent a tweet to Elon Musk on the day of the invasion when the Viasat hack happened and asked for assistance, and Elon Musk immediately authorized the deployment of Starlink to Ukraine.

And ever since, Starlink has become pretty instrumental to the war effort. It’s being used by over 150, 000 Ukrainians. It’s used for command and control and communication. President Zelensky uses it to communicate with allied forces. It’s a very important channel for communication in the war.

I very much respect Elon, but one particular moment where I didn’t was last October, when he made a public announcement and also reached out to the Pentagon saying that SpaceX cannot indefinitely support the provision of Sterling to Ukraine because it’s expensive.

And this really alarmed a lot of us — people that are monitoring the war — because I do realize that companies are profit making institutions and they do not have a mandate to protect individuals or take sides in a war. I completely understand that. But in the middle of a war, when you know your services are critical to the defense of a country that’s under an armed attack, you don’t make a statement like that. That is inappropriate on so many levels.

I’m glad that he reversed that statement. Starlink is still being used in Ukraine, but that moment showed the world, it showed Western allies, it showed Ukraine how risky it is to rely on one particular tool to such an extent in the middle of a war when there’s no guarantee for its continuous provision.

If anything, this taught potential governments under attack in the future that they need to have more binding agreements with the particular companies whose services they’re using to such a critical extent.

Has that happened with regard to Starlink in Ukraine? Have the Ukrainians at all reassessed their relationship with Starlink or with Elon in particular as a result of this?

I think they’re still very much using Starlink. I know they’re relying on other communications as well. They do have a redundancy built in. But Starlink is still a very critical part of the war effort. It’s become a part of the critical infrastructure.

How do you assess Elon’s personal interventions in Ukraine and in the conflict? How do you read his role in the conflict as a charismatic CEO who on the one hand is willing to make these snap decisions to provision this really critical piece of technology and then on other occasions pulling back in weird ways and limiting its availability in certain regions of Ukraine and serving as an intermediary to the Russians at points of the conflict?

I think he is an incredibly influential man, and he has made so much progress on behalf of humanity on so many levels. But I think he has the resources to actually hire a solid policy team. I think he needs to work with experts, subject matter experts. I know you’re laughing, but I think it will really serve him well to have a few advisors who he actually listens to who are subject matter experts on these topics. And I think it will enhance his brand.

It will show that he has nuance and wisdom in his policies. I think he’s uniquely suited to have massive influence as a private actor on a number of geopolitical conflicts. I’m not talking about only about Ukraine. And I think with a solid policy team behind him and advisors he can be a force of nature.

Well, dare to dream that Elon will listen to this and take your advice, and we’ll have made a big difference for the world, and Elon will hire a top notch policy team.

So that was kind of the hardware side of it. Let’s get into the software side of it as well. One of the things you touch on in your paper, which I wasn’t aware of, was the existence of these devices called AWS Snowballs and their role in getting data out of Kyiv.

I’m wondering if you can tell the story of the use of these AWS Snowballs and the effort to save Ukrainian government data while Russian forces are approaching.

This was another example of these unorthodox methods in which companies started to assist Ukraine in the very beginning, because there really was no road map to deal with the situation. Companies had to make very, very brisk risk assessments and decisions. First of all, do we exit Russia? Do we exit Ukraine? And second of all, do we support Ukraine? And in addition to that, do we overtly support Ukraine or do we try to do this through indirect channels?

With regards to the migration to the cloud, this was critical to the resilience and basically to Ukraine staying online from the very beginning of the war. Because before this stage of the war, Ukraine has been conducting, especially the government has been conducting all its data transfers while using data centers that are located and servers that are located physically in Ukrainian government buildings in Ukraine, which would make those physical locations and data centers vulnerable to missile attacks from the Russians.

So the Ukrainian government realized very early on, even before the invasion, that they could be vulnerable to missile attacks. And because of that, they changed the law in Ukraine, allowing for data to be transferred outside of the country.

They issued a public call for help, and Amazon AWS responded almost immediately. And there was a meeting in the Ukrainian embassy in London. I wish this was recorded, and I wish we had like a whole video of this meeting, because I think this would be one of those historic moments when we talk about data migration and sovereignty in the cloud.

Basically, a senior person representing AWS and the Ukrainian ambassador to London met, and they created a list of the essential data of, I believe, 27 Ukrainian ministries, 18 Ukrainian universities, and some of Ukraine’s private sector entities, including Ukraine’s largest bank, PrivatBank.

They literally listed the critical assets and then three days later, the snowball devices, which are mobile devices for data transfer, arrived in Kyiv, where data was loaded and then physically exported outside of the country, dispersing it in data centers all over Europe.

Because of that the Ukrainian government, hospitals, the educational system still managed to function. And it wasn’t just Amazon. It was also Microsoft.

Did they have to smuggle these Snowball devices with all of this data out of Kyiv. There’s an untold sort of spy techno thriller about smuggling Ukrainian data out of a besieged in order to like save the data of the Ukrainian pension system.

Exactly. I’ve seen some of those trucks. Literally the snowballs are loaded on trucks and they’re being exported out of the country. In the first four months over 10 petabytes of data were exported. This is a huge amount.

Another thing to consider is that while the data migration largely was considered a success, some of the data wasn’t standardized properly and wasn’t exported properly because it was basically stored on legacy systems. So the data migration process wasn’t perfect and it wasn’t complete, which makes total sense.

Everyone who works in cybersecurity and data migration will tell you like, this is, this is absurdly fast. Like there must have been mistakes made here. And yes, clearly this wasn’t a perfect migration. So I think there will be a lot of stories that we can learn going forward. with regards to what is the speed versus comprehension and quality of a data migration process that could be conducted in duress or literally while a country is under attack.

But as a result of this data migration, it seems like Ukraine has been able to accelerate its digital services. There’s now this comprehensive app that allows citizens to access government services in a sort of new way, or it existed a bit before the war, I think, but the war has kind of turbocharged this effort.

Absolutely. Now most of their services are conducted on the cloud. Ukrainian representatives have stated on multiple occasions that Amazon and Microsoft basically saved the ability of the government to conduct its functions without interruption or without major interruptions.

Zooming out a bit, I’m wondering if you can talk through what you think has worked and what hasn’t in terms of cyber assistance to Ukraine.

I think we’re going to learn more and more about what worked and what didn’t in different stages of the war in the decades to come. Right now, we still don’t have the perfect picture and we’re still in the fog of war.

But we now know what worked, for example, in the beginning, during the disruptive phases of Russian cyber warfare against Ukraine. The anti DDoS tools that several companies provided, specifically Cloudflare and Google, were instrumental in making sure that Ukraine can survive the DDoS onslaught that it was subjected to. Those companies were very important at first. Cloudflare was even called at some point the Starlink of the initial stages of the war because it was that essential to Ukraine’s ability to defend itself in cyberspace. We also know Microsoft has been helping a lot with cyber threat intelligence.

There was a 24/7 encrypted channel between Microsoft and Ukrainian government already before the conventional stage of the war even began. We also know Mandiant has been instrumental in providing cyber threat intelligence, threat hunting, as well as incident response.

ESET, the Slovakian company. They’re small, but mighty. They have been assisting with taking down malware and mitigating attacks from Inudstroyer2. And you covered that in one of your articles last year. And we know that could have been quite a massive attack turning off the electricity for over two million Ukrainians.

I’m glad you brought up Industroyer 2. For listeners who aren’t aware, this was a very sophisticated piece of malware that was directed at Ukrainian electricity generation services. It looked like an effort to shut off power in Ukraine again. It was discovered before it was ever deployed, and one of the great mysteries is, of course, how it was discovered and who discovered it, which might be a part of this assistance campaign.

I think it bears emphasizing also that the Ukrainians have been on a crash course of building up their own cyber security capacity really since this stage of this conflict with Russia began in 2014.

There have been several attacks on the Ukrainian electricity grid that have been successful, and the Ukrainians have since stood up this really impressive CERT organization that’s been working overtime during the conflict. They deserve a lot of the credit here in the situation as well.

I wonder if we can go back to the tech companies. I’m curious, where do you think they fell short in their assistance to Ukraine?

It’s an excellent question.

From the latest discussions that I’ve had along the sidelines of different conferences, one issue is the coordination of assistance and its timely provision.

Another is that for certain companies, some of the free licenses that they have provided are about to expire. And companies have started discussions with governments with regards to who will be paying for these services going forward and for how long the companies can afford to provide assistance free of charge. There are discussions with governments for potential collaborative funds, where governments could provide that support.

We cannot expect companies to defend a country when that country was attacked by an actual nation state. I think that’s the job of NATO and NATO governments. So one issue would be who pays for the provision of the assistance. And here’s where I also agree with Elon that, yes, Starlink should be a paid service at some point. But I don’t agree with the very disruptive message: Hey, I’m going to announce to the whole world that I’m going to cut the service if you don’t pay me.

Another conversation to have is with regards to the compatibility of different services and the information that cyber threat intelligence companies provide to Ukraine. From what I hear, they’ve been very open to sharing intelligence with the right Ukrainian representatives. In the beginning, it was difficult to identify who the players are. And it was very much, the relationships were built very ad hoc and very quickly. I think going forward, this problem has more or less been addressed in the case of Ukraine.

But in future scenarios, where let’s say Moldova or Georgia, hopefully this never happens, but thinking from the perspective of a geopolitical analyst, those are countries where there’s a certain level of risk that they may be also basically subject to aggression. If this happens in these countries, and companies would be willing to provide assistance, I don’t think anyone has at this point a breakdown of who needs to be contacted to coordinate that assistance.

I just want to put a pin in it and ask you: Do you think that tech companies coming out of the conflict in Ukraine know how to operate in a time of war? Have they learned how to do this work coming out of Ukraine?

I think they have stress tested a lot of their tools, hardware, software, and cyber services. And I think they have some massive lessons learned about cloud migration, cyber threat intelligence, how to communicate with a government in conflict. I think they have, yeah, learned a ton.

Another issue I wanted to bring up with regards to coordination. CDAC is a body that was stood up in March of 2020. CDAC stands for Cyber Defense Assistance Collaborative for Ukraine. And it’s a non-governmental organization that was set up to coordinate assistance from Western, basically private IT companies to Ukraine. And it was set up by one of my co authors on this research, Greg Rattray.

He’s partially coordinating the assistance of over 15 companies, including Microsoft, including Mandiant and a few others that are a part of the organization. And what they are trying to do is have an inventory of the potential services and products and tools foreign companies can offer to Ukraine and then also coordinate with different Ukrainian agencies and understand their needs as quickly as possible. And then basically serve as this focal point where supply meets demand.

In the event of another conflict of this nature warfare, do you think that we would see the same level of private sector assistance? Let’s say China invades Taiwan. Do you think we see the same kind of crash effort to assist Taiwan?

I think no one can answer that with definitive certainty at this point.

There are a lot of questions to consider with regards to market share of companies, their presence and footprint in China, with regards to what U. S. policy will be towards that conflict. How much would the United States get involved? In the case of Ukraine it was very clear who the aggressor and the attacker is. It may not be as clear in other conflicts.

Another major issue is how we define an armed conflict. What if the confrontation just happens in cyberspace? What are the red lines? If Russia had escalated the war in Ukraine just in cyberspace would we have seen such an outburst of support? I doubt it.

Do you think the fact that Russia did not represent a particularly lucrative market made it easy for Western tech companies to pull out of Russia and choose to support Ukraine? To what extent do you think that those kinds of market considerations underpinned the decisions by Western tech companies to step in and aid Ukraine?

It’s a great question. I think there were some debates in boardrooms that I wish we could be privy to. In hindsight, we say, yes, thousands of companies left Russia. Yay for democracy! Success! We showed that our companies have hearts and humanitarian priorities, and they care about victims and aggressors and all of that.

Something that our Ukrainian colleagues highlighted for me a few days ago: They said, Bilyana, don’t take for given the fact that we needed to win the information war at first. We needed to make sure that we make it so damaging for brands of companies that stay in Russia, that they need to exit.

And leaving the Chinese market in the event of an invasion of Taiwan would be a much more difficult decision than the decision to leave the Russian market, right?

It will be, but it’s not impossible. Can we incentivize a transition from a volatile conflict prone market into another market where return on investment could still be high in the long run? Can governments assist companies with that transition? Our governments have a responsibility to think about how to make it easier for companies to exit that market and incentivize a transition into markets that are more stable and like-minded.

Bilyana, thank you so much. This has been a fascinating conversation. Really appreciate you taking the time to share your expertise and your research.