How Resilient Are Our Vital Services in a Cyberwar?

Roy Brander, P.Eng. Vancouver, March 2022.

It's not that it's completely impossible to get into major systems; we see the stolen-data stories all the time. These are not vital services control systems, though, the ones that keep you in water, power, heat. Those are exponentially harder to hack than your magazine subscription data. But, we have seen a few super-hacks now: Colonial Pipelines, causing gasoline line-ups.

Even in that case, however, the control system for Colonial Pipelines was never compromised: the ransomware gang compromised their billing and scheduling system. Colonial claimed that this made them fear their control system might be next; critics wonder aloud whether they weren't about to ship gas if they wouldn't be fully paid. Security consultant and researcher, Terry Ingoldsby, assures me it was more complex than money. "They weren't sure where a load of gasoline ended, and the next load, say diesel, started, without that billing system. They stopped it rather than deliver diesel to a gasoline facility."

Security Through Obscurity

Actual control systems are the hard targets, which is why I did an eye-roll at a column by Matt Gurney at The Line. It opened with a scenario where your power is off, your phone is blank, and the radio stations are all silent.

For a month, I've been trying to fact-check that possibility. It's hard to get responses. CBC did answer me. The Mt. Seymour antenna, that broadcasts all CBC TV and radio to about 2.5 million people in the lower mainland, is indeed protected and provisioned with backup generators and fuel supply.

CTV, on the other hand, finally responded to multiple queries with:

Walton, Dawn

Greetings Roy ~

We won't be sharing corporate information with members of the public.

Sorry about that.

Cheers ~ Dawn

That was typical. Nobody in the business wants to give away that information. They fear giving some sort of secrecy advantage away, more than they want to reassure customers that they can be counted on.

CBC was barely nicer. I made multiple calls and emails, asking if CBC Vancouver was provisioned with backup power and ample fuel. Finally, "Ash, CBC Audience Services", wrote back a whole eight words, "Yes, I can confirm that is the case". No details at all.

I got a somewhat better response from a small local station - not better enough that I'm allowed to name them - which had an actual human on the phone, who cheerfully connected me to another one when I asked for "engineering".

That guy started off demurring that he couldn't tell me anything, wasn't authorized to speak, as always. I shifted to asking if "most radio stations" did have backup power? Did most have CB radios to stay in touch with authorities in case the phones went out? I promised to name neither him, nor his company. He relaxed a bit, and confirmed that "only the smallest Mom and Pop stations" did not have such preparations.

CTV's curt response was much nicer than the people who provide your electricity, your phone, the heat in your house. They aren't about to give away a single bit - literally - of information that might help an attacker. I did not get "no comment", I simply go no reply to any number of contact attempts.

I would have no article for you, except for the Vietnam Lunch Gang. As it happens, I know people who work in some of the most-sensitive computer centres in Western Canada. That's why I knew that Gurney's scenario was over-the-top.

Water

Let's take a moment to cover those truly vital systems: without water and sewer, you have to abandon a city before the plagues start. I can flatly tell you that those plants are cyberwar-proof: not because the computers are impossible to compromise, but because the plants don't really need them. Plant control systems are labour-saving devices for water and sewer: every machine in the plant has a switch to take it off "auto" and onto "manual". Five guys can run the plant with computers, but fifty with walkie-talkies can also run it, with every computer dark.

What about the smaller computers that run individual pieces of equipment - like the one that controlled the sodium-hydroxide-dosing valve for Pinellas county? An operator there watched in astonishment, as the system apparently decided for itself to increase the bleach-disinfectant by a factor of 10,000.

That's where the Pinellas stories were misleading. A plant-operator friend, who taught me what little I know about the Glenmore Plant in Calgary, noted that the control system that was "ordered" by a hacker to increase the chlorine dosage 10,000-fold, would not directly control the valve: the controller would have sent a message to the tiny, built-in computer that ran the dosing system. That little computer is hard-wired to not open the valve further than the legal limit for dosing. It can't be compromised without a screwdriver and a new memory chip. (Additional protections were offered by the valve not being large enough, the tube feeding the bleach isn't big enough either, and there wouldn't have been enough bleach. Newspapers never mention tension-reducing factoids; tensing you up is their business model.)

Logic Lads Who Lunch

The contractors kept coming to join the Vietnam Lunch Gang, long after they parted with Calgary's Engineering Department. They got jobs in other server rooms around town: the ones that store all our oil properties; the ones that run oil and gas through pipes; the ones that provide communications through two very different kinds of wires, or antennas.

Names? I still can't give names, not even corporate names, or attribute the quotes I have for you; "not-authorized-to-speak-for-the-corporation", it's practically one word, these days. What I can give you, are their comments.

Municipal Services

The Calgary IT crew have been preparing against cyberwarfare, for years. He writes:

The Canadian Federal critical infrastructure group [at https://cyber.gc.ca/en/] have some really good cyber folks, some of whom are focused on working with municipalities on cyber security awareness and preparedness.

I used to get daily NATO cyber briefings (NATO even have prep sheets on how to protect an election); but the cool thing is it wasn't very often that the NATO info beat my folks to the punch.

He added that the computing operations for most really large providers also have seats on worldwide advisory boards, like the one hosted by Microsoft; there's a great deal of "mutual support networking" - and those kind of people-networks get stronger in disaster, when the physical networks are embattled.

In short, the people who worry, for a living, about your 911, whether the emergency services radios will work, whether your governments are in touch with each other, have been thinking hard about cyberwar for decades. They not only have defenses, but practice cold-start recovery. The time between intelligence services spotting a concern, and your local government patching the security hole, is mostly measured in hours.

That doesn't mean Calgary's EOC might not be compromised in a cyberwar; it means it will be back up quickly. Every major server room in Canada will literally be in touch with NATO, one jump removed, for fixes.

Will the radio stations have any news for you, if their phones are down? The radio people sound awfully confident. All the emergency services and municipalities around Vancouver use products from "E-Comm911". I asked Kaila Butler of E-Comm, who provide Emergency Services communications to millions in BC, about whether somebody could "hack" their encrypted, computerized radios:

The critical elements of E-Comm's Wide-Area Radio System, used by police, fire and ambulance personnel within Metro Vancouver and parts of the Fraser Valley, are isolated from external access and have multiple redundancy and failover capabilities. We recognize that all networks have associated security risks and this was a key consideration during the design and implementation of the radio system.

There's a lower level of backup, on top of the radio professionals: the radio amateurs. Vancouver has "VECTOR", its ham radio club. About all they do, as a group, when they are not at home, tinkering, is prepare for disasters. Their job will be keeping people in touch with each other, and with disaster services. There are a lot of them, and the more-rural the area, the more radio amateurs you find. You really don't need to worry about your local radio stations being on-the-air, but not having any news.

Jesse Neri, of VECTOR, noted various special radios that emergency services use, cost $6000. The amateurs of VECTOR use radios that the Windtalkers of WW2 could probably repair - no computers inside.

There's something that the water treatment operators and the VECTOR radio amateurs have in common: they understand the technology they use, down to the bare metal. They could run it by hand, physically repair equipment. They understand mondern technology is a convenience, not a necessity. The water-treatment guys could build a filter and chlorine-doser from parts; the VECTOR guys could build a radio from parts. Their middle name is "resilience".

So, that's your city government, your emergency services, your radios to tell you what the government and cops know about the Fall of Civilization. You will in fact know that something is going on, be reassured that repairs are coming.

So, what will be wrong? We'll have water and sewer; we'll probably have power, but are assuming not. What about heat? Phone? Internet?

Wires

Major infrastructure nodes (like cell towers, gas compressors, oil pumps) typically have 16 hours of battery, and 4 days of diesel. With longer outages, "everybody would be competing for diesel" and nodes could drop.

"I'm not sure Matt is actually overstating things. If a state actor like Russia or China really did want to cause havoc, they could."

Then there are attacks directly on the [our system] infrastructure itself. [Our company] goes to some trouble to make sure that the systems that interact with the [hardware control] network are firewalled off from the business network, but there has to be some communication...the systems that run the ... network don't have direct internet connectivity, but if a bad guy has already set up shop inside the [company] business network, and can send requests to servers in the [infrastructure] network.

A soldier must take down an assigned enemy - and it's not cyberwar unless they take down nearly everything. Denying power to Moose Jaw for a day would only give Rideau Hall the giggles. ("Oh, did they have electrical power already?")

Time and a lot of patience, to get into the business network, then "haunt" it, looking for the moments when it's in contact with the control network for maintenance. Unless you have a year to spare, and a hundred hackers that specialize in various systems, different computer brands. Preferably, you want actual spies: humans that get jobs in the company. After haunting the business computers for months, watching emails, getting names, finding the names of servers, finding the rare occasions when the two networks are connected for maintenance or updates, you can strike at the infrastructure networks that run our phones and pipes, from inside those companies, not from the Internet itself.

Yes, that's possible: my expert informants agree. It's possible for a nation-state spending staff-centuries, and tens of millions of dollars, on the attack.

Pipelines

Are many companies in remedial reading class? Perhaps. It's scary is how many organizations are. It's easy to know what to do, tricky to get everyone to do it (and to spend the money to do it). And then people still mess up.

None of my informants have run into an attack that's nation-state-level, the way the Iranian nuclear program was hit with "Stuxnet", so far the greatest hack ever - which probably did cost Israel and America a staff-century (or two). Aware of the possibility, since Stuxnet, our guys are always trolling their own networks, looking for lurkers. Certainly, they have sustained the kind of attacks you hear about in the news: the "denial of service" attacks, where thousands of taken-over home PCs beat upon your servers with millions of empty messages, like a group of diallers tying up a phone-line. Large, serious server rooms have responses to those, in place, and respondents I can't name have already handled such bombardments, and shrugged them off. Most of your services are not in the hands of the remedial reading class.

The Consultants That Serve Them All

Terry Ingoldsby of Amenaza Technologies, cybersecurity consultants, has heavily studied all the research work done by National Laboratories and the largest power companies, into how we make the electrical grid reliable, perhaps our system most-dependent on electronic controls. He's seen some frightening demos, where compromised control systems can literally wreck a generator by tampering with its electrical phase, or timing. He notes that only so many spare parts are kept on-hand, and a program script can loop through every switch or breaker, with damaging instructions to them all.

On the bright side, the electrical systems are perhaps our most cyberwar-ready. They are keenly aware how vital they are, and how often natural events have taken them down. Terry assured me, I was correct that "extensive mitigation measures are in place".

Unfortunately for my little crusade to prove Matt Gurney a worry-wart, calling these guys up was that classic Churchill problem:

Why, you may take the most gallant sailor, the most intrepid airman, or the most audacious soldier, put them at a table together - what do you get? The sum of their fears.

  - Winston Churchill

The sum of Terry's fears went on for a few pages - but what I noticed was that most of them related to physical vulnerabilities, not pure computer attacks. The power grid may be specially-vulnerable to actual damage; the control you need to run it is also enough to damage it - unlike the self-limiting chlorine doser over at the water plant.

Terry adds, however, how very hard it would be to compromise just one facility, much less hundreds around a continent:

Even supposing that someone manages to gain access to the ICS [machine control] network, they still have to figure out which PLC controls the dynamite injector and which controls the soda pop machine! This is apparently quite tricky to do simply by observing network traffic. In most cases external sources of information are required. But these may be available in a variety of ways.

Most heartening of all, his main concern with the cyberattack side is that they've done enough good work already, that they may be overconfident:

I am currently working on an OT attack tree for an industrial facility with over 500 million attack scenarios. The *vast* majority of these scenarios cannot be achieved by any plausible attacker - that's good news! But what is remarkable to me is that most critical infrastructure providers are so smugly confident in their defences that they don't do this type of analysis. BTW - this is exactly the type of analysis that is routinely done for military control systems (such as avionics). Canada also uses this type of analysis in nuclear installations - but it has been astonishingly hard to get people to do it in other fields.

Just Do The Disaster Prepping You Should Have Done, Anyway

Where the optimists (myself) and pessimists (Matt) can agree, is that we probably need to add some level of official requirements to the solid voluntary work that's been done so far. Regulated businesses, like pipelines, wires, telecomm, all have many regulations requiring them to run safe workplaces, run their vital services with responsibility to the public trust. They need hard, mandatory regulations for cybersecurity, as well.

Our odds of waking up in the dark - and worse, the house getting colder - are really very bad. It's possible, but kind of a moonshot: a really major project for hundreds, or thousands of hackers, there being no superhackers. A lot of money and time for a distraction, in most places.

Your own preparations for it are the same as natural disasters: those 72 hours of basic supplies.

Our national preparations, the guys in the actual IT trenches tell me, should mostly be to firm up their best practices as required standards. It's a mutual job, between government civil services, some of our largest private companies, with the military and intelligence in one big loop. With cyberwar, civilian and military blur: we're all on the front line.

Then I Did A Presentation, Did More Reading, More Ukraine News, Changed My Mind

It was the really crucial service of electrical power that dominated the whole CUUG presentation, and subsequent discussion. I did land my points, that Canada has over 100 large power plants, across 5 technologies (coal, gas, nuclear, wind, hydro), and 30 owners - you'd have to learn a lot of different hacks to take down much of our power.

Reading more on it, I found that I'd missed an article from 2018:

What caught me about the article was the 'shrug' attitude of the experts: that Russia had been poking around the US power grid controls for years already, this was the first time that DHS had said so publicly.

I read more on the 2015 power grid hack in Ukraine, and then the 2016 update to it with Industroyer malware. It sank in that we actually know a fair bit from that about Russia's capabilities and approaches, that since then, our side has been studying them as they maneuver in our networks.

And then I started finding the hot-burning news that had come out since the Gurney article. Principally, that Russia tried to take down the Ukraine grid for a third time, and failed:

The Russians gave the game away with the test of Industroyer in 2016. This time, the Ukrainians were ready for them, watched the hack start, they shut it down before it could complete, no power lost.

And that's the Russians at shooting war with an enemy who's recently killed 10,000 of their soldiers. I doubt we are in for worse.

The news continued:

It's not like we're the only ones. This is a global problem. One can hope that the power operators and other vital-services providers. All of this was read in a new context. I'd already understood that hacks into "ICS" (control) networks took some patience. Most attacks are done by entering the "business network", the one that the Head Office uses for HR and accounting, and has Internet connections, then waiting, often for months, for a maintenance connnection to be made with the ICS network. You have to lurk, to haunt, the business network. Even after getting into the control systems, you may have to lurk there for as long again, learning how to seize control of infrastructure.

What I hadn't stopped to think about, before doing the CUUG presentation, was that this was a security exposure for the attacker. A "cyberattack" has to be preceeded by months of "cyberpreparations" and staging. You can't just decide to attack somebody on Monday and be there on Tuesday. It's a full-time job, just maintaining attacks that are ready to go. If you want to attack 100 different power stations, you have to have many employees, full time, getting them mounted and maintained. It's very much like building a military force: you need to start manufacturing tanks and warplanes years before to have the whole force ready when it is needed.

So, in short, we are absolutely "ready for cyberwar", in that the cyberwar started years ago, and goes on every day, attackers and defenders maneuvering around each other, preparing for The Day. It's going on in India and China, and Tennessee and Moose Jaw, all the time.

We are reasonably ready for the cyberattack, because we're already in the cyberwar. From the day before the presentation, they're already working on the wind farms:

Cyberwar Blog?

Far from squashing us with cyberwar, Russia seems more likely to be on the short end of it. The loss of the technical staff is mostly (or entirely) from civilian business that has nothing to do with cyberwar itself - but the loss of talent pool bodes ill for their cyberwarrior staffing in years to come.

May 8, 2022: Russia is the suffering one in this cyberwar

The hackings in the WaPo story are not infrastructure cyberattacks shutting down power and phones; just information-thefts, and annoyances to their IT. But it all does some damage, passwords lost, files destroyed, trouble abounds.

And as for attacks on our cyber infrastructure: mostly crickets.

Copyright, Roy Brander, 2022