Ransomware taught key lessons
Senior managers at Hydro were awakened early on 19 March 2019 with the news that the Norwegian industrial group appeared to have been the victim of a major cyberattack. Nobody knew its scope or potential consequences, only that it was serious.
- Cyber security
This ransomware incident affected 32 000 employees and activities in 40 countries. The question is how a company should respond to a crippling event, which shuts down most of its systems. Where do you start?
“One of the first things we did was to post a warning at our main entrance,” explains Halvor Molland, senior vice president for group communication. “But it had to be handwritten, because nobody could access a printer anymore.”
He was among the executives alerted that night,and has since repeatedly told the story of what happened. And, according to him, he often starts with that notice. It has become a symbol to many of how extensive the attack was.
“The next thing we did was to report the incident to the police – via text message, because the internet was inaccessible,” Molland explains. “We didn’t have much hope that they could crack the case, but it was the right move.”
Asked how the group notified 32 000 employees in 40 countries in these circumstances, he says it involved a combination of what they had been drilled to do and exercising creativity.
Hydro had practised dealing with a cyber attack, detailed plans were available, and the organisation was trained to mobilise quickly.
“But we hadn’t envisaged something of this scope,” he admits. Creative solutions for spreading the word included establishing WhatsApp groups and a dedicated news app for employees.
Chief information security officer Torstein Gimnes Are was another of the Hydro executives awakened that night. His first priority was to secure an overview – which was not straightforward.
“When you suffer a cyber attack, it often aims to isolate, restrict and shut down,” he explains. “But we saw here that most of our business areas in every country we’re in had been hit.
“It quickly became clear that we faced a massive assault which would have major consequences for our systems.”
Hydro has a common IT platform used by all its business areas. And the immediate response was precisely to shut down all network links and servers. When people came to work, they had to start up manual routines where the platform was unavailable.
“Chaos reigned, but we started work at once on building up a secure infrastructure,” says Are. “This was based on a backup system which is independent of our IT platform.”
That job took weeks and months. During the initial phase, the group was left without an IT system – which had varying consequences.
Since production was not directly connected to the group’s IT platform, most of its factories could go on working. With no support systems, however, it was difficult to continue for long.
“They had no access to customer lists or order books, for example,” Are explains.
Hydro’s hydropower facilities are subject to Norway’s emergency preparedness regulations, which require such systems to be separate from a company’s IT platform.
They thereby escaped becoming part of the attack, unlike the factories. But exceptions also existed for the latter, with Hydro’s plant at Lichtervelde in Belgium largely unaffected.
However, that reflected the sales manager’s scepticism about the IT system. Every Monday morning, he therefore printed out all the orders and put them in a ring binder.
“He became a hero, because the factory was more or less unaffected by the attack and could operate as normal,” reports Molland.
In the rest of Hydro’s production network, people were forced to empty wastepaper baskets and containers in search of possible customer data which might have been printed out.
“Most parts of the group faced the equivalent of tossing all computers and mobiles out of the window and trying to keep going for six-eight weeks,” says Molland.
The Hydro management resolved at once to admit what had happened. Early on the morning of the attack, the group issued a stock market statement and a press release about the position.
“We always try to be open,” Molland explains. “I’d say that’s part of our culture. But there are clearly a number of things we can’t say for competitive reasons and the like.
“And here we were also in a position where not everything could be revealed for security reasons.”
Everyone working to restore the Hydro platform knew that the attackers still had access to the same system, and were probably following the media coverage.
So a number of things could not be shared with the outside world, Molland explains. “It was demanding, but we were as open as we could be at any given time.”
The virus attack on Hydro was intended to force it to pay a ransom in order to regain control over of system. But surrendering to this demand was never an option.
“Even if we’d paid, the attackers would still have had access to our system,” explains Are. “So that was always out of the question.”
He reports that the ransomware virus entered the system after an employee opened an attachment to an e-mail from somebody he knew well and had been expected to hear from.
“In other words, there was no point in blaming anyone for this.”
The question then is what you can actually do when e-mails you think are completely secure turn out to contain a virus. Is it impossible to defend against a cyber attack?
“It could seem that way, of course,” says Ar. “And it’s true that if somebody wants to break in and devotes enough resources to doing so, they’ll succeed.
“Having been in such a position and felt the sense of powerlessness and frustration, however, we’ve acquired much useful experience and learnt many lessons in tackling such attacks.
“I find that Hydro’s employees have become more aware and awake, asking questions and seeking help. This is about communicating with people about the threats which exist.”
He adds some helpful advice, starting with backups. “These are naturally important, and should not be connected to the regular IT platform. That will ensure access if you have to rebuild from scratch – like we did.
“We in Hydro then benefit from doing a lot of drills, thinking through different scenarios and regularly questioning whether our backup is good enough. That’s something every company should do, regardless of size.”
It also makes sense to have something on paper which can help to maintain activity should something like this happen, he adds, and points to the segregation in Hydro between the IT platform and the factories, which helped it to sustain production.
Molland reports that the group often runs courses and drills on such issues as phishing and social manipulation, and seeks to devote continuous attention to IT security in the organisation.
“That could involve everything from courses on identifying harmful e-mails to how you should relate to TikTok or ChatGPT.”
To learn more about this story, in Norwegian only, see the Cyberangrepet episode at psa.no/podcast. Halvor Molland and Torstein Gimnes Are describe in their own words how they experienced the 2019 attack.
Lessons learnt by Hydro were not confined to IT security. The group also found that it could be sensible to take a few technological steps back from time to time.
“Independently of the IT system, we see that at our factories have gone too far in some cases with efficiency improvements and automation,” says Molland.
“A few too many buttons are removed. We learnt that reintroducing some manual control systems on machinery makes sense for maintaining output during a crisis.”
When production ceased, the group found that a number of its retirees returned and offered their expertise in manual control of the machinery.
“These devices had been so thoroughly automated that nobody in the workforce had experience of running them manually,” Molland says.
He points to a number of newspaper headlines along the lines of “Hydro saved by the pensioners”, and would not personally go that far.
“Nevertheless, having the technical know-how to run a system both manually and through an IT system is undoubtedly important, and another valuable lesson we’ve learnt.”
This incident left the Hydro group with documented losses of more than NOK 800 million.
Today, however, the Norwegian police know a lot more about who carried out the attack and not least – thanks to all the documentation from Hydro – how better to halt such attacks.
Molland regards this as further confirmation that Hydro did a lot right in the hours after the assault.
“We hadn’t expected the police or the Norwegian National Security Authority (NSM) to discover so much about this case. But it only shows, once again, the importance of being open.”
Read more articles about security:
Security in uncertain times
Safety and security must be viewed collectively
Societal safety in a new era
The threat picture and Norwegian petroleum activities
Coping in troubled times
Tackling the threats
Calm under pressure
Standing firm against a storm
Ransomware taught key lessons
Stronger spotlight on ICT security
Praise for oil and gas sector from armed forces chief