The inside story of the infamous SolarWinds hack

Sudhakar Ramakrishna was sitting down to a birthday meal with his family when he received the call: SolarWinds had suffered a large-scale cyberattack. The date was December 12, 2020 and Ramakrishna was due to start as CEO in a few weeks’ time.

The full scope and severity of the incident was not immediately apparent, but he was still left with a decision to make. Would he abandon the ship, which had sprung a leak under the stewardship of the previous captain, or grab a bucket and begin to bail?

A number of close confidants advised Ramakrishna to abandon the post, while others suggested his skill set and experience in cybersecurity made him the ideal person to preside over the recovery.

Although he took a beat to consider his options, the decision to stay the course was in the end a straightforward one, Ramakrishna told TechRadar Pro. The board was informed he would step back if it was decided SolarWinds would benefit from continuity, but that he was otherwise prepared to pilot the company through the crisis.

In the weeks that followed, Ramakrishna began to collaborate with the executive team behind the scenes. The first priority was to find out exactly what had happened, and how, and the second was to formulate a plan of action that SolarWinds could bring to its customers, partners and the press.

“The idea that an attack can happen to anyone has become more prevalent, but that does not absolve you of the fact it happened to you,” he said. “Every company will have a crisis or two, but what matters is how management reacts.”

A rocky beginning

The attack itself had actually begun many months earlier, in September 2019, when a sophisticated group of cybercriminals with suspected links with the Russian state first gained access to the SolarWinds network.

The threat actors demonstrated remarkable patience, hiding in plain sight while they built up a comprehensive picture of the SolarWinds infrastructure and the company’s product development process.

Among the various SolarWinds products, the attackers were particularly interested in an IT performance monitoring service called Orion, which needs privileged access to the customer’s systems in order to function as designed.

After an initial test run, the hackers injected a malware strain known as SUNBURST into an Orion software update at some point between March and June 2020. The poisoned patch was delivered to circa 18,000 SolarWinds customers, giving the attackers practically unfettered access to the networks of government agencies, security companies and multinational enterprises in the process.

“The industry is not new to security issues, but each comes with its own twist and significance – and this was significant in its own way,” said Ramakrishna.

“The tradecraft used to create the breach was not run-of-the-mill, this was a supply chain attack. This is a well-known concept in the security space, but not a well-exercised one.”

What makes an attack of this kind so difficult to detect, he explained, is that the threat actor need only modify one of many thousands of files to successfully conduct an attack that results in the compromise of a large number of targets.

In the end, the group chose to infiltrate only a subset of the compromised organizations – including Microsoft, Cisco, VMware, Intel and a number of US federal agencies – but the attack has nonetheless been described as one of the most significant in history.

When SolarWinds was alerted to the incident by security firm FireEye, which had detected unusual activity on its own network, the company went into crisis mode. And it was in this climate that Ramakrishna stepped through the doors on his first official day in charge.

However, while the morale among staff was predictably low and the conversations with angry customers often difficult, the crisis at least provided a platform on which Ramakrishna could build.

“In some ways, making change in the midst of a crisis is easier,” he told us. “When everything is perfect, there’s a lot of resistance, but when a company is shell-shocked people are receptive to new ideas.”

On January 7, 2021, Ramakrishna published a blog post that outlined what had been learned about the attack so far, proposed immediate steps to help customers navigate the incident and set out a new framework to prevent a similar attack from recurring in future.

The supply chain conundrum

Although SolarWinds has managed to right itself over the past twelve months, with customer retention now returning roughly to pre-attack levels, the incident had severe effects on the company’s bottom line.

Instead of funnelling resources into product development, sales and demand generation like a normal business would, the company was forced into recovery mode, with its reputation in tatters.

Ramakrishna and his executive team divided up the customer list and began to meet with many of them individually, both to apologize and explain what had happened, and to help them investigate whether their own networks had been breached.

He described this as a highly uncomfortable but essential part of the “healing process” that eventually paved the way to a return to normal business operations.

However, despite the consequences for SolarWinds, there is evidence to suggest the right lessons have not been learned by the wider cybersecurity industry. Since the attack, a number of similar high-profile incidents have taken place, like the Kaseya attack, Log4j and, even more recently, the Okta-Lapsus$ breach.

Asked why he thinks supply chain attacks continue to occur, Ramakrishna explained that the disjointed nature of the collective defense gives a significant advantage to the attacker from the outset.

“This is not just a technology issue, there’s a lot more to it,” he said. “Each one of us is defending against an attacker. But on one side is a coordinated army with a singular purpose, to attack, and on the other is a set of fragmented soldiers.”

Ramakrishna was also critical of the culture of victim shaming, which he believes contributes to an unwillingness among companies to share vital intelligence.

“There is still a lot of victim shaming that happens, so companies often end up fixing problems without saying anything about them. There is definitely hesitation to speak up,” he told us.

“In the event of an incident, it’s important to leverage help from the community. We need to make people aware of issues faster; that mindset needs to establish itself in software security.”

To prevent a supply chain attack of this scale happening again, Ramakrishna also believes businesses need to embrace a new security framework, which he calls “secure by design”.

There are three components to the model: infrastructure security, build system security and the design of the build system itself. But the general idea is to keep modifying the attack surface, so as not to provide an attacker with a fixed target, and to minimize the window of opportunity.

With this objective in mind, SolarWinds has created a “parallel build system” whereby its software is built in three separate locations, which can be changed dynamically. The result of each individual build is then cross-checked with the others to weed out inconsistencies that might betray an attack.

To successfully infiltrate a software patch, therefore, an intruder would have to launch three attacks simultaneously, at precisely the same moment and using precisely the same technique.

“That’s a very difficult thing to do, even for the most persistent cybercriminal,” said Ramakrishna.

The new-look SolarWinds

Ironically, it has been suggested that SolarWinds might now be considered the most secure company in the world. After all, no other organization has undergone quite the same level of scrutiny in the period since the attack was discovered.

Ramakrisha refused to be drawn into commenting on whether or not he believes this characterization to be accurate, but he did say it is something the company is “determined to make true.”

Operating under its secure by design framework, SolarWinds will now look to build upon its foundations in IT monitoring and evolve into a company that can support the hybrid needs of customers, both in the cloud and on-premise.

Ramakrishna has promised a heightened level of automation, and superior visualization and remediation facilities that together will help address the kinds of issues created by digital transformation. The objective is to “reduce complexity, improve productivity and cut costs” for customers, we were told.

With a few rays of sun now starting to peek through the cloud hanging over the company, Ramakrishna is eager to turn his focus towards these central goals. But as our conversation drew to a close, he also took a moment to warn against complacency:

“No company, no matter how much they do, should believe they are immune from attack, because that’s a fallacy,” he said.