One year ago, the Log4j vulnerability was discovered. The discovery immediately stunned the world due to the severity of the exploit and the extensive exposure – it is estimated that over three billion devices using Java were affected. The Log4j vulnerability could lead to remote code execution (RCE), the most lethal cyber threat. For this reason, it has been called “the biggest, most critical vulnerability of the last decade.”
Maintainers were quick to fix a fix for the Log4j vulnerability. Still, it took months for many software vendors to update their applications to the new version. And a year later, one in four Log4j downloads still have the vulnerability according to data from the Maven Central repository. Although this number is steadily decreasing, it still means that millions of applications are exposed to the vulnerability. As a result, Tenable Research found that 72% of organizations are still exposed to the Log4j vulnerability.
Given the severity of the exploit, why haven’t all applications upgraded to the latest version? I recently caught up with Brian Fox, co-founder of software supply chain security company Sonatype, to understand why it has taken so long for organizations to respond to Log4j. Below we will examine why Log4j has gone unlimited for so many distributions. We will also propose solutions to remedy the unsustainable state we are in when it comes to open source software dependency awareness (or lack thereof).
Unpatched Log4j vulnerabilities persist
Sonatype’s research center has tracked over 130 million Log4j downloads since December 2021, of which 34% are vulnerable. And as of this writing, in the last 24 hours, 25% of downloads were vulnerable. This means that one in four downloads of Log4j is still of an outdated, vulnerable version.
The persistence of the Log4j RCE exploit has serious consequences on a global scale. It’s a backdoor that hackers attempt to exploit ten million times an hour, according to a 2021 Akamai study. It’s also being exploited in state-sponsored cyberwarfare. In October 2022, the National Security Agency, the Cybersecurity and Infrastructure Security Agency (CISA), and the FBI jointly revealed the top CVEs exploited by Chinese state-sponsored actors, and the report lists Apache Log4j CVE-2021-44228 Remote Code Execution as the #1 top threat.
Reasons behind lasting Log4j
So why do so many vulnerable downloads persist? Well, the simplest reason is lack of awareness. Most consumers cannot afford to understand the intricacies of the software they use, Fox described. This awareness can be significantly increased through the use of software lists, but sharing SBOMs has not yet become a standard practice. Even if it had, most organizations do not yet have the tools and processes in place to audit SBOMs and manage them at scale.
Furthermore, almost all of these vulnerable downloads come from automated software building processes, Fox says. Machines simply download whatever version is specified in an application’s Project Object Model file, which likely hasn’t been updated in a while. (arguably, open source consumers are responsible for most of the risk if they don’t update dependencies in time.)
And when human engineers do make upgrade decisions, they do not always make the most efficient choices. Sonatype’s State of the Software Supply Chain report found that 69% of all upgrade decisions were suboptimal. The optimal zone is to live near the edge, but not on the bleeding edge since unknown harmful components can live there as well, says Fox.
Root cause solutions
Plugging the hole that is Log4j will require a continued effort to educate developers. “We have to keep talking about it, write about it and keep making the policy [fixing] it’s a standard,” Fox said. But simply patching a vulnerable package is not the cause of the larger software supply chain dilemma.
“People don’t know this is happening because they don’t know what’s in their software,” Fox said. To make open source safer, we need to raise the bar so that both those who sell software and those who use it know what’s inside it. This means raising awareness through greater SBOM use and establishing a culture around addiction management.
By making smarter and more frequent upgrade decisions, you can avoid many of the known vulnerabilities that are so often inserted into packages. Of course, each update costs time and money, and it may not be practical to update with every minor release. Still, you don’t want your dependency updates to pile up to where it takes a “Herculean effort” to update them all. Fox recommends balancing the two extremes—just as a pilot follows using his instrument landing systems, you want to stay in the sweet spot between the indicator lights.
Finally, maturing addiction treatment will undoubtedly require some dedicated tools. Because when you have hundreds or thousands of dependencies in an application, all of which change ten times a year, managing without automation becomes a headache. “You can’t manage it via a spreadsheet — that would be insane,” Fox said.
Final thoughts: Raise the profile
In late 2021, Wired announced that the Log4j vulnerability would “haunt the Internet for years.” And unfortunately, a year later, this statement seems to hold true. Since the Log4j logging system is integrated into so many applications, the exploit has great staying power. “A quarter being vulnerable is not a great place to be,” Fox said.
Bringing awareness to Log4j and similar exploits may also require fixing deeper issues, such as a lack of communication between departments or convincing developers to prioritize security in the software supply chain. “As more companies pay attention, they start answering questions. This is how we avoid this situation in the future.”
Consider the standards and regulations surrounding recalls i [manufacturing industry]. If a car component is defective or poses a safety risk, a recall is issued for all affected parts. Imagine if software vendors were required to respond to vulnerabilities in their code with the same level of persistence. “We need to raise that profile even more,” Fox said.
Do you want more insight into online security? Subscribe to the Cybersecurity as a Business Enabler channel: