Two years ago, many of us in the Java community woke up to a “December to Remember” thanks to a remotely exploitable vulnerability in the Log4j logging library. Many development and security teams spent nights and weekends hunting down vulnerable versions of Log4j – removing it from builds, scanning containers and VMs, and implementing patches. Despite these efforts, Veracode reports that more than a third of Java applications still use vulnerable versions of Log4j.
Most Java engineers now feel that this problem should be handled. We’ve updated our pipelines, changed our builds, and applied the patches. Unfortunately, this is not enough and why we should still be frustrated for two reasons:
- The security tools that scan software are still so noisy that many groups tune them out. The tool always says everything is vulnerable, so why should we bother?
- Although patches may exist, they aren’t always applied or the vulnerable version (without the patch) is still widely deployed.
When a security scanner says Log4j is present, that doesn’t mean that the application is actually vulnerable. While some are, many Java applications contain unused dependencies. An unused dependency is a library that is present in the application but does not actually load or execute. While scanners are not wrong to report that the vulnerability is present, most composition/vulnerability management analyzers cannot differentiate between what code is present and what code is used. During the first December of Log4j, this inability was unfortunate because it sent many patch teams on a wild goose chase to find Log4j in both affected and unaffected places. One common incorrectly flagged library was Quarkus, where enough issues were raised that the dev team published a blog post to clarify that no, their applications are not affected.
Since Quarkus, its extensions, and dependencies do not use the log4j version 2 core library, it is NOT susceptible to this vulnerability… does expose the log4j API jar which in itself is not vulnerable. This is purely a compatibility and translation layer, which maps calls to a different logging backend (JBoss Logging). Therefore, any direct usage of the log4j API is not impacted…
Still tools that do security scans are not smart enough to discern between a transitive dependency and what actually ends up bundled and/or configured in your application. Thus we do see reports from such tools falsely claiming Quarkus or some of its extensions in its ecosystem are affected although in practice they are not.
Quarkus Is Not Affected By the Log4j Vulnerability, December 15, 2021
If scanners want to indicate that Java applications are vulnerable, they need to perform more fine-grained analysis to determine if that code is actually used. In Java applications, Azul JVMs can extract this information by watching the behavior of ClassLoaders to determine if a particular library is in use. This is not a hypothetical scan view, rather it demonstrates that the JVM is loading and initializing that code.
Alert fatigue is extremely annoying
According to the Orca Security 2022 Cloud Security Alert Fatigue Report, alert fatigue is creating real problems for security teams.
- 59% of security teams receive more than 500 cloud security alerts per day
- 43% say more than 40% of their alerts are false positives
- 49% say more than 40% of alerts are low priority
- 56% spend more than 20% of their day reviewing alerts and deciding which ones to deal with first
The overwhelming number of alerts is highly disruptive for teams, careers, and business outcomes.
- 62% say that alert fatigue has contributed to turnover
- 60% say alert fatigue has created internal friction
Of the 55% of respondents who say that critical alerts are being missed:
- 41% say alerts are missed on a weekly basis
- 22% say alerts are missed daily
It’s not that the impact of Log4j isn’t real or worth paying attention to. The North Korea-based group Lazarus Group uses Log4j exploits to deploy remote-access trojans. It’s not that the industry can’t find Log4j and isn’t aware of it either. It’s that teams are forced to skip around between so many unfiltered alerts that they can’t get things done.
Monitoring production stacks
Another need for teams is production monitoring. Detecting and patching in a build phase only works if that secured version is rolled out. One thing we learned from Veracode’s Log4j report is that vulnerable versions of Log4j are still being deployed – sometimes by customers, and sometimes by automated pipelines redeploying old versions without doing another security check.
The two key benefits to monitoring production environments are:
- They have the context to differentiate used versus present. If the JVM has but does not load a vulnerable Log4j, this is where that can be tracked.
- Production tracks what is actually used versus what teams think is used. Many engineering teams think they are done with Log4j, but it keeps re-appearing.
Azul Vulnerability Detection
Azul Vulnerability Detection was designed to address engineering teams’ frustration with security scanners. Rather than reporting entirely on what’s present in applications, the JVM already knows the classpath details and is responsible for loading code.
By monitoring load events, the system can differentiate between Java dependencies that are present but not actually used by the application, saving teams time the need for urgent action every time a new security alert comes in.
Of course we would like to see less of those vulnerable Log4j versions, but just because it’s present doesn’t mean the application is vulnerable.
