Is Log4Shell Fukushima or the tip of the iceberg?

The answer is both! There is already so much written about the recent Log4Shell vulnerability* that trying to add more technical details doesn’t seem like the best use of time. 

That said, I do think that there is value in providing a concise summary of what’s happened, along with some of the best resources we’ve identified to help our clients understand, identify, and remediate the issue.

*On December 9, 2021 the Log4Shell vulnerability was publicly disclosed on GitHub.

Log4Shell is a remote code execution (RCE) zero-day vulnerability (CVE-2021-44228) that was discovered in Apache Log4j, a widely used Java logging library. This vulnerability enables threat actors to take full control of servers without authentication.

Let’s start by taking a quick look at explaining the vulnerability, and then explore the broader issue of why this vulnerability is going to be so difficult to fully eradicate, why this issue is likely just the tip of the iceberg, and what you should be doing over the medium and long term to help reduce your risk to this and similar vulnerabilities in open-source platforms. 

Short and Sour

The Log4Shell vulnerability is borne from the need for consistent and easy-to-use logging by applications, as well as the desire by developers to make their software ever more extendable.

The Apache Log4j is a popular Java library developed and maintained by the Apache foundation and is included in many applications that have very wide use, such as the Apache HTTP server. It is also used by countless other custom Java-based applications developed by organizations because they all need logging; log4j represents probably the shortest path to that end.

The irony here is that logging in applications is a critical component of security, but when implemented without using secure code paradigms that have been in place since the early 2000s, you end up with a nasty vulnerability that is going to be around for a while.

 One of the most concise and clear visuals of the exploitation attack pattern I have seen, is on this is Fig. 1 in a post by Juniper networks:

https://blogs.juniper.net/en-us/security/apache-log4j-vulnerability-cve-2021-44228-raises-widespread-concerns.  

The exploitation follows these steps:

1.  User-controlled input (example: user-agent header value) is submitted to a vulnerable Apache HTTP server and includes a specially crafted string with JDNI reference (to an attacker-controlled server). 
   
   
   
2. When the user-controlled input is logged, the vulnerable server requests the referenced JNDI resource on the attacker-controlled server, which then replies with the location of the exploitation payload class. 
   
   
   
3. The vulnerable server then requests the exploitation payload class from the attacker-controlled server, which then responds with that class, which is then loaded and run by the vulnerable server, and the exploitation is complete.

Mitigation

One of the better visualizations of many of the ways you can mitigate the Log4Shell vulnerability is in a post by the Swiss Government Computer Emergency Response Team referenced here.

They reference each of the steps of the exploitation and indicate preventative controls/actions. While the recommendations are expanding almost by the minute as more and more information becomes available, this is a good example of the different types of approaches that can be taken, as some may be more difficult for organizations to implement than others.

Discovery

You may have noticed we haven’t talked about discovery. And with this vulnerability, the biggest challenge will be identifying all the places that the issue may exist. While you can test for the vulnerability by scanning with Dynamic Application Security Tools (DAST), at best, you will identify some of your most vulnerable services. But this type of detection is not going to be very robust as it is somewhat of a ‘spray and pray’ approach since there is no real way to know what is being logged. 

Why might you use this approach? 

The best reason to use DAST is to determine what is available to attackers who will likely be using this same approach to try to identify vulnerable servers. But after trying to identify the systems most at-risk to remote attackers, you will want to take a much more thorough approach to ensure that all their environments are free from vulnerability.

For most organizations, this will include some form of searching for the instances of the lib4j library on each server/instance. There are already a number of tools built specifically for performing these searches. The company that first coined the name Log4Shell is a good example, which can be found here.

The most fortunate of organizations will already have ongoing network asset discovery in place and will use the asset management and software inventories they have established to quickly identify the vast majority of vulnerable systems within their environments.

Impact

Now to the impact. We mentioned Fukushima, the Japanese nuclear plant that was compromised by an earthquake/tsunami. In a post by Tenable co-founder Renaud Deraison, he points out that – like the radiation and fallout from Fukushima – issues will continue to emerge from this vulnerability for a long time. 

We expect to see similar fallout with the Log4Shell vulnerability; not everyone will do a good job of discovery in their environments. There are also many systems that will not be easy to do the discovery on – printers, video cameras, and other systems that should get regular updates, but often don’t. Many of these devices will not have an obvious attack path, so they may not even be considered when an organization scrambles the resources to try to discover all the systems that use the vulnerable library. Not everyone knows that most printers today have their own web servers, are based on some version of Linux, and are loaded with open-source software. 

So as mentioned in the Tenable post, there will likely be a very 'long tail' of systems that are slowly but surely identified as being vulnerable. For those of us in the business of performing Penetration Testing, we should expect to be very busy for the foreseeable future.

Tip of the Iceberg

If we consider open-source software an iceberg, Log4Shell is the tip you see above the waterline. Organizations rely on millions of lines of open-source code running on countless systems every day. It would be misleading to say this software is any more vulnerable than code developed by commercial companies, but it would also be misleading to say it is any safer. Let’s face it, if there is code, there will be flaws. And if there are flaws, some will be security vulnerabilities.

The biggest issue with most open-source software compared to commercial software is that most organizations feel that the security of the open-source software is not their concern. While they can and do hold commercial companies to vendor risk assessment standards and require code reviews and/or penetration tests, they usually exclude the same level of scrutiny of open-source code, with the single biggest reason being the cost. After all, an organization can pay a developer to write a very capable small application with a few hundred lines of code that takes advantage of millions of lines of open-source code.

In this case, should the organization take on the responsibility of all the open-source code to enable them to write the small application? Some may say yes, but of course, that isn’t really practical; and right or wrong, if it isn’t justifiable cost-wise, it won’t be done.

A better approach could be to share the responsibility across all the users of open-source code. This is the Wikipedia approach. But then, when was the last time you donated to Wikipedia? I have had an increase of notifications lately, even after donating, so my guess is they are having issues with this model.

Renaud mentions another possible solution in his post, “We’re long past time for the creation of a security classification system for open-source code libraries.” This is a great idea in theory, but without a sponsor, with deep pockets, it is likely to sit alongside any number of the other potentially good ideas that will never be realized. 

Back to the Beginning

In the opening statement, we said that we’d try to help you look beyond the Log4Shell issue to help organizations better prepare for the rest of the iceberg you can’t see. After all, there will be a big scramble to identify and fix this latest vulnerability, and all the ‘tool vendors’ will surely be flooding your inbox with webinars and advertisements that outline their solutions.

But what about the future? What can you do to help prevent being in this situation next year or the year after that? Security is never static, and it is never too late to validate your security posture and make sure you have a healthy security program within your organization.

Here are a few key ways to get started:

1. Discovery: Make sure you have an up-to-date list of all the assets running in your environment. You can’t protect, what you don’t know about. Completing a proper Discovery and Asset management scan can go along way in closing gaps in your understanding and mobilizing your security efforts. You can learn more about Asset Discovery here.
   
   
   
2. Check the Cloud: Complete a Configuration Review of your Cloud Services to validate your risk and enable you to manage security considerations of the Cloud. Check out Online's Cloud Services Capability Overview.
   
   
   

3. Secure Development: Analyze your application source code for vulnerabilities and ensure your development teams are using secure development best practices as they build and maintain software. Take a look at our Secure Code Analysis and Review Services here.

If you have any questions about how Log4Shell might be impacting your organization or are simply unsure where to start – don’t hesitate to reach out. We are always happy to chat.

About Will Bechtel, Director, Technical Security Services

As Online's Director, Technical Security Services, Will manages our Technical Security Services team that performs hundreds of technical security assessments each year for over 100 different Clients.

Prior to coming to Online, Will held positions as the Director of Product Management for the Web Application Scanning and Malware Detection Service at Qualys, Application Security Practice Lead for AT&T's Security Consulting and Sr. Consulting Manager in the Application Security Practice for VeriSign's Global Security Consulting.

To contact Will or our Risk, Security, and Privacy team please email rsp@obsglobal.com.