Patching – one of the surefire ways to help your organization mitigate the risk of being compromised due to software defects or security weaknesses. As security professionals, we’ve seen the gambit when it comes to patching, on one end of the spectrum there are organizations where half or more of their servers haven’t been patched in years and on the other end there are those where they validate build specs and spin up new servers multiple times a day.
When vulnerabilities are exposed, malicious actors can take advantage of bugs and create havoc. New vulnerabilities are discovered every day, but organizations that have failed to establish rigorous patch and vulnerability management processes over the years are getting hit the hardest. As Verizon's 2016 Data Breach Investigations Report says, “Older vulnerabilities are still heavily targeted; a methodical patch approach that emphasizes consistency and coverage is more important than expedient patching.” If we all know these things to be true, why don’t we patch, Patch, and PATCH ‘til the cows come home?
So What Is This "Patching" You Speak Of?
Patching is when a piece of software needs an update to fix a defect, tighten up security, and/or improve usability or performance. Unpatched applications and software often open the door to a myriad of risks and vulnerabilities. Patching helps you abide by industry best practices while also complying with any regulatory requirements (i.e. PCI DSS) that might be applicable to your organization. Patching also helps ensure your systems are up to date with the most recent protection against possible malware or unauthorized intrusions. You might be thinking, this seems like common sense and a best practice so why don’t more organizations practice patching more consistently? Well, the short answer is that it doesn’t take very much organizational and architectural complexity to make patching a nightmare. Scenarios you’ve probably encountered in your organization include:
• Servers X, Y, and Z can’t be patched because the latest patch breaks the Widget applications running on those servers.
• The Widget applications cannot be upgraded because the organization is in the process of migrating to an enterprise application to provide those capabilities and, of course, the massive migration project is running behind schedule.
• And just for fun, let’s toss in an asset list of questionable accuracy and non-standard builds – making patch automation near impossible.
Let’s face it, identifying and analyzing risks, dedicating manpower to patching (as opposed to building the shiny new stuff), and following a standardized process can be really hard work. Let’s take a closer look at the most common patching challenges.
What Are Some of the Most Common Challenges When It Comes to Patching?
Patch management is one of the most difficult challenges facing security professionals for a myriad of reasons. Just in the last year, there has been an increase in organization’s allowing employees to bring their own devices (BYOD), applications, vulnerabilities, along with a shortage of security professionals available to do the work. According to a study by ISACA and the RSA Conference, 65% of entry-level cybersecurity applicants lacked the requisite skills for the position.
Below are some additional challenges your organization might face:
Patching ain’t cheap!
It takes time, manpower, tools, and money to scan for vulnerabilities and come up with patching solutions. Like other software changes to your environment and systems, patching should follow your organizations testing and change management practices. If an organization is on a monthly patching cadence this might mean that they only have a couple of weeks to test the patch in their test environments (to make sure the patch will not break something else), get approval for the change, deploy the patch into production, and verify that the patch was successful before the next cycle starts all over again.
Some organizations just don't have the resources to dedicate to patching, especially when they have a complex environment and non-standard builds that require manual patching. It’s not always a straight forward process as you might think it would be – it takes time, money, and effort.
We all know what they say about risk… your biggest risk is how well you do risk management!
As with most things in the realm of information technology and security, patching vulnerabilities should be risk-driven to ensure that your limited resources are focused on the highest risks. We all know that assessing risk is not a one-time deal – it is a continuous process. Vulnerabilities are discovered from a variety of sources such as penetration tests, network and system scans, identified system defects, vendor reports, and other sources such as National Institute of Standards and Technology (NIST) and SANS.
Quite often, the test and scan results provide raw vulnerability scores based on the Common Vulnerability Scoring System (CVSS). But organizations must analyze these risks in the context of how likely they are to impact that organization. For example, a CVSS score might be high, however in the context of things like the architecture/business process, the criticality of the system/data it handles, and the other controls in place, the actual risk might be low. In-place or additional mitigating controls might further offset the risk; however, adding mitigating controls might be more cost and effort than applying the patch. It’s all part of the dance, and you need to find the balance to stay in step with risk, cost, and your organizations business and architectural direction.
For example, a scan result labels an authentication scheme for a web application with a “High” CVSS score. Subsequent analysis indicated that the authentication scheme in question only allowed access to the content area of the web app and it was not possible to escalate privileges to other areas of the app. The worst that could happen is some unauthorized content could be changed, especially considering the organization had monitoring in place to alert of any unauthorized changes as a further mitigating control. In the context of that particular organization and the controls in place, this vulnerability was re-cast as “Low”.
Another important aspect of risk management is reporting in terms of Key Performance Indicators (KPIs), these include coverage, effort, speed, quality, impact, automation percentage, etc. Key Risk Indicators (KRIs) are another important factor to report on, these include vulnerability ranking percentiles, missing update check frequency, asset inventory update frequency, etc. Reporting on these allow the various stakeholders in your organization (i.e. Senior IT, Operations, and Risk Management) to understand how well the patch cycle is working in terms of effectiveness for use of resources as well as mitigating risk.
Hey you, get off of my cloud!
Quite often, your cloud application and infrastructure vendors or providers are responsible for patching servers and applications. As with many other aspects of working with cloud providers and other partners, the key is being clear in your service agreements regarding the particulars of patching responsibility, the patching cycle and process, and establishing ongoing communication channels to stay in sync. In some scenarios it may make more sense to spin up new servers rather than patch. This could be particularly true in Linux environments where there are environment management and automation tools such as Puppet and Chef (note, this is also applicable to Microsoft environments, though not as common).
Fight the Good Fight with a Patch and Vulnerability Management Program
Patch and Vulnerability Management Programs might seem daunting, but with the right approach they can save you time and money while reducing your risk exposure, which pays dividends when it comes to avoiding system compromises and regulatory requirements violations. A successful patch and vulnerability management program starts with a solid policy upon which, over time, defined processes and rules of engagement are established with your other security, IT, and business processes, including: Service Management, Infrastructure Services, DevOps, Systems Development Lifecycle (SDLC), etc. Having said that, you don’t have to boil the ocean to get started. Start with a simple, targeted process, apply continuous improvement and expand at a reasonable rate.
Patch and Vulnerability Management 1-2-3
1 – Write a Policy and Establish a Basic Process
Everything starts with policy. Write a high-level policy that establishes the tone and goals of your patch and vulnerability management program and upon which you can then develop your processes. Developing your process is instrumental to efficiently identifying and assessing risks and vulnerabilities, aligning these risks with the most recent patch releases as they apply to your asset inventory, determining which patches to go with for the current cycle, and testing and deploying those patches. Your process should be risk-driven and based on a cadence that makes sense for your business and your architecture. Typically, the ideal cadence is going to be on a monthly cycle that aligns with your vendors’ patch release cycles. Start with a simple, targeted scope. The main activities in your process should include these fundamentals:
• Discovery (and Re-Discovery)
• Reporting and measurement of business risk
• Prioritization
• Response
• Vulnerability remediation
• Remediation validation
2 - Apply Continuous Improvement and Expand
In the early days of your process, you should review your processes after each cycle with a focus on improving efficiency, and discuss the next scope of your environment to pull into the process. Over time, your continuous improvement reviews can become less frequent, but you should always apply continuous improvement to your processes. Small improvements over time can lead to an impactful reduction of organizational risk, fewer wasted resources, and reduced expenses.
Be careful to avoid latching onto a fancy tool before you understand what your process looks like. It’s a common pitfall and it’s easy to get caught up in the bells and whistles of the tool and become distracted from solidifying your process. But when the time is right, take advantage of tools to stay ahead of the curve and help improve your process.
3 - Automate Your Process
In the beginning, you may find that manual execution of certain tasks is unavoidable. However, as you continually improve your process, it might make sense to look at automation tools to help you streamline routine tasks. Automation can save you time, money, and manpower. There are a number of tools that help you process your scan results and manage vulnerabilities; others that plug into your service management tool for end-to-end remediation tracking; and others that help automatically deploy and validate patches. As mentioned earlier, the key to automation is standard processes and builds!
Don’t Get Caught with Your Patch Down
As with most processes, one-size-does-not-fit-all. You need to develop a customized patch program and cadence that works for your organization and its many needs and requirements. Having said that, there are fundamentals and frameworks that you can use as a starting point. In large, complex environments, patching can be a very daunting undertaking. We’ve found that when working with our clients to get their patching process under control, the key is to start simple and establish a solid process, then expand and automate as you go.
To continue the conversation, send me a message!
Additional resources:
NIST SP 800-40 r3 and the SANS Reading Room are great resources to help guide the development of a mature patch and vulnerability management program.
Learn more about Online Business Systems’ Risk, Security and Privacy practice by clicking here.
Submit a Comment