Coverage and Efficiency of Vulnerability Remediation 

Buzzwords are met with skepticism because they create the illusion that a vendor is trying to “wave a magic wand”—at best, an effort to dazzle; at worst, an intentional hoodwinking. But what if we all—the creators of the technology and the people who use it—could have conversations in the same language?

To us, data science is anything but a buzzword. It’s the heartbeat of our technology—a set of methods for extracting meaning from the noise of security tooling. Make no mistake: security is a signal-to-noise problem, and most of the security tooling built to date are sensors that generate noise. We need these. They are essential. But we also need ways to extract a signal from these sensors. 

When you deploy data science techniques like machine learning, there are ways to measure results. And in the case of vulnerability prioritization, there are two metrics we can look at to help understand the performance of our model. 

Measuring the performance of a prioritization strategy

A naïve approach would look at a metric like accuracy, but in cases when the event we are trying to predict is rare, accuracy fails us. In machine learning, rarity is known as class imbalance—you have far fewer of one outcome than another in your dataset. The reason accuracy fails in these situations is best illustrated through an example:

Imagine you have to report on 100 servers to the CIO. You—the CISO—walk in every morning and say, “We’re all good.” Yet one of the machines has been compromised. In that scenario, you’d be 99% accurate in your reports, but that accuracy is not a very good measure of rare events—or just about anything in security from malware to compromise to exploited vulnerabilities that are rare. 

The first measurement we can look at is efficiency. This measurement seeks to answer the question, “If you remediate some subset of vulnerabilities, what percentage of those are ones that actually pose the risk to your organization (had an associated exploit or successful exploitation)?” The second measurement is coverage, which asks, “Of the known exploits and exploitations out there, how many would your strategy remediate?” 

If you’ve read our Prioritization to Prediction—Volume 2, these terms may sound familiar. In fact, data from the report can help us create a visualization of these two metrics (Figure 1). When we put a prioritization strategy into action, there will inevitably be a trade-off between efficiency and coverage. 

In theory, achieving 100% coverage would require a straightforward strategy: remediate every vulnerability in your business. Is that efficient? Absolutely not. (Not to mention, it’s pretty much impossible in today’s enterprises).

So, the goal becomes remaining as efficient as possible while increasing your coverage. Prioritization strategies across the market have been trying to find a way to do this. Unfortunately, the most popular strategies have a hard time striking a good balance. This is the vulnerability management problem at its core. 

As a baseline, let’s pretend our strategy is to rely on pure chance. If we randomly pick vulnerabilities to remediate, the efficiency of our model—the probability that we’ll fix something that has an exploit or successful exploitation—is about 21%. And that’s because about 21% of vulnerabilities have exploits or exploitations. If we refer back to Figure 1, we’ll see this represented by the yellow dotted line. 

The dark purple bubble represents CVSS 10+, which puts us at 22% efficiency. One percentage point better than relying on blind luck. Your coverage would have been about 7.5%. CVSS 9+, the light purple bubble, puts us about 10% higher coverage, but efficiency remains in the same ballpark. 

The blue bubble further out allows us to visualize CVSS 7+. With this strategy, you’d have 33% efficiency (much better than random) and 55% coverage. Unfortunately, this strategy would require you to remediate about half of the vulnerabilities out there. If ever there was time for a magic wand, this would be it. 

Finding the path to most efficiency

What’s important to note about the strategies plotted in Figure 1 is they are points in time—slices of the data in half that tell you to fix some things and not others, without flexibility on what to fix and the resources required. A couple of problems arise. What if 40% of the vulnerabilities are on one side? You don’t have a magic wand to wave and remediate them—it might take months, or even years. What’s worse, as you’re waiving this wand, new vulnerabilities and machines are coming out or online, so you have to reevaluate on a weekly basis regardless. But what if we could create a model that allows us to take a dynamic approach? 

Let’s visualize this by adding an element to our graph (Figure 2 below). Here, the new red dotted line represents a flexible strategy—in this case, Kenna’s own model. 

Vulnerability management vendors often aim to do exactly this, and it can be hard for buyers to truly understand if the vendor’s prioritization strategy is actually improving upon CVSS or random chance. That’s why it’s important for buyers to be able to ask the right questions. For any vendor—particularly those claiming to leverage machine learning—you should be able to ask critical discovery questions, such as:

• What is the coverage and efficiency of your model?
• What is the tradeoff in efficiency as coverage increases?

That holds true for our customers, too. At around 20% coverage, Kenna’s (now Cisco’s) model has +95% efficiency. At 55% coverage (matching CVSS 7+), it yields 85% efficiency. 

The benefit of this strategy is you can create a path of most efficiency for any coverage level. And the only reason we can do this is that we have a unique data set behind our model, and that sounds like a great topic for the next blog. 

Until then, I encourage you to dig a little deeper into our Prioritization and Prediction research—if you haven’t already. And I’ll end this blog with the same call to action as my previous one: Ask us about our data science and don’t be afraid to ask other vendors, too. Ask these questions, and any others you can think of. 

This blog was originally written for Kenna Security, which has been acquired by Cisco Systems. Learn more about Cisco Vulnerability Management.