7 Questions to Ask Vulnerability Management Vendors

Vulnerability management is a wicked problem. And not in an endearing New England kind of way. “Wicked” is actually a recognized term describing how a problem shifts under your feet as you’re trying to solve it. In fact, the mere act of trying to solve it shifts the problem itself. 

This makes vulnerability management a moving target. And when traditional scanners and CVSS-based prioritization inflate the criticality of large sums of vulns (while understating the criticality of others), you’re faced with tackling an even more difficult task that doesn’t actually get you closer to lowering your risk. 

But there’s good news. The number of vulnerabilities that actually pose a real potential threat to your organization is much smaller than you think. Roughly 2-5% of your observed vulnerabilities are actually exploited. So the real challenge then becomes targeting these high-risk vulns without missing anything vital and without expending too much energy and resources to do so. 

This is why the solutions you choose to help you accomplish this is key. And with this narrowed problem scope, more sophisticated vulnerability management approaches become possible. 

We get it. This can be murky water to wade through. Separating sales pitches and marketing from an actual offering and technology can be exhausting. Research it long enough, and scanners or vulnerability prioritization tools all start sounding alike. 

Fortunately, we have some tips to help you understand how to measure and test a vendor’s performance and effectiveness. 

Remediation strategies: Understanding coverage and efficiency

Before you measure vendors, you must be able to measure yourself. Understanding effective remediation strategies (particularly your remediation strategy) is key. As pioneers in the risk-based vulnerability management (RBVM) space, we’ve evolved our success metrics to focus on coverage and efficiency (sometimes known as precision and recall). 

In a nutshell, coverage refers to the percentage of known risk being remediated. Of those remediated within the net you cast, efficiency represents the number that posed a real threat. For more on this key concept, check out Coverage and Efficiency of Vulnerability Remediation. 

Can you look at your own vulnerability management program and figure out the coverage and efficiency target you have? How efficient you are in achieving that coverage and can you get more efficient? 

7 questions every CISO needs to ask vendors

As you look to lower your risk in as few moves as possible, we’ve compiled some key questions to ask vulnerability management vendors. These should help you narrow your consideration set:

• What percentage of the National Vulnerability Database (NVD) do you have signatures for? First and foremost, be sure to ask a vendor this: Out of the 150,000+ vulns found in NVD, how many would they capture in an authenticated network scan today if they triggered every signature? If they don’t know, they don’t know the base rate for their scanner. That’s a red flag.
• What is your historical false positive rate for authenticated network scans? As good as scanners have gotten, network scans uncover plenty of false positives. A reputable vendor should be measuring these and able to disclose this information. 
• What percentage of vulnerabilities are returned high or critical? Every vendor should be working off some kind of model, and even better if it’s a calibrated model. In your exploratory conversations with a potential vendor, gain an understanding of what their model looks like (calibrated or not). You should be able to ask them if you returned every CVE you have a signature for, what percentage of them are high or critical? 
• On average, how long does it take from CVE release to signature generated? Time-based components impact vendors and customers. And since the NVD itself has been slowing down due to the sheer number of vulns it houses, understanding a vendor’s median delay is important to adjust operating assumptions.
• What is the efficiency of your model at 20% coverage? 50%? Consistently drill down into what efficiency looks like at different coverage levels so you can make an informed decision about risk tolerance. Every RBVM vendor should be able to provide the efficiency they’ve cut over the past year. If a vendor doesn’t disclose those numbers, either their efficiency isn’t improving or they’re just not thinking about it. Either way, it’s not a good sign.
• What is your distribution of the volume of success exploitations? How many years of outcome data do you use? How often is it updated? When you’re considering a data stream that will help you make decisions about risk, vulnerability management, and resource allocation, these are valid questions to pose. A reliable RBVM vendor should be able to answer them by quantifying exploitation data. 
• Is your model calibrated? What percentage of vulnerabilities scored 50/100 (or 5/10) are observed to be exploited? No model is perfect (ours included), but vendors should constantly question the calibration of their model. If a model is calibrated, vendors should be able to deliver retroactive data about how it’s performed over time. An uncalibrated and biased model has downstream consequences for your operations. If you’re stuck using one of those solutions, you’ll need to know about these ahead of time to tweak your remediation SLAs accordingly. 

Make measured, data-driven risk decisions

Vulnerability and risk management boils down to making decisions about prioritization, resource allocation, risk tolerance, etc. Yet as scanners increasingly become a commodity offering and fewer vulnerability data tools require authenticated scanning, the need for data-driven and measured decisions becomes paramount. 

That’s not just the approach you want to take with RBVM — it should be the approach you take in choosing an RBVM vendor.  Your search for the vulnerability management vendors you want to partner with should be thorough and exhaustive. Settling for anything less may leave you in the dark when it really matters. 

For more details on how to measure vendor performance, including a deeper dive into coverage and efficiency, a breakdown and comparison of popular RBVM platforms, and all the charts you could ever hope for, check out this webinar hosted by data science wunderkind Michael Roytman: 7 Questions Every CISO Should Ask Vulnerability Management Vendors.