FPs are Cheap. Show me the CVEs!

The Open Source Security Foundation has developed a benchmarking dataset and tooling to evaluate the effectiveness of SAST tools in detecting real vulnerabilities using CVEs.

• Traditional synthetic test suites are limited in their ability to accurately evaluate the effectiveness of SAST tools in detecting vulnerabilities.
• The Open Source Security Foundation has developed a benchmarking dataset and tooling that uses real vulnerabilities identified by CVEs to evaluate the effectiveness of SAST tools.
• The benchmarking dataset includes metadata such as the fix commit and code locations for each vulnerability, and a CLI to feed the CVEs to security tools and generate a report on their performance.
• The benchmarking tool evaluates the sensitivity and specificity of the security tools in detecting vulnerabilities and recognizing patches.
• The benchmarking tool runs on the user's machine and provides a more accurate evaluation of SAST tool effectiveness than traditional synthetic test suites.

The speaker describes the frustration of security teams and developers when security tools flag hundreds or thousands of false positives, making it difficult to identify real vulnerabilities. The benchmarking dataset and tooling developed by the Open Source Security Foundation aims to address this issue by using real vulnerabilities identified by CVEs to evaluate the effectiveness of SAST tools.

SAST tools are notoriously hard to evaluate and benchmark. The most important thing you want to know about a tool before spending time and money on it: does it give me relevant results? Does it really find the vulnerabilities it promises? Vendors are quick to tell you that their technology will find every vulnerability category out there, and claim to cover every CWE under the sun. But, how do you verify such bold claims? How many vulnerabilities will their tool really uncover, and how many frustrating false positives will you have to trawl through?We've all been there: planting mock vulnerabilities in our code bases to challenge a SAST product. It takes a lot of time, and it really only gets you a synthetic set of vulnerabilities to test against. Or you might run tools against one of the many synthetic benchmarking repositories that are riddled with vulnerabilities. Deep inside you know that those codebases have aged and don't really test coverage for modern web frameworks, and rarely test for vulnerabilities that arise due to complex interplay between dependencies and your own code.If only we could test tools against *real* vulnerabilities! But hold on… We carefully give every major security vulnerability a globally unique CVE identifier and a collection of metadata. Why not use those! We've triaged hundreds of CVEs in open source codebases and identified the fix commit(s) for every single vulnerability. At Black Hat Europe, we will release this benchmarking dataset and tooling to the open source community.This is an initiative by the recently founded Open Source Security Foundation, a part of the Linux Foundation. The working group in which this initiative was developed includes partners from GitHub, Google, Microsoft, Mozilla, and OWASP.