Abstract

Checkmarx commissioned Tolly to evaluate the Checkmarx One Platform against a competing application security solution, with the main focus on comparing Static Application Security Testing and Software Composition Analysis efficacy. The project examined how accurately each solution identified real vulnerabilities while minimizing false positives and false negatives across multiple open-source applications used as reproducible test codebases.  

The report explains that SAST and SCA address different parts of the application risk surface. SAST focuses on proprietary or custom code, where vulnerabilities are not already catalogued in public databases, while SCA examines open-source components for known vulnerabilities, aging packages, and license issues. For this evaluation, Tolly reviewed results from SAST scans on three projects and SCA scans on two projects. Checkmarx analysts classified each result as a true positive, false positive, or false negative, and Tolly spot-checked the findings with an in-house application security expert. According to the report, Checkmarx produced better results in both SAST and SCA, with higher true positives and lower false positives and false negatives than the competing solution.  

In SAST testing, Checkmarx identified 1,261 potential vulnerabilities across the three applications, compared with 611 from the competitor. Of those, 803 were validated as true positives and 458 as false positives for Checkmarx, while the competitor produced 237 true positives and 374 false positives. False negatives were also markedly different: 83 for Checkmarx versus 649 for the competitor. Tolly emphasizes that false positives alone do not tell the full story, because a solution can look conservative while still missing many real issues. Using precision and recall as a better overall view of accuracy, the report states that Checkmarx achieved a recall of 0.91 versus 0.27 for the competitor and a precision of 0.64 versus 0.39.  

The SCA results also favored Checkmarx. Checkmarx identified 57 vulnerabilities across the two tested applications, all of which were validated as true positives, resulting in zero false positives and only eight false negatives. The competitor identified 39 vulnerabilities, of which 35 were true positives and four were false positives, while missing 30 vulnerabilities. Tolly notes that Checkmarx identified more unique CVEs, more packages, more direct dependencies, and more unregistered vulnerabilities, suggesting deeper inspection of the source code. The report also highlights Checkmarx’s exploitable path capability, which correlates SAST and SCA findings to determine whether vulnerable library code is actually reachable from the application. In this test, Checkmarx identified significantly more exploitable paths than the competitor, improving prioritization of remediation. Overall, the report presents Checkmarx One as a more accurate and more actionable application security platform for both custom code and open-source dependency analysis.