Abstract

Varonis commissioned Tolly to evaluate the accuracy of Varonis Interceptor for business email compromise and advanced phishing protection against leading email security providers. The main focus of the project was to compare detection efficacy across four major email threat vectors—BEC or plain-text attacks, QR-based phishing, link-based phishing, and file-based phishing—using zero-hour malicious emails in a Microsoft 365 environment.  

The report compares Varonis with Microsoft Defender for Office 365 (E5), Mimecast, and Abnormal Security. Testing used 253 malicious emails selected from recent OpenPhish submissions and the Varonis phishing threat intelligence database, with all services receiving messages simultaneously via blind copy. According to the summary chart on page 1, Varonis achieved a 99% overall detection rate, compared with 76% for Abnormal Security, 61% for Mimecast, and 23% for Microsoft Defender. Tolly notes that this testing was originally conducted in 2024 and first published as report #224112.  

A key technical point in the report is that strong cloud email security must perform well across all four threat vectors rather than only one or two. The attack corpus included BEC and plain-text fraud, QR-based credential phishing, link-based credential phishing and scams, and file-based threats such as malicious PDFs. The chart on page 2 shows Varonis posting the highest detection rate in every tested category, including 100% detection for BEC attacks, 100% for QR-code threats, 98% for link-based threats, and 90% for file-based threats. By comparison, Abnormal Security reached 89%, 78%, 73%, and 55% respectively; Mimecast showed 0%, 0%, 94%, and 40%; and Microsoft Defender reached 4%, 80%, 21%, and 44%.  

The evaluation was conducted in late January 2024 using Microsoft Graph API integrations for Varonis, Abnormal Security, and Mimecast, with Microsoft Defender tested in a Microsoft 365 E5 environment using Safe Links with URL rewriting and real-time URL scanning enabled. Tolly examined both user-facing folders and vendor management portals, and the management portal status was used as the final determination of whether a message had been detected. Benign messages were interspersed during testing to avoid sending an uninterrupted stream of malicious-looking traffic. Overall, the report presents Varonis Interceptor as the strongest performer in this comparative test, particularly for zero-hour BEC, QR-based phishing, and mixed advanced phishing attacks that can evade traditional signature- and reputation-based email defenses.