Abstract

The Tolly Group authored this PoC Best Practices document to outline how organizations should evaluate AI-assisted transcription, with the main focus on building repeatable, business-relevant tests for transcription accuracy before deployment. Rather than promoting a single product, the document explains how to define scope, choose realistic variables, set up an audio test environment, and measure results in a way that reflects actual business use cases such as finance, healthcare, meetings, customer support, and collaborative applications.  

A central message in the report is that transcription accuracy cannot be assumed simply because AI transcription is bundled into a larger platform. Tolly argues that word-level accuracy is the foundational metric because poor word recognition will undermine any higher-level conversation summary. The document identifies major variables that can significantly affect results, including language, regional accent, technical vocabulary, speech velocity, vocal timbre, background noise, acoustic conditions, overlapping speech, speaker diarization, and language switching. Tolly notes, for example, that some systems struggle with heavy Scottish accents and that specialized terms in medical or financial contexts must be transcribed correctly for the solution to be acceptable.  

The report recommends using recorded speech for repeatability and suggests either human recordings or text-to-speech tools to generate consistent samples across accents and languages. It also highlights the need for audio-routing utilities that can feed prerecorded audio into the microphone input of the application under test. Suggested tooling includes commercial text-to-speech and routing options such as Amazon Polly, Google Cloud Text-to-Speech, Azure Speech Services, IBM Watson Text to Speech, Loopback, and VB-Audio Virtual Cable, as well as open-source alternatives such as eSpeak NG, Coqui TTS, BlackHole, JACK, and PipeWire.  

For measurement, Tolly recommends Word Error Rate as the primary metric and provides guidance for interpreting results: lower than 5% WER is rated Excellent, 5% to below 10% Good, 10% to below 20% Fair, and above 20% Poor. The document also recommends running each sample multiple times because the same system may produce different transcription results from the same audio on different runs; three runs is the minimum recommendation, while Tolly’s own procedure used four runs per sample. Overall, the document presents a practical framework for evaluating AI transcription in a disciplined, repeatable, and application-specific manner.