FA-Bench overview

Two tasks

FA-Bench scores two tasks. They differ in where the words come from.

ASR with timestamps hears only the audio. It returns its own words, each with a start time and an end time. You need both times for a subtitle or a dubbing track, and most speech recognition APIs return them without saying how accurate they are. Each row on the ASR board is one system as shipped, an API or a model. Google Chirp 2, for example, is the chirp_2 model called through Speech-to-Text v2.

Forced alignment is handed the reference transcript. It makes no recognition error, so its timing is all that is scored. FA-Bench also hands each aligner the words Qwen3-ASR recognized. The Transcript switch above a forced alignment board shows that score. F1 Drop is how far it falls below the score on the reference transcript. MFA 3.4, for example, scores 0.658 on the reference transcript and 0.628 on recognized words, an F1 Drop of 0.030.

A cascade of a recognizer and an aligner holds the recognizer fixed, which an ASR cannot. So you find a cascade beside its aligner, and it is never ranked among the ASRs.

Test sets and conditions

Every system runs on two test sets.

You get each test set on clean audio and on four degraded copies. The copies come from the Kaldi augmentation recipes, built from fixed seeds.

A board's headline is the mean of four test cells. The cells are TIMIT core test and Buckeye test, each clean and noisy. A noisy cell is the mean of its four degraded conditions. To rank a board by any one cell, use the menu above it. Pick Buckeye test, noisy, for example, and the table shows that cell's F1 and its error rate alone.

How the score works

The figure scores one ASR output three ways. The paragraphs under it explain each one.

Word timestamp F1 at 20 ms ranks the word boards. Every reference word owns a start time and an end time. A time is a hit when the system produced the same word and placed that time within 20 ms of the annotator's. Precision counts the hits among the system's times. Recall counts them among the reference times. A misheard word costs a system both of its times. So on the ASR board, the score charges recognition errors and timing errors together. The output in the figure scores 0.45.

Word MAE and word boundary F1 score a boundary only when the words on both sides match the reference. So they leave many boundaries out. In the figure, word MAE scores 2 of the boundaries, 12.5 ms, and word boundary F1 at 20 ms is 0.15. Word timestamp F1 scores every word's own start and end. That suits spontaneous speech and its many pauses, since no boundary has to choose between the last word's end and the next word's start.

Phone boundary F1 at 20 ms ranks the phone boards. A phone boundary is a hit when the phones on both sides of it match the reference and its time falls within 20 ms of the annotator's. Each system aligns in its own phone set. So FA-Bench maps the system's phones and the reference phones into TIMIT-39 before it scores them. No ASR returns phones from audio alone, so the phone tier has forced alignment boards only.

An error rate sits beside each F1. MAE is the mean absolute boundary error in ms on clean audio. WER is the word error rate and PER the phone error rate, both in percent.

Ranges and intervals

FA-Bench draws the test utterances again with replacement, 2,000 times from a fixed seed. Every system on a board faces the same draws. The 95% interval beside each mean F1 holds the middle 95% of its scores across the draws. On the ASR board, for example, Google Chirp 2 has a mean F1 of 0.486 with an interval of 0.475 to 0.498.

Range holds the middle 95% of a system's ranks. When two systems' ranges overlap, they are tied on these test sets. On the ASR board, for example, CrisperWhisper has a Range of 2–3 and Google Chirp 3 has 2–3, so the two are tied.

The interval covers the sampling of these utterances. A new set of speakers would widen it, since Buckeye test holds 8 speakers.

Prices

The price column gives each API's list price per audio hour, pay as you go. It is the price of the model and tier FA-Bench called, read from the vendor's own pricing page on October 27, 2026. The column leaves out free tiers and credits, and it takes no volume discount. When a vendor bills batch and real-time calls differently, the column gives the one FA-Bench used.

An open model runs on your own hardware, so it has no price per hour. Olign's row gives $0.18 an hour, which is its launch promotion price. Its list price is $0.21.

Systems and versions

Each API row comes from one model called through one API version. The table gives both, and the month FA-Bench measured the API.

Deepgram and Speechmatics name a build in every response. The other APIs name none.

Current is Yes when the API still answers as it did when it was measured. FA-Bench checked this on October 27, 2026. It sent each API 5 test clips again and compared every word and every time. Two APIs had changed since September. ElevenLabs Scribe v2 had moved some word times 20 to 80 ms earlier, and AssemblyAI returned different words on 2 of 9 clips. FA-Bench measured both again in October, so every row on the boards comes from the model each API serves today.

Notes on individual systems

The marks after some systems' names point to these notes.

• ⚠ MAPS and NeuFA were trained on 7 of the 8 speakers in FA-Bench's Buckeye test set, so most of their Buckeye test audio comes from speakers they were trained on.
• ⚠ CrisperWhisper's authors used TIMIT to build its word timing and don't say which TIMIT files. FA-Bench tests on TIMIT's test files, so if the authors used those too, its TIMIT scores come from audio it has already seen.
• ‡ Deepgram ends the last word of 24% of clean Buckeye test utterances, and Chirp 3 of 15%, more than 0.3 s after the audio does.
• † NeMo-FA appears with its 80 ms model. Its 40 ms model scores higher, and the board's note gives that score.
• FALCON is handed the reference phones, which no other aligner gets, so it is left off the phone board and named in that board's note.

FA-Bench and Artificial Analysis

Artificial Analysis also ranks speech to text APIs, by its AA-WER v2 score. The table sets the two benchmarks side by side. Each column uses what that benchmark publishes about itself.

Artificial Analysis details come from its methodology page and its AA-WER v2 announcement, read on October 11, 2026.

Reproduce every number

You can check any number on the leaderboard yourself. The scoring code and the test-set recipes are on GitHub, and the paper on arXiv describes the method. Every board offers its numbers as a CSV. Run a system on your own copies of TIMIT and Buckeye, then score it with FA-Bench.

Olewave makes both FA-Bench and Olign. Olign no longer takes part in FA-Bench, so no board ranks it. Its last results, from October 2026, close each table unranked. Every system on the boards is scored by the same public code against the same human annotations.

Next steps

• See the leaderboard
  Every system ranked, with its interval and its Range.
• Get the code on GitHub
  Score your own system against the same human annotations.
• Read the paper on arXiv
  The method behind every score on this page.

← Back to the leaderboard