FA-Bench: A Benchmark for Evaluating Phone- and Word-Level Timestamp Accuracy in Forced Aligners
Most forced aligner benchmarks aren't reproducible. FA-Bench establishes a standardized baseline — phone- and word-level boundary accuracy against hand-labeled ground truth, on clean audio and four degradation types.
Most forced aligner benchmarks aren't reproducible. Differing phoneme sets, custom test splits, and moving metrics make direct model comparisons nearly impossible — which leaves speech researchers unable to run clean ablation studies, and voice-AI engineers unable to tell which system to actually use.
We created FA-Bench to establish a standardized baseline.
What we tested
- 15+ aligners and timestamped ASR systems
- Phone- and word-level boundary accuracy against hand-labeled ground truth
- Performance across clean audio + 4 degradation types — reverb, noise, music, babble
Scoring runs on read speech (TIMIT) and spontaneous speech (Buckeye), using each corpus's own boundaries with no new annotation — times as annotated, labels folded to the shared TIMIT-39 set. Results are split into two tracks that never share a leaderboard: aligners are given the reference transcript, while timestamped ASRs decode their own words, so their timing error carries recognition error too. Everything is seeded, flag-gated, and reproducible from a single command.
Key takeaway
Olign 1.0 (Beta) outperformed both MFA and WhisperX in baseline accuracy — and demonstrated significantly higher noise robustness under heavy degradation.
High accuracy on studio audio is table stakes. High accuracy on real-world audio is what matters.
The systems
BFA · Charsiu · CrisperWhisper (FA) · MAPS · Montreal-Forced-Aligner (MFA) · Olign · Parakeet-TDT · Qwen3-ForcedAligner · stable-ts · TorchAudio (FA) · WhisperX
Olign is in beta and access is by request — it needs credentials from Olewave (info@olewave.com); every other system installs from its own recipe and needs nothing from us.
Metrics
- Boundary MAE / median (x̃) / signed (δ̄), dual-edge, on the matched path only, with bootstrap 95 % CIs; threshold accuracy at t=10 / 50 / 100 ms — the tail-sensitive companion to MAE.
- S / D / I and PER — why a gold phone left the matched path: relabelled,
never emitted, or invented. The anti-gaming guard: a system that skips hard
phones to flatter its MAE shows up in
D. - Boundary detection @20 ms — P/R, F1, over-segmentation and R-val. Paired by time and blind to labels, so a substitution at the right moment is free here — which is exactly what the matched-path metrics above cannot see.
- The word tier carries the same set at the same 20 ms tolerance — MAE, P/R, F1, OS, R-val — and is the only phone-independent one, so it is the only place a word-only system appears at all.
Every table spans clean and the four degradations on one row.
Full results
Benchmark code and dataset splits are live: TIMIT (read US English) · Buckeye (spontaneous English) · Methodology
FA-Bench ships manifests + recipes, never audio. TIMIT (LDC) and Buckeye (OSU registration) must be obtained under their own licences and staged by you.
License: PolyForm Noncommercial 1.0.0 — free for any noncommercial purpose (research, teaching, personal study, public-interest and government work). Commercial use requires a licence from Olewave, LLC.