Oleksandr Kuzmenko, EditorPublished 4 min readAI-assisted · editor-reviewed
TL;DR
- The Open ASR Leaderboard now includes private held-out test splits for Egyptian and Nigerian English.
- Liva AI provided natural conversational data collected from real consumer application usage.
- Results are mapped to Kachru's Three Circles to benchmark native versus global English accuracy.
Key facts
- New Accents Added
- Egyptian English and Nigerian English
- Evaluation Framework
- Kachru's Three Circles (Inner vs Outer & Expanding)
- Data Source
- Liva AI real-world product interaction data
- Prior Private Splits
- 11 splits from Appen/DataOceanAI plus Voice Arena
Held-Out Evaluation for Accented Speech
Hugging Face has updated the Open ASR (Automatic Speech Recognition) Leaderboard with private test splits for Egyptian and Nigerian English. Sourced in collaboration with Liva AI, the new evaluation sets capture natural conversational audio from live consumer applications rather than synthetic speech or scripted reading tasks. Held-out splits prevent "benchmaxxing," ensuring published metrics reflect real-world generalization rather than test-set memorization.
Kachru's Three Circles Benchmarking
The platform categorizes transcription performance using Braj Kachru's Three Circles model, contrasting Inner Circle native variants (US, UK, Canada, Australia) with Outer and Expanding Circle dialects (Egypt, Nigeria, India). For engineering teams building voice agents and real-time transcription pipelines, this granular split reveals domain-specific word error rate (WER) degradation, allowing developers to stress-test multimodal models before deploying to global end users.
When to use
- Selecting or fine-tuning speech-to-text models for international users outside North America.
- Auditing transcription error rates against held-out private test sets to avoid benchmaxxing.
When NOT to use
- Building strictly text-based LLM pipelines that do not ingest voice audio.
- Applications targeting exclusively domestic US English native speakers.
What to do today
- Inspect the Open ASR Leaderboard Private tab before picking a production speech-to-text model
- Compare word error rates between Inner Circle and Expanding Circle splits for your target locales
- Test your voice agent prompt fallback when non-US transcription confidence drops below threshold
Step names are taken from What to do today.
Step names are taken from What to do today. Inspect the Open ASR Leaderboard Private tab before picking a production speech-to-text model. Compare word error rates between Inner Circle and Expanding Circle splits for your target locales. Test your voice agent prompt fallback when non-US transcription confidence drops below threshold
- Inspect the Open ASR Leaderboard Private tab before picking a production speech-to-text model
- Compare word error rates between Inner Circle and Expanding Circle splits for your target locales
- Test your voice agent prompt fallback when non-US transcription confidence drops below threshold