Baian vs other AI
Ranked against OpenAI, Google and ElevenLabs.
First, Baian has to hear the words correctly. These public tests compare that step: turning speech into text. We ran Baian on the test sets used to measure other speech AI.
See the resultsLive speech: transcription rankingsErrors in words, or characters for Korean and Japanese. Lower is better.
-
ArabicLive
Baian5.1%1stSpeechmatics Enhanced6.5%2nd -
KoreanLive
Baian2.9%1stGoogle Gemini 3.5 Live3.8%2nd
Do these numbers measure translation?
No. They count differences between the recognised text and the test’s reference text. Baian then uses recent speech as context when translating. A transcription error does not, by itself, tell you whether the translated meaning is wrong.
See the translation testsRecorded speech
First in five more languages.
French and Japanese are shared first places. These tests compare transcription from finished recordings.
Recorded speech: transcription rankingsErrors in words, or characters for Korean and Japanese. Lower is better.
-
SpanishRecorded
Baian3.3%1stOpenAI4.3%2nd -
RussianRecorded
Baian4.7%1stGoogle Gemini 2.5 Flash6.4%2nd -
HindiRecorded
Baian1stRank only -
FrenchRecorded
Baian4.3%Joint 1stAssemblyAI4.6%Joint 1st -
JapaneseRecorded
Baian2.51%Joint 1stElevenLabs2.52%Joint 1st
Live and recorded speech
How much lower were the error rates?
- About half OpenAI’s errors in live ArabicBaian got 5.1% of words wrong, compared with OpenAI’s 10.5% on the same test set.
The numbers behind this
Word error rate on Arabic FLEURS, real time: Baian 5.1%, OpenAI GPT Realtime 10.5%, ElevenLabs Scribe v2 10.8%, Google Gemini 3.5 Live 6.7%. Competitor figures from AssemblyAI’s real-time benchmark, September 2026; Baian’s run on the same test set, October 2026.
- Fewer errors in Gulf Arabic mixed with EnglishBaian’s live interpreter got 15.6% of words wrong. OpenAI got 43.7%, working from finished recordings.
The numbers behind this
Word error rate, Perle test set (300 Saudi recordings, May 2026): Baian live interpreter 15.6%, ElevenLabs 21.3%, OpenAI GPT-4o Transcribe 43.7% (2.8× Baian’s), Microsoft Azure 50.6%, Google Chirp 3 62.4% (4×). Baian ran in real time; the others on finished recordings.
- First for live KoreanBaian got 2.9% of characters wrong, compared with Google’s 3.8% and OpenAI’s 7.1%.
The numbers behind this
Character error rate on Korean FLEURS, real time: Baian 2.9%, Google Gemini 3.5 Live 3.8%, OpenAI GPT Realtime 7.1%. AssemblyAI real-time benchmark, September 2026; Baian’s run October 2026.
- Fewer errors in recorded UrduBaian’s caption tool got 13.2% of words wrong, compared with OpenAI’s 19.7%.
The numbers behind this
Word error rate on Urdu FLEURS: Baian caption tool 13.2%, OpenAI GPT-4o Transcribe 19.7%, Qwen3.5-Omni 20.8%. Qwen3.5-Omni paper, April 2026; Baian’s run October 2026.
Live Arabic (live)
| System | 016% | Result |
|---|---|---|
| Baian | 5.1% | |
| Speechmatics Enhanced | 6.5% | |
| Google Gemini 3.5 Live | 6.7% | |
| Speechmatics Standard | 8.4% | |
| Amazon (AWS) | 8.5% | |
| Meta Muse | 8.9% | |
| OpenAI GPT-4o Transcribe | 9.4% | |
| OpenAI GPT Realtime | 10.5% | |
| ElevenLabs Scribe v2 | 10.8% | |
| OpenAI GPT-4o mini Transcribe | 11.7% | |
| AssemblyAI Universal | 12.1% | |
| Microsoft Azure | 12.5% | |
| Gladia | 15.1% |
How this was measured
AssemblyAI does not say which FLEURS text field it scored or how it treats Arabic vowel marks. Our 5.1% uses the cleaned text with vowel marks removed. Scored against the raw text, Baian’s live interpreter gets 6.75%, still well below OpenAI GPT Realtime (10.5%) and ElevenLabs (10.8%). AssemblyAI streamed each short clip on its own with the language set in advance; we streamed 15-minute recordings with automatic language detection, the setting customers use.
Live Korean (live)
| System | 010% | Result |
|---|---|---|
| Baian | 2.9% | |
| Google Gemini 3.5 Live | 3.8% | |
| AssemblyAI Universal | 4.3% | |
| ElevenLabs Scribe v2 | 4.6% | |
| Amazon (AWS) | 4.9% | |
| Speechmatics Standard | 5.2% | |
| Speechmatics Enhanced | 5.4% | |
| Meta Muse | 6.1% | |
| Gladia | 6.6% | |
| OpenAI GPT-4o Transcribe | 6.7% | |
| OpenAI GPT Realtime | 7.1% | |
| OpenAI GPT-4o mini Transcribe | 7.6% | |
| Microsoft Azure | 9.4% |
How this was measured
With spaces kept, Baian’s live interpreter scores 3.26%, still the lowest. Same setup difference as the Arabic chart: we streamed long recordings with automatic language detection.
Saudi Arabic mixed with English (live)
| System | 063% | Result |
|---|---|---|
| Baian | 15.6% | |
| ElevenLabs Scribe v2 | 21.3% | |
| OpenAI GPT-4o Transcribe | 43.7% | |
| Microsoft Azure | 50.6% | |
| Google Chirp 3 | 62.4% |
How this was measured
The other four systems were tested by Perle’s authors on finished recordings. Baian’s live interpreter ran in real time, which is harder. On meaning (BERTScore, closer to 1 is better) Baian’s live interpreter scored 0.956 against ElevenLabs’ 0.938.
Saudi Arabic mixed with English (recorded)
| System | 063% | Result |
|---|---|---|
| Baian | 9.5% | |
| ElevenLabs Scribe v2 | 21.3% | |
| OpenAI GPT-4o Transcribe | 43.7% | |
| Microsoft Azure | 50.6% | |
| Google Chirp 3 | 62.4% |
How this was measured
On meaning (BERTScore, closer to 1 is better) the Caption Tool scored 0.971 against ElevenLabs’ 0.938. Scored with Perle’s own rules and model.
Urdu transcription (recorded)
| System | 021% | Result |
|---|---|---|
| Baian | 13.2% | |
| OpenAI GPT-4o Transcribe | 19.7% | |
| Qwen3.5-Omni | 20.8% |
Transparency
Interpretation services rarely show how they test.
We checked what each one says about its own results.
| Service | Published accuracy evidence |
|---|---|
| Baian | Results on named public tests, with dates and methods. See them ↗ |
| Interprefy | Its own private test only (“up to 98%”), scored with its own method. Source ↗ |
| Wordly | A 98% to 99% claim with no named test set. Source ↗ |
| KUDO | A private May 2023 test that KUDO itself calls out of date. Source ↗ |
| Interactio | No accuracy figures. Source ↗ |
| Maestra | No accuracy figures. Source ↗ |
| Timekettle | An “up to 98%” claim with no named test set. Source ↗ |
| Microsoft Teams Interpreter | No figures; Microsoft says accuracy isn’t guaranteed. Source ↗ |
Hear it with your own event audio.
Bring a short recording, your languages and your audience size. We’ll show you the listener experience.
Arrange a demoSources and methodsEvery number on this page, with its test and date
- Live speech recognition (AssemblyAI real-time benchmark)AssemblyAI runs 25 August to 23 September 2026; Baian runs 13 to 15 October 2026
1st in Arabic (5.1%) and Korean (2.9%), 2nd in Russian and Danish. Fewer errors than all three OpenAI real-time models in 13 of the 14 languages we publish.
AssemblyAI real-time benchmarkMethod
FLEURS: short Wikipedia sentences read aloud. Measures how many words come out wrong.
AssemblyAI published scores for 14 real-time engines. We ran Baian’s live interpreter on the same test set.
We joined clips into 15-minute recordings with 1.5-second gaps and played them into Baian’s live interpreter in real time, like a microphone, with automatic language detection. AssemblyAI streamed each clip separately with the language set in advance. Scoring ignores punctuation, capital letters, Arabic vowel marks and how numbers are written. Our charts show 13 of AssemblyAI’s 14 engines; the full table is on AssemblyAI’s site.
- Saudi Arabic mixed with English (Perle)Paper May 2026; Baian runs October 2026
Caption Tool 9.5% of words wrong, Baian’s live interpreter 15.6%, ElevenLabs 21.3%, OpenAI 43.7%, Azure 50.6%, Google 62.4%.
Perle, May 2026 (arXiv)Method
300 recordings of Saudi speakers switching between Arabic and English. Measures words wrong and meaning kept.
Perle’s authors tested ElevenLabs, OpenAI, Microsoft Azure and Google. We ran both Baian products on the same recordings.
Scored with Perle’s own rules and meaning model. The Caption Tool ran in transcription-only mode; Live used default settings with no language hint.
- Recorded speech recognition (Verli)Verli July 2026; Baian October 2026
1st in Spanish, Russian and Hindi; joint 1st in French and Japanese.
Verli speech-to-text accuracy, July 2026Method
200 FLEURS sentences per language. Measures words wrong.
Verli tested 15 or 16 engines per language. We ran Baian’s caption tool on the same test set.
Verli tested each engine as it was in July 2026 on its own 200-sentence FLEURS sample. We used the same test set and scoring rule on a different 200 sentences. Verli’s scoring rule splits Hindi words at vowel signs, which roughly halves everyone’s error rate, so we publish Hindi as a rank only.
- Baian against professional translators (FLoRes)October 2026
Caption Tool scored above the translators in Hausa, Swahili, Tamil and Urdu (preferred on 67% to 75% of sentences). Baian’s live interpreter had about the same overall quality in Arabic, Hindi and Korean.
FLoRes-200 professional translationsMethod
148 English sentences that also have professional human translations. Measures translation quality with Google’s MetricX-24 automatic judge.
Baian ran the test. The human translations come from the public FLoRes-200 set.
Baian worked from the spoken audio; the translators worked from text. We compared scores sentence by sentence and counted a result only when the 95% range did not cross zero and two runs agreed. “About the same” means the test found no clear difference; it does not prove the translations were identical.
- English speech to Egyptian Arabic (WMT26)15 October 2026
2nd of 30 systems, and preferred over the professional post-edited translation on 91 of 133 clips (68%).
WMT26 general translation testMethod
133 English YouTube clips, the speech test from the 2026 Conference on Machine Translation.
Baian scored every published system’s output, and its own, with Google’s MetricX-24 automatic judge.
Baian received the same automatic transcript as the other systems. WMT’s own human raters rank systems differently, so we always describe this as an automatic score.
- Urdu speech recognition (Qwen3.5-Omni paper)Paper April 2026; Baian October 2026
13.2% of words wrong, the lowest in the comparison. OpenAI GPT-4o Transcribe got 19.7%.
Qwen3.5-Omni paper, April 2026 (arXiv)Method
FLEURS Urdu recordings. Measures words wrong.
The Qwen3.5-Omni authors published scores for several systems; we ran the Caption Tool on the same test set.
Scored with the standard Whisper rule: lower case, punctuation removed.
- Live delay13 to 15 October 2026
Median 2.1 seconds across 53 recordings in 15 languages (1.3 to 2.7 seconds per recording).
Method
How long after a sentence ends its finished translation appears.
Baian, on its production Live service.
Measured on Baian’s production Live service from 13 to 15 October 2026, from the end of each spoken sentence to the finished English line in our captured listener feed. Earlier words in a long sentence wait longer than this. Spoken audio arrives a little after the text and was not measured.
- European Parliament interpreters2019
Median 2.69 seconds behind the speaker, across 180 speeches.