Resources

Benchmarks

Our own measurements, each labelled with the model it ran on. Synthetic test callers, never customer calls.

Every figure on this page comes from our own benchmarks: scripted test callers played as audio through our engine and our self-hosted speech and language models. None of them are customer calls. Each series names the language model it ran on. The series are kept separate and are never added together or compared with each other.

Card-servicing calls

Gemma 4 26B-A4B on A100, 2026-10-20, 80-call card-servicing run.

Eighty scripted card-holder calls to an invented bank, played one at a time as 8 kHz audio through the real speech-to-text, language model and text-to-speech, into a graph agent with function tools against a fake bank. A call passes only when every expectation of its scenario holds: the step it ends on, the steps it visits and avoids, the values it extracts, the bank operations it requests, refusals and hang-ups. Nothing that needs a verified caller may happen without a verification.

61 of 80 calls passed (76.2%).

English callers' speech goes through a Hindi-only transcriber today, a known limit of the transcriber rather than of the agent.

"Reached" and "passed" are different counts: a fraud call can reach the fraud step and still fail another expectation of its scenario.

Every row counts calls that passed every expectation of their scenario. "Reports fraud" reads 9 of 11 although all 11 fraud reports reached the fraud step (see the safety table).

The Hindi transcriber's word error rate on this run was 9.5% on the Hindi and Hinglish lines.

Response time on calls

Gemma 4 26B-A4B on A100, 2026-10-20, 80-call card-servicing run. From the caller stopping to the agent's first audio, measured at the caller's end of the simulated phone line, waiting for the end of the turn included. 253 spoken turns.

Time the model takes per turn

Gemma 4 26B-A4B on A100, 2026-10-20. A text replay of the same 80 conversations straight against the language model, one at a time: no speech-to-text, no voice, no silence detection. These are model times only, not what a caller waits.

The same conversations on a prompt agent, which has no routing request: 277 ms median total model time per turn.

Interruptions (barge-in)

Measured on Gemma 4 E4B, October 2026; 26B results to follow.

Before and after one round of engine work. Test callers talk over the agent mid-sentence.

94 calls after the change, 59 before.

Response time on the E4B series

Measured on Gemma 4 E4B, October 2026; 26B results to follow. See Voice engine: response time.