Introduction
VaaniEval records every voice-agent call as a portable session package — audio, transcripts, provider spans and turn timings — so you can answer "why was that call bad?" with evidence instead of guesses.
VaaniEval is observability and evaluation for voice agents: the systems that listen to a caller, think with a model, call a tool or two, and speak back.
A voice call is not a web request. It is a chain of turns, each one a race against the caller's patience, and each one stitched together from three or four providers that all report time differently. When someone says "the bot was slow and it got my destination wrong", a normal APM can tell you an HTTP request took 7.4 seconds. It cannot tell you which turn the caller was waiting through, that the model was silently retried inside that turn, or that the transcription heard "Dubai" when the caller said "Manali".
VaaniEval is built to answer exactly those questions.
What you get
A complete recording of every call
Timeline-aligned stereo audio with the agent on the left and the caller on the right, the transcript, every provider span, and the turn structure — written to disk during the call and uploaded afterwards.
Turn-level latency you can defend
Listening, thinking and speaking time per turn, attributed to the STT, LLM, tool or TTS span that actually caused the wait — including retried HTTP attempts hidden inside a single framework call.
Transcription review
Replay a recorded call against a stronger challenger model, see every word production disagreed on, and let a judge flag only the disagreements that could have changed the conversation.
A fleet view and alerts
Roll the same measurements up across agents, providers, models and environments, and get told when a threshold you care about is breached.
How it fits together
VaaniEval has two halves, and they are deliberately separate.
A capture SDK runs inside your agent process. It writes a session package —
manifest.json, events.jsonl, call.audio — to a local spool directory as the
call happens. It never blocks the media path on a network call. There are two
SDKs, Python and Node.js, and they emit a
byte-compatible format.
A dashboard ingests those packages, stores them, and answers questions about them. It is a FastAPI service you run yourself.
The split matters. Because capture is local-first, a network problem between
your agent and the dashboard costs you an upload, not a call. Because the package
is a plain directory of files, you can inspect it with cat and jq before you
trust anything the dashboard tells you.
Every number the dashboard shows is derived from the events in the package.
There is no separate metrics pipeline to drift out of sync — if you disagree
with a value, you can recompute it from events.jsonl yourself.
Where to start
Python quickstart
Record your first call from a Python agent, including LiveKit Agents, and open it in the console.
Node.js quickstart
Record your first call from a Node.js agent with automatic fetch instrumentation.
Tour the live demo
Thirty real, anonymised calls. No install required — read the product before you wire anything up.
Core concepts
Sessions, turns, operations, milestones and the call clock — the five ideas everything else is built from.
What VaaniEval is not
Being clear about this saves you an evaluation cycle.
- It is not a real-time monitor. Upload is explicit and happens after the call ends, so a call becomes visible one call-duration later. If you need sub-second alerting on a call in progress, this is the wrong tool today.
- It is not a ground-truth accuracy benchmark. The STT comparison scores your production transcript against a challenger model, not a human reference. The result is labelled estimated WER / model disagreement, never accuracy. See Metrics.
- It is not multi-tenant SaaS in this release. The dashboard you self-host has no authentication and no tenant isolation. Read Self-hosting before you put it anywhere but a private network.
- It does not replace your agent framework's own tracing. It complements it. VaaniEval cares about the caller's experience of time and words; framework tracing cares about your code.
Source
The platform is three private repositories:
| Repository | What it is |
|---|---|
vaanieval-observer-backend | The FastAPI dashboard, console and STT evaluation engine |
vaanieval-observer-python-sdk | The Python capture SDK, including the LiveKit Agents integration |
vaanieval-observer-nodejs-sdk | The Node.js capture SDK |
VaaniEval is in closed beta
The repositories below are private while the product is in closed beta, so the links will 404 unless your GitHub account has been granted access. Neither SDK is published to a public package registry yet — both install from Git.
To get access, email shubham@vaanieval.com or book a call — a short conversation about your use case tells us whether VaaniEval fits, and gets you onboarded with the setup that matches your stack.