Observability and evaluation for voice agents
Voice agents fail in ways a normal APM cannot see: the caller waited nine seconds, the transcription dropped a word that changed the booking, the model was retried twice inside one turn. VaaniEval records the whole call — audio, transcripts, provider spans and turn timings — and gives you the one turn that explains the complaint.
Python and Node.js SDKs · self-hosted dashboard · your audio never leaves your infrastructure
One recorded call in the console — four turns, two failures, and the 24.2 second wait that caused the complaint.
Find the slow turn, not the slow service
A call is a chain of turns. VaaniEval times each one end to end — listening, thinking, speaking — and attributes the wait to the STT, LLM, tool or TTS span that actually caused it, including retried HTTP attempts hidden inside one framework call.
Hear what the caller heard
Every call is stored as timeline-aligned stereo PCM with the agent on the left and the caller on the right, plus a transcript that follows playback. When a metric looks wrong, you play the exact second it describes.
Check the transcript, not just the timing
Replay a recorded call against a stronger challenger model, see every word your production STT disagreed on, and have a judge flag only the disagreements that could have changed the conversation.
Ten minutes to your first recorded call
Install the SDK, point it at a locally running dashboard, and open the call you just made. No provider lock-in, no agent framework required.