Python
Record your first voice-agent call from Python and open it in the console — about ten minutes, no agent framework required.
By the end of this page you will have a recorded call sitting in a local dashboard, with audio you can play and a trace you can expand.
Before you start
- Python 3.10 or newer for the SDK
- Python 3.11 or newer for the dashboard
- Access to the private repositories. Neither SDK is on PyPI yet, so both are installed from Git.
VaaniEval is in closed beta
The repositories below are private while the product is in closed beta, so the links will 404 unless your GitHub account has been granted access. Neither SDK is published to a public package registry yet — both install from Git.
To get access, email shubham@vaanieval.com or book a call — a short conversation about your use case tells us whether VaaniEval fits, and gets you onboarded with the setup that matches your stack.
Nothing here requires LiveKit, OpenAI, or any particular provider. If you are on LiveKit Agents, do this page first anyway — then swap in the LiveKit integration, which wires all of it up for you.
1. Run the dashboard
Clone and install
git clone https://github.com/shubhamofbce/vaanieval-observer-backend.git dashboard
cd dashboard
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txtStart it
uvicorn app.main:app --reload --port 8000The console is served at http://localhost:8000/. It will be empty — you have
not recorded anything yet.
The local dashboard has no authentication and no tenant isolation, and it accepts any non-empty API key without validating it. That is deliberate for a developer loop. Do not expose it beyond localhost or a private network. See Self-hosting.
2. Install the SDK
pip install "git+https://github.com/shubhamofbce/vaanieval-observer-python-sdk.git"The SDK has no required runtime dependencies. Install the extra that matches the HTTP client your agent already uses, so provider calls are instrumented automatically:
pip install "vaanieval-observer[httpx] @ git+https://github.com/shubhamofbce/vaanieval-observer-python-sdk.git"3. Record a call
This script records one synthetic call: two turns, a transcription span, a model span, and audio on both channels. Run it against the dashboard you just started.
import asyncio
import math
import struct
from vaani_observer import VaaniObserver
SAMPLE_RATE = 16000
FORMAT = {"encoding": "pcm_s16le", "sample_rate_hz": SAMPLE_RATE, "channels": 1}
def tone(hz: int, ms: int) -> bytes:
"""A second of audible PCM, so the recording has something to play."""
frames = int(SAMPLE_RATE * ms / 1000)
return b"".join(
struct.pack("<h", int(9000 * math.sin(2 * math.pi * hz * n / SAMPLE_RATE)))
for n in range(frames)
)
async def main() -> None:
vaani = VaaniObserver(
endpoint="http://localhost:8000",
api_key="local-dev", # the local dashboard does not validate this
spool_directory="./.vaani-spool",
endpoints=[
{"id": "llm", "type": "llm", "url": "https://api.openai.com/v1"},
],
)
session = vaani.start_session(agent_id="quickstart-agent")
for index in range(2):
turn = session.start_turn()
# The caller speaks.
stt = session.start_operation(type="stt", turn_id=turn.id, provider="demo")
stt.event("speech_started")
session.record_inbound_audio(tone(220, 900), FORMAT)
await asyncio.sleep(0.9)
stt.event("final_transcript")
stt.end(status="ok", response={"transcript": f"caller line {index + 1}"})
# The model thinks.
llm = session.start_operation(
type="llm", turn_id=turn.id, provider="demo", model="demo-model"
)
await asyncio.sleep(0.4)
llm.event("first_token")
llm.end(status="ok", response={"tokens": 128})
# The agent speaks.
tts = session.start_operation(type="tts", turn_id=turn.id, provider="demo")
tts.event("first_byte")
session.record_outbound_audio(tone(440, 1200), FORMAT)
tts.end(status="ok")
turn.end()
finalized = await session.end(outcome="completed")
print("package written to", finalized.directory)
await vaani.upload_package(finalized)
print("uploaded session", finalized.session_id)
asyncio.run(main())python record_a_call.pyYou should see something like:
package written to ./.vaani-spool/6f1c...-...
uploaded session 6f1c...-...4. Open the call
Refresh http://localhost:8000/. Your call is in the rail on the left. Click it
and you get the waveform, the turn markers, and a trace you can expand down to
individual milestones.
Look at the spool directory too — .vaani-spool/<session-id>/ contains
manifest.json, events.jsonl and call.audio. Everything the dashboard
shows is derived from those three files. See
Session package.
What just happened
VaaniObserver was configured, not connected
Constructing the observer installs HTTP and websocket instrumentation in-process
and validates your endpoint rules. It does not open a connection to the
dashboard. endpoint and api_key are only used by upload_package().
The session spooled to disk as the call ran
Every event and audio chunk was appended to ./.vaani-spool/<session-id>/ by a
dedicated writer thread, so no filesystem syscall ran on the event loop. Audio
arrives every 20 ms in a real agent, and a blocking write there is audible.
end() finalized the package
It closed open spans, flushed the writer, composed the two mono tracks into one
timeline-aligned stereo call.audio (agent left, caller right), and wrote
manifest.json last — via a temp file and rename, so a reader never sees a
half-written manifest.
upload_package() shipped it, after the call
Three steps: POST /v1/sessions with the manifest, PUT each object to the URL
the dashboard returned, then POST .../complete with byte sizes and SHA-256
digests. Upload is always explicit and always post-call — the library never
uploads on the live media path.
Wire it into a real agent
The synthetic script called start_operation() by hand. In a real agent you will
usually want two things instead:
- Automatic HTTP capture. Any
httpxoraiohttprequest made insidewith session.context():whose URL matches one of your configuredendpointsis timed and recorded for you, with no code change at the call site. - Turn grouping. Wrap the work for one caller utterance in
with session.with_turn(turn.id):so auto-instrumented calls land on the right turn.
session = vaani.start_session(agent_id="support")
turn = session.start_turn()
with session.with_turn(turn.id):
# Instrumented automatically because api.openai.com matches an endpoint rule.
response = await client.chat.completions.create(...)Read Recording a call next, or go straight to LiveKit Agents if that is your framework.
Troubleshooting
upload_package() needs both endpoint and api_key on the observer. If you
only want local spooling, do not call it — session.end() alone writes a
complete, valid package to disk.
POST /complete marks a session ready only when it imported at least one
operation from events.jsonl; otherwise the status is partial. A partial
session is still browsable. The usual cause is a session that ended before any
start_operation() call, or operations that were never end()ed — an operation
is written to events.jsonl when it ends, not when it starts.
Audio capture requires pcm_s16le input with an integer sample_rate_hz and
channels; anything else raises a ValueError rather than recording silently
(a non-bytes chunk raises TypeError). The format also cannot change within a
session. Check that capture.audio was not set to False — record_*_audio()
returns False instead of raising when capture is disabled or the session has
already ended.
Auto-instrumentation is inert unless there is an ambient session and the URL
matches a configured endpoint rule. Confirm the call is inside
with session.context(): and that your endpoints list has a rule whose url
is a prefix of the request URL. See
Endpoints and instrumentation.
Introduction
VaaniEval records every voice-agent call as a portable session package — audio, transcripts, provider spans and turn timings — so you can answer "why was that call bad?" with evidence instead of guesses.
Node.js
Record your first voice-agent call from Node.js, with automatic fetch instrumentation and websocket accounting.