VaaniEval
Reference

Configuration

Every SDK option, environment variable and dashboard setting in one place, with defaults and the tradeoff each one carries.

Python SDK

VaaniObserver(
    endpoint=None,
    api_key=None,
    spool_directory=None,
    capture=None,
    instrumentations=None,
    endpoints=None,
    upload=None,
    strict=False,
)
OptionTypeDefaultNotes
endpointstrNoneDashboard base URL; only used by upload
api_keystrNoneBearer token; not validated by the local dashboard
spool_directorystr | Pathcwd/.vaani-spoolPackage output directory
capturedictsee belowWhat to record
instrumentationsdict{http: True, websocket: True}Which clients to patch
endpointslist[dict][]Provider URL rules
uploaddict{retries: 3}Upload behaviour
strictboolFalseRaise instead of degrading

Node.js SDK

new VaaniObserver({
  endpoint: undefined,
  apiKey: undefined,
  spoolDirectory: undefined,
  capture: undefined,
  instrumentations: undefined,
  endpoints: undefined,
  upload: undefined,
  strict: false,
});

Identical shape in camelCase. instrumentations uses { fetch, websocket } rather than { http, websocket }.

capture

Option (Python / Node)DefaultEffect
audio / audioTrueRecord caller and agent PCM
http_bodies / httpBodiesFalseCapture request and response bodies
websocket_text_frames / websocketTextFramesFalseCapture text frame contents
stt_content / sttContentFalseCapture transcript text
payload_max_bytes / payloadMaxBytes16384Size bound on every payload

Audio is on by default. A recorded call is personal data in most jurisdictions. http_bodies and stt_content put prompts and caller speech in plain text into events.jsonl and the dashboard's database — usually the reason a deployment needs a privacy review. See Capture and privacy.

endpoints

{"id": "llm", "type": "llm", "url": "https://api.openai.com/v1", "match": "path"}
FieldRequiredValues
idyesUnique; duplicates raise at construction
typeyesstt | llm | tts
urlyeshttp, https, ws or wss
matchnopath (default) | origin | exact

An unknown match raises TypeError. A URL matching more than one rule at the same scheme precedence raises Ambiguous Vaani endpoint rules for …. See Endpoints and instrumentation.

strict

ValueBehaviourUse it
False (default)Instrumentation errors are swallowed; degradation surfaces in capture_statusProduction
TrueErrors raiseStaging and CI

The default trades visibility for safety: a bug in observability cannot take down a call, but capture can degrade silently. Your signals that it happened are capture_status in the manifest and the unverifiable class in the dashboard — not an exception.

LiveKit integration environment

Read by VaaniLiveKitRecorder.from_env().

VariableDefaultPurpose
VAANI_ENABLEDfalseMaster switch — off by default
VAANI_ENDPOINThttp://localhost:8000Dashboard base URL
VAANI_API_KEYlocal-devBearer token
VAANI_SPOOL_DIR.vaani-spoolPackage output directory
VAANI_CAPTURE_AUDIOtrueRecord PCM
VAANI_CAPTURE_HTTP_BODIEStrueCapture bodies
VAANI_CAPTURE_STT_CONTENTtrueCapture transcripts
VAANI_PAYLOAD_MAX_BYTES16384Payload bound
VAANI_AGENT_IDlivekit-agentDefault agent id
VAANI_UPLOADtrueUpload after finish()

These defaults are more permissive than the core SDK'sVAANI_CAPTURE_HTTP_BODIES and VAANI_CAPTURE_STT_CONTENT default to true. Set them to false for production traffic unless you have reviewed the privacy implications.

from_env() never raises: on failure it returns an inert recorder whose methods are no-ops and whose enabled is False. Check recorder.enabled at startup if you want a misconfiguration to be visible.

Dashboard

VariableDefaultPurpose
VAANI_DATA_DIR./dataSQLite file and uploaded objects
ELEVENLABS_API_KEYChallenger transcription; required to evaluate
OPENAI_API_KEYSemantic risk judge
STT_EVAL_JUDGE_MODELgpt-4o-miniJudge model
VAANI_ENV_FILE./.envos.pathsep-separated dotenv paths

Fixed limits, not configurable:

ConstantValue
MAX_UPLOAD_BYTES128 MiB
ALLOWED_OBJECTSevents.jsonl, call.audio, caller.audio, agent.audio
Challenger worker pool2 threads
COHORT_SAMPLE_LIMIT25 sessions
CHALLENGER_MODELSelevenlabs_scribe_v2
SettingProductionStaging
audioOn, with consent + retention sweepOn
http_bodiesOffOn
stt_contentOff unless evaluating STTOn
websocket_text_framesOffOn while debugging
payload_max_bytes1638465536
strictFalseTrue
SDK sourcePinned tag or commitBranch is fine
Spool cleanupRequiredOptional

Two things neither SDK does for you: cleaning up spool directories after upload, and enforcing a retention policy. Both will fill a disk if left alone. Add a sweep of .vaani-spool on your agent hosts and of $VAANI_DATA_DIR/objects on the dashboard host.

Next

On this page