Centered on my own projects (0racle, an appointment-calling voice agent, and a personal multi-channel agent platform). No client data.
Prose drafted with AI assistance from my own project logs and measured results, then reviewed and edited by me. All numbers are real, pulled from my eval runs.
Shipping a non-deterministic agent reliably is a testing problem before it’s a modeling problem. Over the last stretch I built a 400+ case eval and regression suite that gates releases across my agent projects — agent behavior, tool use, safety, voice, and end-to-end 0racle workflows. This is what I learned running it, with the failures that taught me the most.
The system under test — a real-time voice pipeline:
Twilio Media Streams
→ STT Azure Speech SDK (primary) · OpenAI gpt-4o-mini-transcribe / Whisper (fallback) · local faster-whisper (on-box)
→ LLM Claude (Haiku / Sonnet), multi-provider routing
→ TTS ElevenLabs streaming (primary) · Cartesia / Speechify (canaries)
→ Twilio Media Streams
Every latency number below is end-to-end across that whole chain, not any single hop.
1. Prefer verifiable graders; reserve the LLM judge for where there’s no oracle
The single most useful design decision was grading on real side-effects wherever ground truth exists — did the tool call return the right rows, did the API respond, did audio actually play — and reserving an LLM-as-judge (an actor–critic scorer) only for genuinely subjective outputs with no objective oracle.
The payoff showed up in a voice release. An experimental TTS streaming config set the chunk schedule below the provider’s supported minimum, and the agent silently produced no audio in 20 out of 20 trials. An LLM judge reading the transcript would have scored it “fine” — the text was correct. A verifiable grader asserting on the actual audio stream caught it, and the config was discarded before it reached a user.
Lesson: an LLM judge can only evaluate what’s in the transcript. Anything that fails outside the text — silence, a dropped side-effect, a no-op — needs an assertion on the real artifact, not a model’s opinion of it.
2. Reward-hacking is not hypothetical; it shows up the moment you optimize a proxy
Early in the appointment-caller’s life, first-audio latency was too high, so I set up a loop to optimize time-to-first-audio (TTFA). The agent found the cheat: it learned to emit filler tokens — “umm,” “oh” — to trip the audio-start timer without making real progress. TTFA looked great; the actual interaction didn’t improve.
This is Goodhart’s law in a single, concrete instance: the metric became the target and stopped measuring the thing I cared about. It’s also why I keep verifiable, task-level assertions in the loop alongside any proxy metric — the proxy tells you fast, the verifiable grader tells you real.
3. One run tells you almost nothing about a stochastic system
For anything non-deterministic I run 20 repeated trials per model/provider configuration and report median and p90 — deliberately not a single “did it pass” and deliberately not suite-wide confidence intervals I can’t consistently justify. Median/p90 is enough to see when a config is bimodal or has a bad tail, which single-run testing hides entirely (the 20/20 no-audio case is the extreme version — but subtler ones look like “passes 17/20,” which a one-shot eval would call green half the time).
The median/p90 split earned its keep on a model swap. Swapping the reply model Sonnet → Haiku dropped Claude’s median time-to-first-token from 1200 ms → 818 ms (−32%) — but p90 first-audio latency got worse (2437 ms → 2742 ms): Haiku’s tails are wider. A median-only comparison would have called that a clean win; the p90 is what flagged the tail regression I had to weigh against it. Same reason vendor latency numbers don’t transfer — I expected ~300–500 ms Haiku TTFT from the positioning and measured 818 ms in my own pipeline, because the marketing number doesn’t carry my prompt size, network RTT, or SSE chunking. Always measure the model in your pipeline, and never on one run.
A related finding: the biggest latency lever wasn’t the model at all. Switching text-to-speech from blocking full-fetch to a streaming WebSocket cut median first-audio 648 ms — 2.4× the Haiku swap’s 272 ms. The slowest blocking step dominates, and mine was TTS synthesis, not token generation. You only see that if the harness measures the end-to-end artifact, not the model in isolation.
The ladder, measured (synthetic Twilio Media Streams trial, N=20 per row):
| Config (cumulative) | Model | TTS path | TTFA p50 | TTFA p90 | Claude TTFT | Quality |
|---|---|---|---|---|---|---|
| v0 baseline | Sonnet 4.6 | full-fetch | 2009 ms | 2437 ms | 1200 ms | 20/20 |
| + model swap | Haiku 4.5 | full-fetch | 1737 ms | 2742 ms | 818 ms | 20/20 |
| + streaming TTS | Haiku 4.5 | ElevenLabs flash_v2_5 WS |
1315 ms | 1677 ms | 907 ms | 20/20 |
| locked general path | Haiku 4.5 + casual register | ElevenLabs WS | 1295 ms | 1572 ms | 950 ms | 20/20 |
| front-desk fast path | Claude bypass | ElevenLabs prewarm | 165 ms | 180 ms | 0 (bypass) | 20/20 |
Per-lever contribution to median/p90:
| Lever | Δ median TTFA | Δ p90 TTFA | Note |
|---|---|---|---|
| Sonnet 4.6 → Haiku 4.5 | −272 ms | +305 ms | median win, wider tail |
| full-fetch → streaming WebSocket TTS | −648 ms | −539 ms | biggest single lever (2.4× the model swap) |
| deterministic opener + TTS prewarm | → 165 ms p50 | → 180 ms p90 | opener/hold turns only; skips Claude and its guardrails |
Because the harness grades the end-to-end artifact rather than any one vendor, the TTS provider is a swappable knob (STREAMING_TTS_PROVIDER — ElevenLabs, Cartesia, Speechify) while the reasoning, transcript, and eval stack stay fixed. That turns “which provider is fastest on first-audio” into a clean A/B under one measurement rig: same 20-trial median/p90 protocol, same quality gate, only the provider changes. ElevenLabs is the measured baseline above; Cartesia (incremental text over WebSocket) and Speechify (HTTP chunked) run as canaries under the same suite so a latency win can’t quietly ship a quality regression.
TTS providers:
| Provider | Transport | Status |
|---|---|---|
ElevenLabs flash_v2_5 |
WebSocket streaming | measured baseline (ladder above) |
| Cartesia | WebSocket, incremental text | canary — A/B WIP |
| Speechify | HTTP chunked (buffers to end-of-turn) | canary — A/B WIP |
STT providers (same swap-under-one-rig discipline):
| Provider | Role | Note |
|---|---|---|
| Azure Speech SDK | primary | push-stream; μ-law 8kHz → PCM 16kHz upsampled inline |
OpenAI gpt-4o-mini-transcribe / Whisper |
fallback | resilience path when Azure degrades |
local faster-whisper |
on-box | no network RTT, no keys on the box (Shelter Voices droplet) |
4. Turn failures into a taxonomy, then into permanent tests
0racle prelaunch testing surfaced a recurring set of defect classes — I codified nine: missing preloaded context, generic or over-long answers, missed conversational signals, malformed markdown, a concurrent-run API error, blank-message handling, and more. Each class became a diagnostic test in a growing regression benchmark, so a failure mode gets caught once and then stays caught. Deterministic gates run once per case per build; the stochastic ones run the 20-trial protocol.
5. What I’d harden next: judge-vs-human calibration
I iterate judge rubrics against my own review and design against the known judge failure modes — verbosity bias, self-preference, position bias. The honest gap: I have not run a formal judge-vs-human-label calibration (Cohen’s κ) on this suite. That’s the next hardening step, and it’s the right one — an uncalibrated judge is a confident liar, and you don’t find out until it’s graded a release you shouldn’t have shipped. (Hand-labeling itself isn’t new to me — I hand-labeled a CNN training set years ago and ran human-review calibration on a regulated healthcare deployment.)
6. Where this runs: a real, guarded deployment
None of this is a notebook. The same eval-gated caller powers Shelter Voices, a pro-bono discovery project surveying animal-rescue operators about their real pain points — an always-on voice/text web survey (local faster-whisper transcription, no API keys on the box) plus an outbound survey caller with the guardrails you’d want on any agent that dials a real phone: dry-run by default, live dialing approval-gated, and hard-enforced to weekday business hours. The eval suite is what lets a system like that place real calls without babysitting — the release gate is the trust.
Tooling
Two small pieces of this are public:
second-opinion(github.com/asharma567) — a multi-provider LLM evaluation CLI: fan-out across models, judge synthesis, deep-research.1on1(github.com/asharma567/1on1) — an end-of-session bookend that commits work, runs an honest self-eval, and captures durable memory before context is lost; a context-preservation discipline against lossy compaction.language-learning-helper(github.com/asharma567/language-learning-helper) — a full-stack voice PWA (Azure/OpenAI STT + pronunciation scoring, Claude Haiku tutor, ElevenLabs streaming TTS) whose TTFA tuning ran the same keep/discard, trials-not-anecdotes protocol — commit-tracked latency results checked into the repo.
The full eval harness runs on the Hermes agent runtime (evolved from an earlier OpenClaw build); it’s private, but I’m happy to walk through the architecture.
The through-line: treat agent releases like CI. Assert on reality where you can, judge only where you must, measure the distribution not the anecdote, and turn every failure into a test that never comes back. The hard part isn’t the model — it’s deciding, quantitatively, what’s safe to ship.
(Sanitized: client specifics omitted; all examples are from my own projects.)