Infer
We’re building the operating system for insurance agencies using AI agents to handle tasks like lead qualification, live call support, call auditing, renewals, and customer retention. You’ll own the evaluation system that ensures our voice agents continuously improve — catching regressions before customers are affected. This is a critical infrastructure role where your work directly impacts product quality and customer trust.
About the Role
You’ll be responsible for building and maintaining the evaluation framework that scores voice agent quality across transcription, LLM reasoning, tool use, TTS, and full-conversation outcomes. You’ll design voice agent behavior, drive STT and TTS accuracy improvements, curate and grow our evaluation datasets, and build benchmarks we can run against any new model in days. You’ll also run a red-team pipeline that probes for failure modes and wire eval signals into CI so regressions get caught before they ship.
What You Will Do
Build and maintain the eval framework that scores voice agent quality across transcription, LLM reasoning, tool use, TTS, and full-conversation outcomes
Design voice agent behavior: system prompts, tool use, conversation flow, error recovery, and guardrails for real-time interactions
Drive STT and TTS accuracy improvements by comparing providers, fine-tuning models, and running open models like Whisper
Curate and grow our evaluation datasets to ensure robust testing and continuous improvement
Build benchmarks we can run against any new model in days to ensure consistent quality
Run a red-team pipeline that probes for failure modes and vulnerabilities
Wire eval signals into CI so regressions get caught before they ship
Build self-improvement loops that auto-feed production failures back into the eval set
What We Are Looking For
ML engineering experience shipping production systems
Strong Python and a working ML stack (PyTorch, Huggingface, pandas, scikit-learn)
Hands-on experience designing LLM-based agents: prompting, tool/function calling, multi-turn state, structured outputs
Hands-on experience building evals or eval frameworks for ML, LLM, or voice systems. Built LLM-as-judge eval pipelines and know their failure modes
Practical experience with ASR/STT comparing providers, fine-tuning, or running open models like Whisper
Practical experience with TTS systems (ElevenLabs or open models)
Comfortable working with audio data: sample rates, codecs, noise, alignment
Nice to Have
Designed voice agents specifically handled barge-in, interruption recovery, disfluencies, and natural turn-taking at the prompt/behavior layer
Experience with diarization, VAD, or endpointing models
Audio dataset curation, labeling, or annotation pipelines
Trained or fine-tuned ASR or TTS models from scratch or on domain audio
Experience with active learning or data-flywheel patterns over production traffic
Open-source contributions to AI/ML frameworks
Familiarity with cost/latency tradeoffs across model providers for real-time voice
If you’re passionate about building AI systems that improve over time and want to be part of a team that’s pushing the boundaries of what’s possible, we want to hear from you.