Fieldguide · San Francisco, CA, USA
At Fieldguide, we're building AI agents for the most complex audit and advisory workflows. We're a San Francisco-based Vertical AI company building in a $100B+ market undergoing rapid transformation.
About the Role
We're hiring an AI Engineer, Quality to own the evaluation infrastructure that ensures our AI agents perform reliably at enterprise scale.
Responsibilities:
Design and build a unified evaluation platform that serves as the single source of truth for all of our agentic systems and audit workflows
Build observability systems that surface agent behavior, trace execution, and failure modes in production, and feedback loops that turn production failures into first-class evaluation cases
Owning the evaluation infrastructure stack including integration with LangSmith and LangGraph.
Translate customer problems into concrete agent behaviors and workflows
Integrate and orchestrate LLMs, tools, retrieval systems, and logic into cohesive, reliable agent experiences
Rapid Model Evaluation: Build automated pipelines that evaluate new models against all critical workflows within hours of release
Design evaluation harnesses for our most complex Agentic systems and workflows
Implement comparison frameworks that measure effectiveness, consistency, latency, and cost across model versions
Design guardrails and monitoring systems that catch quality regressions before they reach customers