The ad says“Deep understanding of how to evaluate AI systems - LLM-as-judge, red-teaming and adversarial testing, synthetic scenario generation, and multi-turn and agentic evaluation - and a clear sense of how evals themselves fail. You’ve deployed evals and automated AI tooling in production, not just prototyped them”
The ad says“Deep understanding of how to evaluate AI systems - LLM-as-judge, red-teaming and adversarial testing, synthetic scenario generation, and multi-turn and agentic evaluation - and a clear sense of how evals themselves fail. You’ve deployed evals and automated AI tooling in production, not just prototyped them”
The ad says“Hands-on experience with the eval tooling ecosystem (LangSmith, Braintrust, DeepEval, Ragas, Promptfoo, or similar)”
The ad says“Hands-on experience with the eval tooling ecosystem (LangSmith, Braintrust, DeepEval, Ragas, Promptfoo, or similar)”
The ad says“Enough fluency in statistics and experimental design to partner with a data scientist on calibration and reliability”
The ad says“Red-teaming or AI safety experience - prompt injection, jailbreaks, adversarial and stress testing”
The ad says“Hands-on experience with the eval tooling ecosystem (LangSmith, Braintrust, DeepEval, Ragas, Promptfoo, or similar)”
The ad says“Deep understanding of how to evaluate AI systems - LLM-as-judge, red-teaming and adversarial testing, synthetic scenario generation, and multi-turn and agentic evaluation - and a clear sense of how evals themselves fail. You’ve deployed evals and automated AI tooling in production, not just prototyped them”
The ad says“Experience with automated, eval-driven model or prompt optimization”
The ad says“Deep understanding of how to evaluate AI systems - LLM-as-judge, red-teaming and adversarial testing, synthetic scenario generation, and multi-turn and agentic evaluation - and a clear sense of how evals themselves fail. You’ve deployed evals and automated AI tooling in production, not just prototyped them”
The ad says“Red-teaming or AI safety experience - prompt injection, jailbreaks, adversarial and stress testing”
The ad says“Experience with automated, eval-driven model or prompt optimization”
The ad says“Hands-on experience with the eval tooling ecosystem (LangSmith, Braintrust, DeepEval, Ragas, Promptfoo, or similar)”
The ad says“Hands-on experience with the eval tooling ecosystem (LangSmith, Braintrust, DeepEval, Ragas, Promptfoo, or similar)”
The ad says“Deep understanding of how to evaluate AI systems - LLM-as-judge, red-teaming and adversarial testing, synthetic scenario generation, and multi-turn and agentic evaluation - and a clear sense of how evals themselves fail. You’ve deployed evals and automated AI tooling in production, not just prototyped them”
The ad says“Enough fluency in statistics and experimental design to partner with a data scientist on calibration and reliability”
The ad says“Red-teaming or AI safety experience - prompt injection, jailbreaks, adversarial and stress testing”
The ad says“Deep understanding of how to evaluate AI systems - LLM-as-judge, red-teaming and adversarial testing, synthetic scenario generation, and multi-turn and agentic evaluation - and a clear sense of how evals themselves fail. You’ve deployed evals and automated AI tooling in production, not just prototyped them”
The ad says“Can design the workflow and build a functional UI for non-engineers like clinicians and labelers”
We are a small, interdisciplinary team - engineers, data scientists, designers, product managers, and clinicians - building Nuna’s AI health coach. Our products are only as good as the care and science behind them, and your piece is how we know the coach is safe and working. You’ll own the evaluation system for the team building the coach: a data scientist partners with you on the science, clinicians and designers supply the ground truth, and the engineers shipping the agents depend on the signal you produce to decide what ships
This is a net-new, build-first role for someone who wants to own how we evaluate our AI agents end to end. You’ll build the harnesses, datasets, judges, and release gates that tell us whether the coach is safe and good, and you’ll own both that infrastructure and the evals that run on it
This is not a test-execution role - you write the code and own the system, rather than running tests someone else designed. You’ll make the day-to-day calls on standards, methods, and trade-offs, often with incomplete information and the freedom to define the right answer yourself. Comfort with ambiguity is part of the job
Build testing harnesses and evaluation infrastructure for our agentic products and our internal agentic tooling
Own our evals end to end - both the architecture and the content - with support from data science and clinical partners
Make every agentic deployment run through the testing apparatus before it ships, and own the release gates that keep unsafe or low-quality behavior from reaching patients
Build the ground truth, judges, and metrics, and validate that the evaluation itself can be trusted: calibration to human labels, reliability, and honest confidence on every number, in partnership with our data scientist
Build functional tooling for labeling and review workflows, so clinicians, coaches, and designers can author and review evaluation scenarios without an engineer in the loop
Help close the loop from evaluation results to model and prompt refinement, working toward systems that iterate safely with less human hand-holding
Benefits
Medical, vision, and dental coverage
401K plans with company matching
Commuter subsidies
Flexible work-from-home policy
Discretionary time off
Free meals and snacks
Clubs for hikers, the LGBTQIAP+ community, bookworms, karaoke stars, and more
Tech Talks on timely healthcare and technology topics
Presentations given by industry leaders
Create Days (company-wide “hackathons” to solve problems at Nuna and give back to our community)
Fitness discounts and reimbursements
Onsite massages
Weekly meditation sessions guided by Nuna yogis
Team fitness challenges (bike rides, marathons, 7-Minute Workouts, and more)
Preferred experience
A genuine interest in improving healthcare alongside an interdisciplinary team, with the judgment to tell a launch-blocking issue from a nice-to-have
A testing mindset applied to building the measurement system, not running tests against a spec: adversarial instinct, coverage thinking, regression discipline, and documentation others can build on
You use AI in your daily work and build tools that make the people around you more effective
Deep understanding of how to evaluate AI systems - LLM-as-judge, red-teaming and adversarial testing, synthetic scenario generation, and multi-turn and agentic evaluation - and a clear sense of how evals themselves fail. You’ve deployed evals and automated AI tooling in production, not just prototyped them
Can design the workflow and build a functional UI for non-engineers like clinicians and labelers
Significant experience building and shipping reliable production systems and tooling
Enough fluency in statistics and experimental design to partner with a data scientist on calibration and reliability
You’ve built in an early-stage or fast-moving environment
Experience with automated, eval-driven model or prompt optimization
Red-teaming or AI safety experience - prompt injection, jailbreaks, adversarial and stress testing
Hands-on experience with the eval tooling ecosystem (LangSmith, Braintrust, DeepEval, Ragas, Promptfoo, or similar)
Experience in healthcare or another regulated, high-trust domain, and familiarity with the regulatory landscape
