The ad says“Experience building testing, evaluation, or data infrastructure for complex systems (AI/ML experience strongly preferred)”
The ad says“Comfort writing production-quality code (we use Python and TypeScript)”
The ad says“Comfort writing production-quality code (we use Python and TypeScript)”
Values
The ad says“Bonus: experience evaluating LLMs, agentic systems, or AI-assisted developer tools”
The ad says“Bonus: experience evaluating LLMs, agentic systems, or AI-assisted developer tools”
The ad says“Familiarity with modern ML systems and evaluation techniques (e.g., offline metrics, online evaluation, regression testing for models or prompts)”
The ad says“Familiarity with modern ML systems and evaluation techniques (e.g., offline metrics, online evaluation, regression testing for models or prompts)”
As a Senior Software Engineer on Sentry’s AI/ML team, you’ll be responsible for building the evaluation infrastructure that measures the accuracy, reliability, and real-world performance of our AI systems
This role is critical to ensuring that our debugging agents and AI-powered features behave correctly, safely, and predictably as they scale
You’ll design datasets, benchmarks, and test harnesses that turn ambiguous AI behavior into measurable signals, helping the team ship AI with confidence
Design and build robust evaluation frameworks to measure accuracy, reliability, regressions, and edge cases in AI systems
Create and curate high-quality datasets, golden test cases, and benchmarks grounded in real production data
Build automated test harnesses and metrics pipelines to continuously evaluate models, prompts, and agentic workflows
Partner closely with applied AI engineers and product leaders to define what “good” looks like and translate it into measurable criteria
Own the evaluation lifecycle for major AI initiatives, from early experimentation through production monitoring
Benefits
Competitive Compensation + Equity
Medical, Dental, Vision Insurance
Commuter Stipend
Professional Development Stipend
Health & Wellness Benefits
Charitable Matching Program
Flexible PTO
Paid Parental Leave
401k Plan
Preferred experience
Enjoy turning fuzzy product goals and model behavior into concrete tests and metrics
Like building foundational infrastructure that unlocks faster iteration and higher confidence for the entire AI team
Care deeply about correctness, rigor, and measurement in AI systems
Thrive in cross-functional environments and enjoy influencing model design through better evaluation
Minimum 5+ years of professional experience with a Bachelor’s degree in computer science, machine learning, or a related field
Experience working with structured and unstructured datasets, labeling workflows, or data quality pipelines
Comfort writing production-quality code (we use Python and TypeScript)
Experience building testing, evaluation, or data infrastructure for complex systems (AI/ML experience strongly preferred)
Bonus: experience evaluating LLMs, agentic systems, or AI-assisted developer tools
Familiarity with modern ML systems and evaluation techniques (e.g., offline metrics, online evaluation, regression testing for models or prompts)
