The ad says“Speech/audio processing or ASR evaluation”
The ad says“Run experiments that move accuracy. Design and execute evaluation experiments on large, representative datasets using LangSmith and BigQuery, and track quality with our performance metrics”
The ad says“Google Cloud Platform, BigQuery, Docker, or Kubernetes”
The ad says“Google Cloud Platform, BigQuery, Docker, or Kubernetes”
The ad says“Strong Python skills and fluency with the standard data toolkit (Pandas, NumPy, Jupyter). SQL is a plus”
The ad says“Google Cloud Platform, BigQuery, Docker, or Kubernetes”
The ad says“Run experiments that move accuracy. Design and execute evaluation experiments on large, representative datasets using LangSmith and BigQuery, and track quality with our performance metrics”
The ad says“Fine-tuning or building synthetic datasets for LLMs”
The ad says“We’re looking for a sharp, analytical data scientist to own the evaluation and quality loop of our AutoQA product. You can be early in your career (recent graduates with strong LLM fundamentals are welcome), we have room and a clear path for you to grow”
The ad says“Experience with LangSmith or similar LLMOps/evaluation tooling”
The ad says“Strong Python skills and fluency with the standard data toolkit (Pandas, NumPy, Jupyter). SQL is a plus”
The ad says“Strong Python skills and fluency with the standard data toolkit (Pandas, NumPy, Jupyter). SQL is a plus”
The ad says“Strong Python skills and fluency with the standard data toolkit (Pandas, NumPy, Jupyter). SQL is a plus”
The ad says“Evaluate across the full stack. Beyond scoring quality, you’ll help validate retrieval (RAG/IR), tool calling, and speech pipelines (transcription and diarization quality)”
The ad says“Speech/audio processing or ASR evaluation”
Values
The ad says“Strong Python skills and fluency with the standard data toolkit (Pandas, NumPy, Jupyter). SQL is a plus”
We’re looking for a sharp, analytical data scientist to own the evaluation and quality loop of our AutoQA product. You can be early in your career (recent graduates with strong LLM fundamentals are welcome), we have room and a clear path for you to grow
Translate customer rubrics into AutoQA instructions. Work with real customer quality criteria and turn them into precise, testable instructions that LLMs can score reliably
Run experiments that move accuracy. Design and execute evaluation experiments on large, representative datasets using LangSmith and BigQuery, and track quality with our performance metrics
Build the datasets that make evaluation possible. Curate raw production data into golden datasets with balanced coverage, and generate synthetic data to cover the rare cases that matter most. QA is a discipline of rare occurrences: distributions are skewed, positives are scarce, and resourcefulness beats volume
Build LLM-as-a-judge pipelines to assess system quality internally and make evaluation repeatable
Make results actionable. Your experiments should end in a decision: change a prompt, adjust which tools the system uses, surface context the AI is missing, or flag where new capabilities are needed. You’ll work with the AI team to ship those decisions
Evaluate across the full stack. Beyond scoring quality, you’ll help validate retrieval (RAG/IR), tool calling, and speech pipelines (transcription and diarization quality)
Join customer calls with the team to understand how QA leaders define quality, and feed what you learn back into the product
What you’ll grow into:
Shaping how customers monitor quality themselves, catch drift, and keep their AutoQA setup improving over time
Smarter categorization and routing of conversations to power analytics and get the right tickets to the right evaluation
You’ll join our AI team (two data scientists, an AI engineer, and an ML engineer) and collaborate closely with our data engineers, frontend engineers, designer, product manager, and CX teams. You’ll have mentorship from day one and real ownership fast
Benefits
Personal development program
Mentorship program
Visa sponsorship
Stock options
Team activities
Referral bonus
Workations
Free lunch
Remote work
Home office setup
Flexible hours
Open holiday policy
Preferred experience
Excellent written and verbal communication. You’ll present findings to the team and join customer conversations
Good applied statistics: experiment design, classifier evaluation, precision/recall trade-offs, and working with heavily imbalanced data
A solid understanding of how LLMs work and hands-on experience prompting them for accuracy (coursework, thesis, internships, or side projects all count; production experience is a bonus)
0 to 2 years of industry experience. Recent graduates with strong relevant work are encouraged to apply
Strong Python skills and fluency with the standard data toolkit (Pandas, NumPy, Jupyter). SQL is a plus
A team player who’s comfortable wearing multiple hats. We’re an early-stage company and things move fast
Product sense and empathy for end users. You’ll be building for QA managers and support agents, not just for benchmarks
An analytical, evidence-first mindset: you’d rather measure than assume
Fine-tuning or building synthetic datasets for LLMs
Speech/audio processing or ASR evaluation
Google Cloud Platform, BigQuery, Docker, or Kubernetes
Experience with LangSmith or similar LLMOps/evaluation tooling
