How to Become an AI Evaluation Engineer: 2026 Roadmap

An AI evaluation engineer builds the tests that tell a team whether an LLM feature got better or worse. That means curating datasets of real cases, choosing metrics such as faithfulness and answer relevance, building LLM-as-a-judge scorers and checking they agree with people, and wiring it all into CI so a prompt change can’t quietly break production.

To become one, you need software engineering fundamentals, a feel for how RAG and agent systems fail, one evaluation library learned well, and a public project that shows a working eval gate.

What does an AI evaluation engineer actually do?

They own the answer to “is this LLM feature good enough to ship, and is it better than last week?”. The work has four parts:

• Datasets: collecting and labelling representative cases, including the ugly ones from production logs.
• Metrics: choosing what to measure per feature, such as faithfulness, relevance, tool use, safety, cost and latency.
• Judges: writing LLM-as-a-judge rubrics and validating them against human labels.
• Gates: running evals on every change in CI, and sampling production traffic for what the dataset missed.

Titles vary more than the work. Postings we found used “Member of Technical Staff, LLM Evaluation”, “Machine Learning Engineer, AI Evaluation & LLM Systems” and “AI Engineer, Evaluation”, so search for the work (“LLM evaluation”, “evals”), not one title.

AI evaluation vs traditional QA: what changes when code is non-deterministic?

Traditional QA checks that a program returns an exact expected result. AI evaluation checks that a non-deterministic system scores well enough, often enough, on a representative sample.

	Traditional QA	AI evaluation
Pass condition	Output equals the expected value	Score above a threshold, aggregated over many cases
The oracle	The spec	A reference answer, a human label or a judge model
Flaky results	A bug	Built in; handled with thresholds, repeats and bigger samples
Cost per run	Near zero	Real money when judges are LLM calls

Regression suites, versioned test data and fast-checks-first all carry over. What’s new is reasoning about distributions instead of single outputs.

The core tooling stack: Ragas, DeepEval, TruLens and LangSmith

You need one metrics library, and optionally one platform for traces and dashboards. Versions are the ones installed for this post in October 2026.

Tool	What it is	Best for
Ragas 0.4.3	Apache-2.0 library of RAG and agent metrics	Scoring retrieval and answers
DeepEval 4.2.8	Apache-2.0 pytest-style testing framework	Eval suites that gate CI
TruLens 2.15.0	MIT tracing and “feedback function” library from Snowflake	Evaluating instrumented app traces
LangSmith SDK 0.14.4	MIT client for LangChain’s hosted platform	Hosted traces, datasets and human review

Start with DeepEval if your goal is a CI gate, or Ragas if you care most about retrieval quality. Three things we hit while installing them:

• pip install ragas failed on import: ModuleNotFoundError: No module named 'langchain_community.chat_models.vertexai', because pip picked langchain-community 0.4.2. Pinning langchain-community<0.4 fixed it.
• Ragas metrics moved to ragas.metrics.collections in 0.4.x, and Ragas has no toxicity metric. Older tutorials import from ragas.metrics.
• TruLens judge providers are separate packages from trulens itself.

4 evaluation dimensions you must master

Most RAG evaluation comes down to four questions, each comparing a different pair of things. All four score from 0 to 1, and the definitions below come from the libraries’ source code.

• Faithfulness: splits the answer into claims and scores the share the retrieved context supports. Low means hallucination. It can’t tell you the context was the wrong document.
• Answer relevance: Ragas generates questions from the answer and compares them with the real question using embeddings; DeepEval judges each statement’s relevance. Low means evasive or off-topic. A confident wrong answer can still score well.
• Context recall: splits a reference answer into statements and scores how many the retrieved context supports. Low means retrieval missed something, so fix the retriever, not the prompt. It’s the only one that needs a reference answer.
• Toxicity: DeepEval judges each opinion in the answer. In DeepEval 4.2.8 the score is the share judged not toxic, and the test passes at or above the threshold; some older guides describe the opposite, so check your version. It’s a regression test, not a safety system.

How to implement LLM-as-a-judge in a GitHub Actions CI/CD pipeline

An LLM eval gate runs on pull requests that touch prompts, retrieval or model settings. It runs cheap deterministic checks first, then the judge metrics, and blocks the merge when scores drop below thresholds.

The example is a small repository: a toy RAG app (rag_app.py, keyword retrieval plus a Claude answer), a golden dataset, a retrieval test, judge tests and the workflow. Everything except the judge calls was run for this post; we didn’t use an API key, so no judge scores are shown.

1. A golden dataset

evals/golden.jsonl has one real question per line, with a reference answer and the document that contains it:

{"input": "How long do refunds take to appear?", "expected_output": "Refunds take 5 to 10 business days to appear on the original payment method.", "source_doc": "refunds"}

2. Deterministic checks first

evals/test_retrieval.py checks that the right document is retrieved and that it ranks first. It’s free and takes milliseconds:

"""Deterministic retrieval check: free, fast, runs before any LLM judge."""
import json
import pathlib

from rag_app import DOCS, retrieve

GOLDEN = [json.loads(line) for line in
          pathlib.Path(__file__).with_name("golden.jsonl").read_text().splitlines()]


def hit_rate(k: int) -> float:
    hits = sum(DOCS[case["source_doc"]] in retrieve(case["input"], k=k) for case in GOLDEN)
    return hits / len(GOLDEN)


def test_right_doc_is_retrieved():
    assert hit_rate(k=2) >= 0.9


def test_right_doc_ranks_first():
    assert hit_rate(k=1) >= 0.75


if __name__ == "__main__":
    for k in (1, 2, 3):
        print(f"k={k}: hit rate {hit_rate(k):.2f}")

Both checks earn their place. A pull request added a general FAQ page to the knowledge base. The right document stayed in the top two, but the FAQ now ranked first for every question:

=================================== FAILURES ===================================
__________________________ test_right_doc_ranks_first __________________________
    def test_right_doc_ranks_first():
>       assert hit_rate(k=1) >= 0.75
E       assert 0.0 >= 0.75
E        +  where 0.0 = hit_rate(k=1)
evals/test_retrieval.py:21: AssertionError
=========================== short test summary info ============================
FAILED evals/test_retrieval.py::test_right_doc_ranks_first - assert 0.0 >= 0.75
1 failed, 1 passed in 0.16s

On main, both tests pass.

3. The LLM-as-a-judge tests

evals/test_rag.py runs the app on each case and scores it with four DeepEval metrics, with Claude as the judge:

"""LLM-as-a-judge regression tests. Run with: deepeval test run evals/test_rag.py"""
import json
import pathlib

import pytest
from deepeval import assert_test
from deepeval.metrics import (AnswerRelevancyMetric, ContextualRecallMetric,
                              FaithfulnessMetric, ToxicityMetric)
from deepeval.models import AnthropicModel
from deepeval.test_case import LLMTestCase

from rag_app import answer, retrieve

JUDGE = AnthropicModel(model="claude-opus-5-5")
GOLDEN = [json.loads(line) for line in
          pathlib.Path(__file__).with_name("golden.jsonl").read_text().splitlines()]


@pytest.mark.parametrize("case", GOLDEN, ids=[c["input"][:40] for c in GOLDEN])
def test_rag_answer(case):
    contexts = retrieve(case["input"])
    test_case = LLMTestCase(
        input=case["input"],
        actual_output=answer(case["input"], contexts),
        expected_output=case["expected_output"],
        retrieval_context=contexts,
    )
    assert_test(test_case, [
        FaithfulnessMetric(threshold=0.9, model=JUDGE),
        AnswerRelevancyMetric(threshold=0.8, model=JUDGE),
        ContextualRecallMetric(threshold=0.8, model=JUDGE),
        ToxicityMetric(threshold=0.9, model=JUDGE),  # DeepEval 4.x: higher = less toxic
    ])

One trap: a plain pytest run from the repository root failed with ModuleNotFoundError: No module named 'rag_app'. Two lines in pyproject.toml fix it:

[tool.pytest.ini_options]
pythonpath = ["."]

4. The workflow

.github/workflows/llm-eval.yml passes actionlint 1.7.12 with no findings. requirements.txt pins deepeval==4.2.8:

name: llm-eval

on:
  pull_request:
    paths:
      - "rag_app.py"
      - "prompts/**"
      - "evals/**"
  workflow_dispatch:

concurrency:
  group: llm-eval-${{ github.ref }}
  cancel-in-progress: true

permissions:
  contents: read

jobs:
  eval:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip
      - run: pip install -r requirements.txt
      - name: Deterministic checks (free, run first)
        run: pytest evals/test_retrieval.py -q
      - name: LLM-as-a-judge evals
        if: ${{ github.event.pull_request.head.repo.fork != true }}
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          DEEPEVAL_TELEMETRY_OPT_OUT: "1"
        run: deepeval test run evals/test_rag.py --display failing

Add ANTHROPIC_API_KEY as a repository secret and make the eval job a required status check, so a failing eval blocks the merge.

What makes the gate trustworthy

• Cheap checks first: the judge step only runs if retrieval passed.
• Forks don’t get secrets: the if: skips the judge for fork pull requests instead of failing on a missing key.
• Pin the judge model and versions: a judge upgrade changes scores even when your app didn’t, so upgrade it in its own pull request.
• Plan for noise and cost: set thresholds a little below current scores, measure spread with deepeval test run --repeat, and keep runs to relevant pull requests. For hard spend limits, see LLM API budget caps.

2 resume-worthy evaluation projects to build today

Project 1: a RAG regression gate in CI

Turn the example into a public repository over a real document set, such as an open-source project’s docs:

1. Build a golden dataset of 50 or more questions, including unanswerable ones where “I don’t know” is correct.
2. Add deterministic checks and the four judge metrics, gated in GitHub Actions.
3. Open one pull request that breaks retrieval and one that fixes it, and link the red and green runs from the README.

It proves you can turn “the bot seems worse” into a check that fails before merge.

Project 2: an LLM-judge calibration study

Most eval suites never check their judge. Show that you do:

1. Generate about 100 answers, including deliberately degraded ones (wrong document, injected false claim).
2. Label each one yourself for faithfulness.
3. Run your judge on the same answers and measure agreement with sklearn.metrics.cohen_kappa_score(human_labels, judge_labels).
4. Change one thing at a time (rubric, judge model, examples), re-measure, and publish what moved agreement.

It proves you know an eval is a measurement that needs validating, which is the core of the job.

Salary expectations and interview questions for AI evaluation roles

Posted pay tracks ML and AI engineering at the same level. Two US postings, as listed in 2026:

• Microsoft, “Member of Technical Staff, LLM Evaluation”: US base $139,900 to $274,800 at IC5 and $163,000 to $296,400 at IC6, higher in the Bay Area and New York.
• Apple, “Machine Learning Engineer, AI Evaluation & LLM Systems”, Cupertino: base $150,400 to $225,300, plus equity.

These are data points, not averages. Contractor “LLM evaluator” roles that rate model outputs are a different, usually hourly, job.

Interview questions to prepare for

1. How would you evaluate a RAG chatbot from scratch? Dataset, separate retrieval and answer metrics, a judge, and how you’d validate it.
2. Faithfulness is high but users say answers are wrong. Why? Faithful to the wrong context: check context recall.
3. How do you know your judge is right? Agreement with human labels, reading the disagreements, re-checking after judge changes.
4. Scores dropped 5 points after a model upgrade. Ship? Find which cases changed, rule out judge noise with repeats, weigh the metrics the product depends on.
5. How would you keep CI eval costs down? Deterministic checks first, path filters, cheaper judges where agreement holds.
6. What’s position or verbosity bias in judges? Favouring the first option or longer answers, as the MT-Bench paper (Zheng et al., 2023) showed; swap order and control for length.

A 12-week roadmap from software engineer

1. Weeks 1 to 3: build a RAG app and a small two-tool agent. You can’t evaluate failures you haven’t seen.
2. Weeks 4 to 6: learn the four metrics in DeepEval or Ragas, and hand-label your first 50 cases.
3. Weeks 7 to 9: build the CI gate, plus one custom rubric judge for something the libraries don’t cover.
4. Weeks 10 to 12: run the calibration study and publish both projects.

Statistics helps more than deep learning theory: sampling, confidence intervals and inter-rater agreement come up constantly. For RAG background, see agent memory vs RAG vs long context.

FAQ

Do I need a machine learning degree?

No. It’s an engineering role: Python, testing discipline and knowing how LLM apps fail matter more than research credentials. A public eval gate plus a calibrated judge is strong evidence on its own.

Which should I learn first, Ragas or DeepEval?

DeepEval if you want CI gates, since its tests run under pytest; Ragas if retrieval quality is your focus. The concepts transfer.

Can I use a local model as the judge to save money?

Yes, both libraries accept custom models. Check its agreement with your human labels first: a cheaper judge only saves money if it’s still right.

Bottom line

An AI evaluation engineer turns “the AI seems worse” into a number, a threshold and a failing check before release. Learn the four metrics, build a CI gate with deterministic checks before LLM judges, and prove your judge agrees with people. Those two projects cover most of what an evaluation interview will ask.