Phase 3: Evaluation Harness

Note

Ollama required for live scores. This page shows the evaluation framework and test set. Running RAGAS to produce actual scores requires Ollama running with llama3 and the ECB speeches ingested into ChromaDB. See RAG Testing Methodology for the full approach.

Related issues: #9 Gold-standard test set (closed), #10 RAGAS scoring (closed), #11 Wrap evaluation in pytest (closed), #12 Publish results (closed).

1 Test question set

tests/test_questions.json contains 20 fixed questions drawn from ECB monetary policy topics. The set is versioned and does not change between evaluation runs.

Code

import sys, json
sys.path.insert(0, "..")

from pathlib import Path
from IPython.display import HTML, display

with open("../tests/test_questions.json") as f:
    questions = json.load(f)

print(f"Total questions : {len(questions)}")
print(f"In corpus       : {sum(1 for q in questions if q.get('in_corpus', True))}")
print(f"Abstention      : {sum(1 for q in questions if not q.get('in_corpus', True))}")

all_tags = sorted({t for q in questions for t in q.get('tags', [])})
print(f"Tag types       : {all_tags}")

Total questions : 20
In corpus       : 19
Abstention      : 1
Tag types       : ['abstention', 'analytical', 'climate', 'communication', 'digital', 'factual', 'geopolitical', 'inflation', 'instruments', 'mandate', 'measurement', 'mechanism', 'programmes', 'rates', 'temporal']

1.1 Question breakdown by tag

Code

from collections import Counter

tag_counts = Counter(t for q in questions for t in q.get("tags", []))

rows = "".join(
    f"<tr><td style='padding:4px 8px'>{tag}</td>"
    f"<td style='padding:4px 8px'>{count}</td></tr>"
    for tag, count in sorted(tag_counts.items())
)
display(HTML(
    "<table style='border-collapse:collapse'>"
    "<thead><tr style='border-bottom:2px solid #dee2e6'>"
    "<th style='padding:4px 8px'>Tag</th><th style='padding:4px 8px'>Count</th>"
    "</tr></thead><tbody>" + rows + "</tbody></table>"
))

Tag	Count
abstention	1
analytical	10
climate	1
communication	3
digital	1
factual	9
geopolitical	1
inflation	3
instruments	2
mandate	2
measurement	1
mechanism	1
programmes	1
rates	3
temporal	5

1.2 Sample questions

Code

sample_ids = ["q001", "q005", "q010", "q018"]
sample = [q for q in questions if q["id"] in sample_ids]

rows = "".join(
    f"<tr>"
    f"<td style='padding:6px 8px;font-family:monospace'>{q['id']}</td>"
    f"<td style='padding:6px 8px'>{q['question']}</td>"
    f"<td style='padding:6px 8px;text-align:center'>{'✓' if q.get('in_corpus') else '✗ abstention'}</td>"
    f"<td style='padding:6px 8px'>{', '.join(q.get('tags', []))}</td>"
    f"</tr>"
    for q in sample
)
display(HTML(
    "<table style='width:100%;border-collapse:collapse'>"
    "<thead><tr style='border-bottom:2px solid #dee2e6'>"
    "<th style='padding:6px 8px'>ID</th><th style='padding:6px 8px'>Question</th>"
    "<th style='padding:6px 8px'>In corpus</th><th style='padding:6px 8px'>Tags</th>"
    "</tr></thead><tbody>" + rows + "</tbody></table>"
))

ID	Question	In corpus	Tags
q001	What is the ECB's quantitative definition of price stability?	✓	factual, mandate
q005	How does the ECB describe the transmission mechanism of monetary policy?	✓	analytical, mechanism
q010	What was the ECB's largest single interest rate increase and when did it occur?	✓	factual, temporal, rates
q018	What is the ECB's policy on interest rates as of 2025?	✗ abstention	abstention, temporal

2 Evaluation pipeline

rag.evaluate provides three functions that compose into the full evaluation loop:

Code

display(HTML(
    "<table style='width:100%;border-collapse:collapse'>"
    "<thead><tr style='border-bottom:2px solid #dee2e6'>"
    "<th style='padding:6px;text-align:left'>Function</th>"
    "<th style='padding:6px;text-align:left'>Input → Output</th>"
    "</tr></thead><tbody>"
    "<tr><td style='padding:6px'><code>load_test_questions()</code></td>"
    "<td style='padding:6px'>JSON path → list of question dicts (validates required keys)</td></tr>"
    "<tr><td style='padding:6px'><code>run_evaluation()</code></td>"
    "<td style='padding:6px'>Chain + questions → RAGAS scores + per-question results dict</td></tr>"
    "<tr><td style='padding:6px'><code>save_results()</code></td>"
    "<td style='padding:6px'>Results dict → <code>tests/results/phase_N_YYYYMMDD.json</code></td></tr>"
    "</tbody></table>"
))

Function	Input → Output
`load_test_questions()`	JSON path → list of question dicts (validates required keys)
`run_evaluation()`	Chain + questions → RAGAS scores + per-question results dict
`save_results()`	Results dict → `tests/results/phase_N_YYYYMMDD.json`

2.1 Running the evaluation locally

# Requires: ollama serve + ChromaDB populated (run scripts/ollama_ready.py first)
from rag.query import load_vectorstore, build_rag_chain
from rag.evaluate import load_test_questions, run_evaluation, save_results

vectorstore = load_vectorstore()
chain = build_rag_chain(vectorstore, model="llama3", k=4)
questions = load_test_questions("tests/test_questions.json")

results = run_evaluation(chain, questions, phase="phase_2")
out = save_results(results)
print(f"Results saved to {out}")
print(results["scores"])

3 RAGAS metrics

The four metrics scored for every question:

Code

metrics = [
    ("Faithfulness", "Does the answer contradict the retrieved chunks?",
     "Low → model is hallucinating or ignoring context"),
    ("Answer Relevancy", "Does the answer address the question?",
     "Low → answer is off-topic or evasive"),
    ("Context Precision", "Are retrieved chunks relevant to the question?",
     "Low → retriever is pulling back noise"),
    ("Context Recall", "Does the retrieved context contain what's needed?",
     "Low → relevant material isn't being found"),
]

rows = "".join(
    f"<tr><td style='padding:6px'><strong>{m}</strong></td>"
    f"<td style='padding:6px'>{what}</td>"
    f"<td style='padding:6px'>{interp}</td></tr>"
    for m, what, interp in metrics
)
display(HTML(
    "<table style='width:100%;border-collapse:collapse'>"
    "<thead><tr style='border-bottom:2px solid #dee2e6'>"
    "<th style='padding:6px'>Metric</th><th style='padding:6px'>Measures</th>"
    "<th style='padding:6px'>Low score means</th>"
    "</tr></thead><tbody>" + rows + "</tbody></table>"
))

Metric	Measures	Low score means
Faithfulness	Does the answer contradict the retrieved chunks?	Low → model is hallucinating or ignoring context
Answer Relevancy	Does the answer address the question?	Low → answer is off-topic or evasive
Context Precision	Are retrieved chunks relevant to the question?	Low → retriever is pulling back noise
Context Recall	Does the retrieved context contain what's needed?	Low → relevant material isn't being found

4 Results

Results files are saved to tests/results/ after each evaluation run and named phase_<label>_<YYYYMMDD>.json. The table below is populated from the most recent file if one exists.

Code

import os
results_dir = Path("../tests/results")

if results_dir.exists() and any(results_dir.glob("*.json")):
    latest = sorted(results_dir.glob("*.json"))[-1]
    with open(latest) as f:
        r = json.load(f)

    print(f"Phase     : {r.get('phase')}")
    print(f"Timestamp : {r.get('timestamp')}")
    print(f"Questions : {r.get('n_questions')}")
    print()
    scores = r.get("scores", {})
    for metric, value in scores.items():
        bar = "█" * int(value * 20)
        print(f"  {metric:<22} {value:.3f}  {bar}")
else:
    print("No results files found in tests/results/.")
    print("Run the evaluation locally with Ollama to generate scores.")

Phase     : phase_2
Timestamp : 2026-05-15T14:23:17
Questions : 10

  faithfulness           0.890  █████████████████
  answer_relevancy       0.786  ███████████████
  context_precision      0.714  ██████████████
  context_recall         0.673  █████████████

4.1 Per-question breakdown

Code

if results_dir.exists() and any(results_dir.glob("*.json")):
    latest = sorted(results_dir.glob("*.json"))[-1]
    with open(latest) as f:
        r = json.load(f)

    per_q = r.get("results", [])
    if per_q:
        def score_cell(v):
            if v is None:
                return "<td style='padding:5px 8px;text-align:center;color:#999'>—</td>"
            colour = "#d4edda" if v >= 0.80 else "#fff3cd" if v >= 0.60 else "#f8d7da"
            return f"<td style='padding:5px 8px;text-align:center;background:{colour}'>{v:.2f}</td>"

        rows = ""
        for q in per_q:
            conf_badge = (
                "<span style='color:#28a745;font-weight:bold'>✓ high</span>"
                if q.get("confidence") == "high"
                else "<span style='color:#dc3545;font-weight:bold'>⚠ low</span>"
            )
            rows += (
                f"<tr>"
                f"<td style='padding:5px 8px;font-family:monospace'>{q['id']}</td>"
                f"<td style='padding:5px 8px;max-width:320px'>{q['question']}</td>"
                f"<td style='padding:5px 8px;text-align:center'>{conf_badge}</td>"
                + score_cell(q.get("faithfulness"))
                + score_cell(q.get("answer_relevancy"))
                + score_cell(q.get("context_precision"))
                + score_cell(q.get("context_recall"))
                + f"</tr>"
            )

        display(HTML(
            "<table style='width:100%;border-collapse:collapse;font-size:0.9em'>"
            "<thead><tr style='border-bottom:2px solid #dee2e6'>"
            "<th style='padding:5px 8px'>ID</th>"
            "<th style='padding:5px 8px'>Question</th>"
            "<th style='padding:5px 8px;text-align:center'>Confidence</th>"
            "<th style='padding:5px 8px;text-align:center'>Faith.</th>"
            "<th style='padding:5px 8px;text-align:center'>Ans.Rel.</th>"
            "<th style='padding:5px 8px;text-align:center'>Ctx.Prec.</th>"
            "<th style='padding:5px 8px;text-align:center'>Ctx.Rec.</th>"
            "</tr></thead><tbody>" + rows + "</tbody>"
            "<tfoot><tr><td colspan='7' style='padding:4px 8px;font-size:0.85em;color:#666'>"
            "Colour key: <span style='background:#d4edda;padding:1px 6px'>≥ 0.80</span> "
            "<span style='background:#fff3cd;padding:1px 6px'>0.60–0.79</span> "
            "<span style='background:#f8d7da;padding:1px 6px'>&lt; 0.60</span>"
            "</td></tr></tfoot>"
            "</table>"
        ))
    else:
        print("Results file found but contains no per-question data.")
else:
    print("No results files found in tests/results/.")
    print("Run the evaluation locally with Ollama to generate scores.")

ID	Question	Confidence	Faith.	Ans.Rel.	Ctx.Prec.	Ctx.Rec.
q001	What is the ECB's quantitative definition of price stability?	✓ high	0.92	0.88	0.83	0.79
q002	When did the ECB begin raising interest rates in the 2022 tightening cycle and by how much?	✓ high	0.87	0.85	0.78	0.74
q003	What tools does the ECB use to implement monetary policy?	✓ high	0.89	0.82	0.71	0.68
q004	What is the Pandemic Emergency Purchase Programme (PEPP) and why was it introduced?	✓ high	0.94	0.91	0.86	0.82
q005	How does the ECB describe the transmission mechanism of monetary policy?	✓ high	0.85	0.79	0.73	0.71
q006	What was the ECB's assessment of core inflation in the euro area in 2023?	✓ high	0.88	0.84	0.76	0.69
q007	What is the Transmission Protection Instrument (TPI)?	✓ high	0.91	0.87	0.82	0.77
q008	How has the ECB approached forward guidance in its communication?	✓ high	0.83	0.81	0.74	0.70
q009	What does the ECB say about the role of wages in the inflation outlook?	✓ high	0.86	0.78	0.69	0.65
q018	What is the ECB's policy on interest rates as of 2025?	⚠ low	0.95	0.31	0.22	0.18
Colour key: ≥ 0.80 0.60–0.79 < 0.60

Note

q018 note: The low answer relevancy and context scores for q018 are expected and correct. The corpus does not cover 2025 ECB policy, so the model correctly abstains rather than hallucinating. High faithfulness (0.95) reflects the model staying within the bounds of what was retrieved — in this case, correctly recognising there was nothing useful to say.

See RAG Testing Methodology for interpretation guidance and known limitations of each metric.