Building a RAG System I Can Explain

TU Dublin CPD Foundations of AI — Capstone

Alan McDonagh

2026-05-31

The problem

  • Solvency II guidance runs to thousands of pages
  • Existing search tools were slower than just reading
  • A colleague piloted RAG — useful, but:
    • No way to log or replay bad answers
    • No trust signal for the user
    • No ability to explain failures to stakeholders

The goal

Build something small enough to explain and evaluate honestly.

Two goals, equal weight:

  1. Understand how a RAG pipeline actually behaves
  2. Get better at explaining it to non-technical audiences

The stack

Layer Tool
RAG framework LangChain
Vector store ChromaDB (local)
LLM Ollama (air-gapped)
Evaluation RAGAS + pytest
Docs Sphinx + Quarto

Nothing leaves the laptop.

The approach — 5 phases

Phase Goal Status
0 Repo setup + hello world ✅ Complete
1 Document ingestion ✅ Complete
2 RAG query pipeline ✅ Complete
3 Evaluation harness (RAGAS) ✅ Complete
4 Feedback + confidence In progress

Evaluation design

Test questions drawn from source documents → known answers.

Check two things per question:

  • Did retrieval pull back the right chunks?
  • Is the answer grounded in what was retrieved?

Wrapped in pytest so regressions show up like broken unit tests.

Key ethical points

  • Accuracy risk: confident-sounding wrong answers in a regulated setting
  • Transparency: users must know what the system is and isn’t doing
  • KB completeness: missing docs = silent partial answers

None of these are reasons not to proceed — they shape how results are presented.

What’s built

  • Query logging — every RAG call appended to JSONL; failed queries flagged for replay
  • Confidence signal — is_uncertain, n_sources, unique_documents on every ask() call
  • Sphinx HTML docs — autodoc from rag/ module, zero-warning build
  • RAGAS evaluation — 20-question gold-standard test set; 4 metrics per run

What’s next

  • Run RAGAS evaluation with real CBI speeches corpus loaded
  • Enterprise deployment writeup (Phase 5)
  • Plain-English one-pager for non-technical stakeholders