Phase 4 — Confidence Signal

Surfacing uncertainty to the user without extra LLM calls

1 Why a confidence signal matters

A RAG system that gives confident-sounding answers is more dangerous than one that admits uncertainty — the user can’t calibrate their trust in the output. Issue #14 adds a lightweight confidence signal to every ask() call. It costs nothing at inference time (no extra LLM calls, no RAGAS scoring) and gives the user three concrete signals to display.

---

2 The signal

from rag.query import confidence_signal, ABSTAIN_PHRASE

# What the signal contains
import inspect
print(inspect.getsource(confidence_signal))

def confidence_signal(
    answer: str,
    sources: list[dict],
    scores: list[float] | None = None,
    used_fallback: bool = False,
) -> dict:
    """Return a lightweight confidence estimate based on answer, sources, and scores.

    No additional LLM calls are made; this is a heuristic only.

    Signals:

    - ``is_uncertain``: ``True`` when the model admitted it could not find the answer
      (detected via :data:`ABSTAIN_PHRASE`).
    - ``n_sources``: number of chunks retrieved.
    - ``has_sources``: ``True`` when at least one chunk was retrieved.
    - ``unique_documents``: number of distinct source documents.
    - ``top_score``: highest similarity score across retrieved chunks, or ``None``.
    - ``avg_score``: mean similarity score, or ``None``.
    - ``confidence_level``: categorical label — ``"High"`` (top_score >= 0.8),
      ``"Medium"`` (>= 0.6), ``"Low"`` (>= 0.4), ``"Very Low"`` otherwise.
    - ``used_fallback``: ``True`` when the always-answer fallback fired.

    Args:
        answer: The answer string returned by :func:`ask`.
        sources: The list of metadata dicts returned by :func:`ask`.
        scores: Optional list of relevance scores parallel to *sources*.
        used_fallback: Pass ``True`` when the always-answer fallback was used.

    Returns:
        Dict with confidence fields described above.
    """
    is_uncertain = ABSTAIN_PHRASE in answer
    n_sources = len(sources)
    unique_docs = len({s.get("source", "") for s in sources if s.get("source")})

    top_score = round(max(scores), 3) if scores else None
    avg_score = round(sum(scores) / len(scores), 3) if scores else None

    if is_uncertain or n_sources == 0:
        level = "Very Low"
    elif top_score is None:
        level = "Unknown"
    elif top_score >= 0.8:
        level = "High"
    elif top_score >= 0.6:
        level = "Medium"
    elif top_score >= 0.4:
        level = "Low"
    else:
        level = "Very Low"

    return {
        "is_uncertain": is_uncertain,
        "n_sources": n_sources,
        "has_sources": n_sources > 0,
        "unique_documents": unique_docs,
        "top_score": top_score,
        "avg_score": avg_score,
        "confidence_level": level,
        "used_fallback": used_fallback,
    }

---

3 Demo — certain answer

sources = [
    {"source": "speech_042.txt", "speaker": "Lagarde"},
    {"source": "speech_107.txt", "speaker": "Lane"},
    {"source": "speech_042.txt", "speaker": "Lagarde"},  # duplicate file
]
answer = "The ECB targets inflation at 2% over the medium term."

sig = confidence_signal(answer, sources)
for k, v in sig.items():
    print(f"  {k:20s}: {v}")

  is_uncertain        : False
  n_sources           : 3
  has_sources         : True
  unique_documents    : 2
  top_score           : None
  avg_score           : None
  confidence_level    : Unknown
  used_fallback       : False

The model gave a real answer, retrieved 3 chunks from 2 distinct documents. is_uncertain is False — safe to present to the user.

---

4 Demo — uncertain answer (abstention)

sig = confidence_signal(ABSTAIN_PHRASE, [])
for k, v in sig.items():
    print(f"  {k:20s}: {v}")

  is_uncertain        : True
  n_sources           : 0
  has_sources         : False
  unique_documents    : 0
  top_score           : None
  avg_score           : None
  confidence_level    : Very Low
  used_fallback       : False

The model admitted it could not find an answer in context. is_uncertain is True — the UI should flag this and prompt the user to rephrase or check whether the topic is covered by the knowledge base.

---

5 What the signal looks like in ask()

confidence is returned alongside answer and sources in every ask() call — no extra parameters needed:

from rag.query import ask

result = ask(chain, "What is the ECB's inflation target?")

print(result["answer"])
# → "The ECB targets inflation at 2% over the medium term."

print(result["confidence"])
# → {"is_uncertain": False, "n_sources": 4, "has_sources": True, "unique_documents": 3}

---

6 Displaying uncertainty to the user

def render_answer(result: dict) -> None:
    """Print a formatted answer with confidence indicators."""
    conf = result["confidence"]
    answer = result["answer"]

    if conf["is_uncertain"]:
        label = "⚠️  LOW CONFIDENCE"
        note  = "The system could not find an answer in the loaded documents."
    elif not conf["has_sources"]:
        label = "⚠️  NO SOURCES"
        note  = "No chunks were retrieved. Check the knowledge base is populated."
    elif conf["n_sources"] < 2:
        label = "🟡 MODERATE"
        note  = f"Answer based on {conf['n_sources']} chunk from {conf['unique_documents']} document(s)."
    else:
        label = "✅ CONFIDENT"
        note  = (f"Retrieved {conf['n_sources']} chunks from "
                 f"{conf['unique_documents']} document(s).")

    print(f"[{label}]")
    print(f"Answer: {answer}")
    print(f"Note:   {note}")
    print()


# Simulate a successful retrieval
render_answer({
    "answer": "The ECB targets 2% inflation over the medium term.",
    "sources": [{"source": "s1.txt"}, {"source": "s2.txt"}, {"source": "s1.txt"}],
    "confidence": confidence_signal(
        "The ECB targets 2% inflation over the medium term.",
        [{"source": "s1.txt"}, {"source": "s2.txt"}, {"source": "s1.txt"}],
    ),
})

# Simulate an abstention
render_answer({
    "answer": ABSTAIN_PHRASE,
    "sources": [],
    "confidence": confidence_signal(ABSTAIN_PHRASE, []),
})

[✅ CONFIDENT]
Answer: The ECB targets 2% inflation over the medium term.
Note:   Retrieved 3 chunks from 2 document(s).

[⚠️  LOW CONFIDENCE]
Answer: I don't know based on the available speeches.
Note:   The system could not find an answer in the loaded documents.

---

7 Confidence signal at a glance

Field	Type	Meaning
is_uncertain	bool	Model returned the abstain phrase — answer not grounded
n_sources	int	Number of chunks retrieved from the vectorstore
has_sources	bool	At least one chunk was retrieved
unique_documents	int	Number of distinct source files in the retrieved set

Note

This is a heuristic signal, not a calibrated probability. A high n_sources is necessary but not sufficient for a good answer — retrieval can pull back irrelevant chunks. For rigorous scoring, run RAGAS via rag.evaluate.run_evaluation().

Tests: tests/test_query.py — 8 new tests covering confidence_signal() and the confidence key in ask() return values; all CI-safe (no Ollama required).