Setup & Environment Tests

1 Overview

This page renders the live results of the last environment and setup validation test run. The data is loaded from test-results.json at render time — re-render after running tests to see updated results.

Run tests with:

# Run all tests and save results (from repo root):
pytest tests/ -v --json-report --json-report-file=test-results.json

# Setup/environment checks only (requires Ollama running):
pytest tests/test_setup.py -v --json-report --json-report-file=test-results.json

# Ingestion pipeline tests only (no Ollama needed):
pytest tests/test_ingest.py -v --json-report --json-report-file=test-results.json

All tests passed | 130 passed 0 failed 2 skipped | 31.17s

Status	Class	Test	Duration
✅ PASSED	test_app_file_exists	App file exists	0.001s
✅ PASSED	test_app_imports_from_rag_query	App imports from rag query	0.001s
✅ PASSED	test_app_calls_ask	App calls ask	0.001s
✅ PASSED	test_app_calls_build_rag_chain	App calls build rag chain	0.001s
✅ PASSED	test_app_calls_load_vectorstore	App calls load vectorstore	0.001s
✅ PASSED	test_app_uses_cache_resource	App uses cache resource	0.001s
✅ PASSED	test_app_queries_ollama_for_model_list	App queries ollama for model list	0.001s
✅ PASSED	test_app_handles_uncertain_answers	App handles uncertain answers	0.001s
✅ PASSED	test_app_has_mode_selector	App has mode selector	0.001s
✅ PASSED	test_app_has_performance_timing	App has performance timing	0.001s
✅ PASSED	test_app_has_fallback_toggle	App has fallback toggle	0.001s
✅ PASSED	test_app_renders_without_exception	App renders without exception	0.385s
✅ PASSED	test_app_title_is_present	App title is present	0.023s
✅ PASSED	test_app_has_question_text_input	App has question text input	0.020s
✅ PASSED	test_app_has_ask_button	App has ask button	0.021s
✅ PASSED	test_app_ask_button_disabled_when_no_question	App ask button disabled when no question	0.020s
✅ PASSED	test_app_shows_answers_after_question_submitted	App shows answers after question submitted	0.064s
✅ PASSED	test_app_shows_error_gracefully_when_resources_fail	App shows error gracefully when resources fail	0.023s
✅ PASSED	test_ask_returns_expected_keys	Ask returns expected keys	0.001s
✅ PASSED	test_ask_confidence_certain_for_real_answer	Ask confidence certain for real answer	0.001s
✅ PASSED	test_ask_confidence_uncertain_for_abstain	Ask confidence uncertain for abstain	0.001s
✅ PASSED	test_ask_n_sources_matches_retrieved_docs	Ask n sources matches retrieved docs	0.001s
✅ PASSED	test_ask_unique_documents_counted_correctly	Ask unique documents counted correctly	0.001s
✅ PASSED	test_ask_sources_are_metadata_dicts	Ask sources are metadata dicts	0.001s
✅ PASSED	TestDiagramsQmd	File exists	0.000s
✅ PASSED	TestDiagramsQmd	Has rag architecture mermaid	0.001s
✅ PASSED	TestDiagramsQmd	Has fallback chain mermaid	0.001s
✅ PASSED	TestDiagramsQmd	Has paperbanana reference	0.001s
✅ PASSED	TestDiagramsQmd	Has issue link	0.001s
✅ PASSED	TestGenerateScript	File exists	0.001s
✅ PASSED	TestGenerateScript	Defines two diagrams	0.001s
✅ PASSED	TestGenerateScript	References paperbanana	0.001s
✅ PASSED	TestGenerateScript	Output dir points to assets	0.001s
✅ PASSED	TestNavRegistration	Diagrams in quarto yml	0.001s
✅ PASSED	TestOutputReport	Diagrams referenced in report	0.001s
✅ PASSED	TestAssetsDirectory	Assets dir exists	0.000s
✅ PASSED	test_questions_file_exists	Questions file exists	0.000s
✅ PASSED	test_questions_is_valid_json	Questions is valid json	0.002s
✅ PASSED	test_questions_minimum_count	Questions minimum count	0.001s
✅ PASSED	test_questions_required_keys	Questions required keys	0.001s
✅ PASSED	test_questions_unique_ids	Questions unique ids	0.001s
✅ PASSED	test_questions_no_empty_text	Questions no empty text	0.001s
✅ PASSED	test_questions_has_abstention_entries	Questions has abstention entries	0.001s
✅ PASSED	test_questions_tag_variety	Questions tag variety	0.001s
✅ PASSED	test_load_test_questions_returns_list	Load questions returns list	0.001s
✅ PASSED	test_load_test_questions_missing_file	Load questions missing file	0.000s
✅ PASSED	test_load_test_questions_bad_format	Load questions bad format	0.002s
✅ PASSED	test_save_results_writes_file	Save results writes file	0.002s
✅ PASSED	test_save_results_creates_parent_dirs	Save results creates parent dirs	0.002s
✅ PASSED	test_save_results_default_path_includes_phase	Save results default path includes phase	0.003s
✅ PASSED	test_run_evaluation_structure	Run evaluation structure	0.003s
✅ PASSED	test_load_speeches_csv	Load speeches csv	0.004s
✅ PASSED	test_chunk_documents	Chunk documents	0.003s
✅ PASSED	test_ingest_to_chromadb	Ingest to chromadb	0.005s
✅ PASSED	test_ingest_to_chromadb_missing_file	Ingest to chromadb missing file	0.000s
✅ PASSED	test_ingest_to_chromadb_empty_csv	Ingest to chromadb empty csv	0.005s
✅ PASSED	test_real_csv_loads	Real csv loads	0.918s
✅ PASSED	test_real_csv_metadata_shape	Real csv metadata shape	0.947s
✅ PASSED	test_real_csv_chunking	Real csv chunking	0.961s
✅ PASSED	test_chunk_quality_no_empty_chunks	Chunk quality no empty chunks	0.003s
✅ PASSED	test_chunk_quality_minimum_length	Chunk quality minimum length	0.003s
✅ PASSED	test_chunk_quality_metadata_preserved	Chunk quality metadata preserved	0.003s
✅ PASSED	test_chunk_quality_respects_chunk_size	Chunk quality respects chunk size	0.003s
✅ PASSED	test_real_chunk_quality	Real chunk quality	1.230s
✅ PASSED	test_is_failed_query_detects_sentinel	Is failed query detects sentinel	0.000s
✅ PASSED	test_is_failed_query_detects_sentinel_embedded	Is failed query detects sentinel embedded	0.000s
✅ PASSED	test_is_failed_query_returns_false_for_real_answer	Is failed query returns false for real answer	0.000s
✅ PASSED	test_is_failed_query_empty_string	Is failed query empty string	0.000s
✅ PASSED	test_log_query_returns_path	Log query returns path	0.002s
✅ PASSED	test_log_query_writes_valid_jsonl	Log query writes valid jsonl	0.002s
✅ PASSED	test_log_query_marks_failed_true	Log query marks failed true	0.002s
✅ PASSED	test_log_query_includes_timestamp	Log query includes timestamp	0.002s
✅ PASSED	test_log_query_includes_sources	Log query includes sources	0.002s
✅ PASSED	test_log_query_appends_multiple_entries	Log query appends multiple entries	0.002s
✅ PASSED	test_log_query_creates_parent_dirs	Log query creates parent dirs	0.002s
✅ PASSED	test_load_log_returns_list	Load log returns list	0.002s
✅ PASSED	test_load_log_returns_empty_for_missing_file	Load log returns empty for missing file	0.000s
✅ PASSED	test_replay_failed_filters_only_failed	Replay failed filters only failed	0.002s
✅ PASSED	test_replay_failed_empty_when_all_pass	Replay failed empty when all pass	0.002s
✅ PASSED	test_replay_failed_returns_empty_for_missing_file	Replay failed returns empty for missing file	0.000s
✅ PASSED	test_ask_logs_when_log_path_provided	Ask logs when log path provided	0.001s
✅ PASSED	test_ask_does_not_log_when_no_log_path	Ask does not log when no log path	0.001s
✅ PASSED	test_load_vectorstore_delegates	Load vectorstore delegates	0.000s
✅ PASSED	test_load_vectorstore_default_args	Load vectorstore default args	0.000s
✅ PASSED	test_load_vectorstore_returns_vectorstore	Load vectorstore returns vectorstore	0.001s
✅ PASSED	test_build_rag_chain_returns_chain	Build rag chain returns chain	0.003s
✅ PASSED	test_build_rag_chain_uses_similarity_search_with_scores	Build rag chain uses similarity search with scores	0.001s
✅ PASSED	test_build_rag_chain_passes_k_to_search	Build rag chain passes k to search	0.022s
✅ PASSED	test_build_rag_chain_filters_by_score_threshold	Build rag chain filters by score threshold	0.013s
✅ PASSED	test_build_rag_chain_uses_model	Build rag chain uses model	0.001s
✅ PASSED	test_confidence_signal_certain_answer	Confidence signal certain answer	0.000s
✅ PASSED	test_confidence_signal_uncertain_answer	Confidence signal uncertain answer	0.000s
✅ PASSED	test_confidence_signal_counts_unique_sources	Confidence signal counts unique sources	0.000s
✅ PASSED	test_confidence_signal_handles_missing_source_key	Confidence signal handles missing source key	0.000s
✅ PASSED	test_confidence_signal_empty_sources	Confidence signal empty sources	0.000s
✅ PASSED	test_confidence_signal_scores_populate_top_and_avg	Confidence signal scores populate top and avg	0.000s
✅ PASSED	test_confidence_signal_no_scores_gives_none	Confidence signal no scores gives none	0.000s
✅ PASSED	test_confidence_signal_high_level	Confidence signal high level	0.000s
✅ PASSED	test_confidence_signal_medium_level	Confidence signal medium level	0.000s
✅ PASSED	test_confidence_signal_low_level	Confidence signal low level	0.000s
✅ PASSED	test_confidence_signal_very_low_level_no_sources	Confidence signal very low level no sources	0.000s
✅ PASSED	test_confidence_signal_used_fallback_false_by_default	Confidence signal used fallback false by default	0.000s
✅ PASSED	test_confidence_signal_used_fallback_propagated	Confidence signal used fallback propagated	0.000s
✅ PASSED	test_ask_returns_answer_sources_confidence	Ask returns answer sources confidence	0.001s
✅ PASSED	test_ask_answer_value	Ask answer value	0.000s
✅ PASSED	test_ask_sources_length	Ask sources length	0.000s
✅ PASSED	test_ask_sources_are_metadata_dicts	Ask sources are metadata dicts	0.000s
✅ PASSED	test_ask_sources_contain_required_keys	Ask sources contain required keys	0.000s
✅ PASSED	test_ask_passes_question_to_chain	Ask passes question to chain	0.000s
✅ PASSED	test_ask_sources_speaker_metadata	Ask sources speaker metadata	0.000s
✅ PASSED	test_ask_confidence_is_certain_for_good_answer	Ask confidence is certain for good answer	0.000s
✅ PASSED	test_ask_confidence_is_uncertain_for_abstain	Ask confidence is uncertain for abstain	0.000s
✅ PASSED	test_ask_confidence_n_sources_matches_docs	Ask confidence n sources matches docs	0.000s
✅ PASSED	test_ask_returns_scores_key	Ask returns scores key	0.002s
✅ PASSED	test_ask_scores_populate_confidence_signal	Ask scores populate confidence signal	0.000s
✅ PASSED	test_ask_fallback_fires_on_abstain	Ask fallback fires on abstain	0.001s
✅ PASSED	test_ask_fallback_not_fired_without_fallback_llm	Ask fallback not fired without fallback llm	0.000s
✅ PASSED	test_ask_fallback_not_fired_for_good_answer	Ask fallback not fired for good answer	0.001s
✅ PASSED	TestDependencies	Langchain installed	0.001s
✅ PASSED	TestDependencies	Langchain ollama installed	0.000s
✅ PASSED	TestDependencies	Chromadb installed	0.000s
✅ PASSED	TestDependencies	Rag module importable	0.000s
✅ PASSED	TestOllamaService	Ollama service running	2.012s
✅ PASSED	TestOllamaService	Ollama models available	2.012s
✅ PASSED	TestOllamaModels	Default model available	2.018s
✅ PASSED	TestLLMProviders	Ollama provider	0.046s
✅ PASSED	TestLLMProviders	Openai provider optional	1.286s
⚠️ SKIPPED	TestLLMProviders	Anthropic provider optional	0.001s
⚠️ SKIPPED	TestLLMProviders	Gemini provider optional	0.001s
✅ PASSED	TestEnvironment	Python version	0.001s
✅ PASSED	TestEnvironment	Env file exists	0.000s
✅ PASSED	TestEnvironment	Data directory structure	0.000s

2 Test categories

2.1 `tests/test_setup.py` — environment checks (local only, requires Ollama)

Class	What it checks
`TestDependencies`	LangChain, ChromaDB, RAG module importable
`TestOllamaService`	Ollama running on `localhost:11434`
`TestOllamaModels`	Default model (`OLLAMA_MODEL` env var) downloaded
`TestLLMProviders`	Ollama required; OpenAI / Anthropic / Gemini optional
`TestEnvironment`	Python ≥ 3.11, `.env` exists, required directories present

2.2 `tests/test_ingest.py` — ingestion pipeline (runs in CI, mocked dependencies)

Test	What it checks
`test_load_speeches_csv`	CSV loads and converts to LangChain Documents with correct metadata
`test_chunk_documents`	Chunks preserve metadata and produce more chunks than source documents
`test_ingest_to_chromadb`	End-to-end ingestion with mocked Ollama and ChromaDB
`test_ingest_to_chromadb_missing_file`	Raises `FileNotFoundError` on missing CSV
`test_ingest_to_chromadb_empty_csv`	Raises `ValueError` on empty CSV
`test_real_csv_loads`	(local only) Real ECB CSV loads >100 speeches
`test_real_csv_metadata_shape`	(local only) First 10 real speeches have required metadata keys
`test_real_csv_chunking`	(local only) First 5 real speeches chunk correctly

3 Troubleshooting

Ollama not running

ollama serve   # start in a separate terminal

Model missing

ollama list                  # see installed models
ollama pull llama3.2:1b      # download the default

Import errors — re-sync dependencies:

uv sync

4 Next steps

Once all tests pass, proceed to Phase 1: Document Ingestion. See the Project Report for details.