Evaluation engine

Core

report_analyst/ · Reciprocal Public License. This is the open core: the Streamlit analyst, chunking, question sets, and benchmarking. It ships in the core Docker image.

EvaluationEngine scores a retrieval run against a reference dataset. The members below are generated from the method docstrings.

report_analyst.core.benchmark.evaluation_engine.EvaluationEngine

Engine for evaluating retrieval performance against benchmark datasets

evaluate_retrieval(dataset, retrieval_results, config, k_values=None)

Evaluate retrieval results against ground truth

Parameters:

Name	Type	Description	Default
dataset	BenchmarkDatasetContent	Benchmark dataset with ground truth	required
retrieval_results	Dict[str, List[Dict]]	Dict mapping question_id to list of retrieved chunks	required
config	RetrievalConfig	Retrieval configuration used	required
k_values	Optional[List[int]]	List of K values to compute metrics for	None

Returns:

Type	Description
EvaluationMetrics	EvaluationMetrics with computed scores

compare_flexible_datasets(reference_dataset, input_dataset, k_values=None)

Compare two flexible benchmark datasets (supports both IR and IE).

The reference dataset is treated as ground truth (e.g., "climretrieve", "chatreport"). The input dataset contains the actual results to evaluate.

For IR datasets: Compares retrieved chunks (by chunk_id and position). For IE datasets: Compares answers/analysis (by query_id, comparing text similarity).

Parameters:

Name	Type	Description	Default
reference_dataset	BenchmarkDataset	Reference dataset (ground truth)	required
input_dataset	BenchmarkDataset	Input dataset (actual results) to evaluate	required
k_values	Optional[List[int]]	List of K values to compute metrics for (only used for IR)	None

Returns:

Type	Description
EvaluationMetrics	EvaluationMetrics comparing input_dataset against reference_dataset