Reference Example

Evaluation & Results

Methodology, rubric, and the full reference-deployment results — plus how to reproduce them with the bundled harness.

Methodology

The curator was evaluated against a manually curated 281-query hand-labeled suite (example/eval/MCP_Evaluation_Sheet_-_TF_IDF_Tool_Curator.xlsx), spanning multiple personas and seven query categories:

Category	Count
Single-Tool Direct	189
Multi-Tool Complex	25
Semantic / Paraphrase	24
Edge Case / Negative	21
Access Denied	7
Need Clarification	7
No Tool	8

Each case was labeled Correct, Partially Correct (some but not all expected tools selected), Wrong (incorrect tool invoked), or Skipped (no suitable tool surfaced). The accuracy score:

Accuracy Score = (Correct + 0.5 × Partially Correct) / Total Queries

Access Denied / Need Clarification / No Tool aren't curator questions

These three categories test full-system behavior (persona access control, the model's own decision to ask a clarifying question) rather than retrieval — the curator has no mechanism to signal "no relevant tool exists." curator_eval.py reports these separately rather than scoring them against hit@k.

Results

No curator 100% accuracy 87 tools 11,860 tokens; TF-IDF curator 88.4% accuracy 33 tools 9,135 tokens; hybrid curator 94.8% accuracy 15 tools 1,561 tokens; -86.8% prompt tokens — Accuracy and token cost across all three generations, reference deployment.

Version	Correct	Partial	Wrong	Skipped	Correct %	Accuracy score
No curator, original descriptions	~225/281	—	—	—	~80%	~80%
No curator, refined descriptions	281/281	0	0	0	100.0%	100.0%
TF-IDF curator	235/281	27	9	10	83.6%	88.4%
Hybrid curator	262/281	9	3	7	93.2%	94.8%

The 100% no-curator row is a ceiling, not a deployment target — it costs roughly 11,860 prompt tokens per query (87 role-permitted tools sent unfiltered, before the user has even asked anything). The curator’s goal is approaching that ceiling while cutting the token cost by an order of magnitude.

Token impact

Version	Tools sent	Prompt tokens, first student query
No curator	87	~11,860
TF-IDF curator	~33 (top-15 + all action tools)	~9,135
Hybrid curator	~15	~1,561

The hybrid curator’s 86.8% token reduction relative to the no-curator baseline comes with a 93.2%→94.8% accuracy improvement over the TF-IDF version, not a trade-off against it — the gap between TF-IDF’s ~33 tools and the hybrid’s ~15 is mostly TF-IDF’s blanket “always include every action tool” rule, versus the hybrid’s targeted, retrieval-driven dependency expansion.

94.8%

accuracy score, hybrid curator

86.8%

token reduction vs. no curator

~20ms

curator overhead per query

<2%

of total response latency

Per-query latency breakdown: roughly 15ms for BGE query encoding, under 1ms each for FAISS and BM25 search — under 2% of total response latency against the 1–3 second range typical of the LLM call itself.

Reproducing these numbers

cd example
python eval/collect_tools.py                 # snapshot all 191 tool descriptions once
python eval/curator_eval.py --top-k 15        # hit@k / recall@k against the bundled sheet
python eval/curator_eval.py --top-k 15 --lenient   # score using equivalences.json covers/aliases

No live LLM call happens in either script — collect_tools.py needs the MCP servers reachable once, and curator_eval.py runs entirely against the cached snapshot afterward. See Configuration & Domain Synonyms for the iteration loop this harness is meant to support on your own domain.