Reference Example

Failure Analysis

19 of 281 reference-deployment cases didn't come back fully correct: 3 wrong, 7 skipped, 9 partially correct. The patterns matter more than the individual cases.

Mindmap of TF-IDF and hybrid curator failure cases by category, with example queries and root causes — All remaining failure cases for both the TF-IDF and hybrid curator generations, grouped by root cause.

Wrong (3 cases)

Two of the three involve mark_attendance versus update_attendance — near-identical faculty tools on the same server. This reads as a model reasoning failure, not a retrieval miss: both tools were very likely retrieved correctly, and the model picked the wrong one between them. The fix lives in the descriptions, not the curator — sharpening the distinction (“recording a new session” vs. “correcting an existing entry”) gives the model less room to guess wrong between two tools it can both already see.

The third involves a query about students “struggling or at risk of failing,” expecting two tools (get_my_teaching_assignments + get_students_needing_attention) — a synonym entry existed but wasn’t enough on its own. Called the single hardest query in the set: a vocabulary gap and a multi-tool requirement compounding each other.

Skipped (7 cases, two distinct causes)

Pool crowding (3 cases). Single-tool, plainly worded queries with obvious vocabulary overlap — but from large-pool roles (faculty/admin) where several near-matches collectively push the correct tool just outside the top-k. This is explicitly a top_k / anchor-threshold problem, not a vocabulary problem — no amount of synonym tuning fixes a tool that’s ranked 16th when top_k=15.

Vocabulary gap (1 case). “Wrongly marked absent… get it corrected” was expected to resolve to submit_grievance; “corrected” simply doesn’t connect to “grievance” for either retriever. Directly fixable via domain_synonyms.json.

Abstraction + multi-tool (3 cases). Heavily abstract, jargon-paraphrased, multi-tool queries — e.g. “identify learners exhibiting concurrent academic underperformance…” — are the hardest combination this curator handles. Abstraction alone is recoverable with synonyms; abstraction combined with multiple required tools compounds the difficulty in a way no single fix fully resolves.

Partially correct (9 cases)

Mostly multi-tool queries missing exactly one expected tool, for one of three reasons:

A compound-split marker fires, but one half’s retrieval still falls short on its own.
The query uses plain “and”, which by design doesn’t trigger compound splitting (see Tool Curator Architecture for why).
A “give me everything” dashboard-style query carries three or four distinct intents, exceeding the per-sub-query top-k budget — typically two or three tools come through and the last doesn’t.

What this taxonomy is for

None of the four root causes above are university-specific — near-identical-tool confusion, pool crowding in large role-scoped tool sets, vocabulary gaps, and abstraction-plus-multi-tool queries are patterns any deployment of this curator is likely to hit, just with different tool names. See Lessons Learned for the fix that goes with each.