Lessons Learned

Practitioner guidance from building and evaluating this curator — none of it specific to universities.

The single largest move in this project wasn't a retrieval technique: it was rewriting tool descriptions.

Going from ~80% to 100% (no-curator baseline) with zero code changes, purely by tightening 191 descriptions into a contract format, means any curator’s accuracy ceiling is set by description quality before anything else. Check description quality before tuning top_k, the anchor threshold, or anything else — a curator cannot retrieve a distinction that the tool descriptions themselves don’t make.

Two components only earned their place through testing

• The BM25 zero-score guard — nobody designed for “what happens when a query shares zero keywords with every tool” upfront; it surfaced during evaluation. Anyone combining dense and sparse retrieval should specifically test the no-keyword-overlap case rather than assume it degrades gracefully on its own.
• Dependency expansion replacing blanket action-tool inclusion — the TF-IDF generation’s “always include every action tool” rule was simple but expensive and imprecise. Targeted, rule-based dependency expansion was both cheaper and more accurate (wrong-case count dropped from 9 to 3 across generations).

Skipped cases need two different fixes for two different causes

Vocabulary gaps get fixed in domain_synonyms.json. Pool crowding gets fixed by tuning top_k or the anchor threshold — per role, not globally, since crowding scales with pool size and a global top_k bump helps the large-pool roles at the cost of padding every small-pool role’s declarations for nothing. Reaching for the synonym file every time fixes the vocabulary case and does nothing for the crowding case — misdiagnosing which one you’re looking at wastes a full evaluation cycle.

Compound splitting is incomplete by design

Plain “and”/”or” are deliberately excluded from the split-marker pattern because they’re too common in ordinary single-intent queries. This means genuinely compound queries phrased with plain “and” won’t split, and “give me everything” queries carrying three-plus intents can still exceed the per-sub-query budget. This is a known, accepted gap, not an oversight — see Limitations & Future Work for where a query-rewriting step ahead of the curator could help.

Near-identical tool pairs need distinguishing language in both descriptions

When two tools do genuinely different things but read almost identically (mark_attendance vs. update_attendance), the failure that results usually isn’t a curator problem at all — both were probably retrieved correctly, and the model chose between them incorrectly. The fix is sharpening both descriptions so the distinction is unambiguous to the model reading them, not adjusting retrieval.

Separate curator failures from model failures

A case labeled “wrong” doesn’t always mean retrieval missed the right tool — sometimes the right tool was in the candidate set and the model picked the wrong one anyway. Conflating these two failure modes leads to tuning retrieval parameters to fix a problem that actually lives in the tool descriptions or the model’s own reasoning, which burns evaluation cycles without moving the number that’s actually broken.