Configuration & Domain Synonyms

Retrieval quality on a new domain is governed almost entirely by one JSON file, not by anything in the curator's internal logic — which is domain-agnostic by design and shouldn't need touching.

get_curator() parameters

domain_synonyms.json

This file supplies additional keyword text per tool, appended to that tool’s description before indexing — closing the gap between an embedding model’s general-domain training and your organization’s actual internal terminology, abbreviations, and phrasing.

{
  "submit_grievance": "raise complaint file issue lodge protest formal complaint",
  "get_account_balance": "check balance how much money available funds statement"
}

Keys are exact tool names. Values are space-separated keyword strings, not full sentences — both the embedding model and BM25 respond better to keyword density than to grammatically complete prose. Keys starting with _ are treated as comments and ignored by the loader. An annotated starting template lives at accuracy_boosters/tool_context.template.json.

Writing effective entries. For each tool, enumerate the different ways a real user might request it without knowing its technical name, then pull every distinct keyword out of those phrasings:

• Cover both directions of a binary concept. This is the single most commonly missed lever — for an eligibility-check tool, include both "eligible qualify allowed" and "barred denied blocked rejected ineligible". Users ask about both sides of the same check.
• Mix jargon, abbreviations, and plain language in the same entry — a single user population rarely uses only one register.
• For large tool sets, derive the first pass from existing evaluation queries rather than writing from scratch: aggregate every distinct word used across the test queries expected to resolve to a given tool. This grounds the synonym file in observed phrasing instead of guessed coverage.

Edits are picked up automatically — the disk cache key incorporates a hash of this file’s contents, so any change forces a rebuild on the next build_index() call.

dependency_rules.json (optional)

Some tools are only useful in combination with others. “Can I sit for my exams?” resolves cleanly to get_exam_eligibility, but a complete answer also needs get_attendance_summary and get_fee_dues — tools the query never mentions. Retrieval will sometimes surface these companions on its own; this file makes the inclusion deterministic instead of probabilistic.

{ "get_exam_eligibility": ["get_attendance_summary", "get_fee_dues"] }

from tool_curator import load_dependency_rules, apply_dependency_rules

rules    = load_dependency_rules("/path/to/dependency_rules.json")
selected = apply_dependency_rules(curator.select(query, pool), pool, rules)

Each key is a trigger tool; its value is the list of companions force-included whenever the trigger is selected (if present in the caller’s authorized pool). Add rules reactively — in response to a specific evaluation failure where a companion is consistently missing — not speculatively. Over-applying this inflates the result set and works against the token-reduction goal the curator exists to serve.

equivalences.json (evaluation-only)

This file has zero effect on runtime retrieval — it only makes evaluation scoring fair when one tool’s output subsumes another’s:

{
  "covers":  { "get_account_overview": ["get_account_balance", "get_recent_transactions", "get_account_overview"] },
  "aliases": { "tool_name_a": ["tool_name_b"] }
}

covers lists, for a comprehensive tool, every specific tool whose role it already fulfills (including itself). aliases handles two functionally-identical tools under different names — a naming inconsistency between two servers, or an accidental duplicate. Both are consumed only by curator_eval.py --lenient.

The evaluation loop

1. Build a labeled test set — each row maps a representative user query to the exact tool(s) expected to answer it. Cover 3–5+ queries per tool, deliberately including paraphrased/colloquial phrasing alongside direct phrasing; paraphrase coverage is typically where retrieval loses the most recall.
2. Snapshot the tool pool: python example/eval/collect_tools.py connects to your running servers once and caches every tool’s (name, description) pair to disk, so the eval script needs no live LLM or server connection on every run afterward.
3. Run the evaluation: python example/eval/curator_eval.py --top-k 15 [--lenient]. It reports hit@k (all expected tools for a query were present in the top-k selection), recall@k (fraction present, for multi-tool queries), a per-category breakdown, and per-query failure detail.
4. Diagnose and fix each failure using the table below, then re-run. Reaching stable, high recall typically takes several passes — the reference deployment went through roughly a dozen iterations before recall stabilized in the mid-90s.
5. Confirm before moving on — a recall regression right after an edit usually means an overly broad synonym entry is pulling in matches for unrelated tools. Narrow the entry rather than deleting it, and re-run.

Diagnostic reference