Tool Curator Architecture

Dense retrieval, sparse retrieval, and a rank-based fusion — plus four mechanisms layered on top that exist because the reference evaluation found the core three weren't quite enough on their own.

Why simpler approaches aren’t enough

Role-based filtering alone cuts a 191-tool registry down to a persona’s slice (say, 87 tools for a student), but 87 is still too many for reliable single-query selection, and filtering by role does nothing to reflect that any one question only needs one or two of them.

TF-IDF keyword retrieval catches exact terminology but misses paraphrase — “raise a formal complaint” scores nothing against submit_grievance unless the synonym file already spells that mapping out.

Dense retrieval alone closes the paraphrase gap but can dilute precise-terminology matches (“CGPA” competing against several semantically-similar tools) and gives no way to force a plainly-named tool to the top when it’s the obvious match.

The curator combines dense and sparse retrieval specifically because their failure modes don’t overlap.

Dense retrieval — BGE + FAISS

BAAI/bge-small-en-v1.5 (130MB, 384 dimensions) is trained for asymmetric retrieval — short queries against longer descriptions, which is exactly the query-vs-tool-description shape. BGE requires a query-side prefix that documents don’t get:

BGE_QUERY_PREFIX = "Represent this sentence: "
q_emb = model.encode(BGE_QUERY_PREFIX + query, normalize_embeddings=True).astype(np.float32)

Embeddings are L2-normalized, so FAISS IndexFlatIP (exact inner-product search) is equivalent to cosine similarity. For a few hundred tools at 384 dimensions the whole index is under a megabyte and search completes in under a millisecond on CPU — this is what closes the gap between “what is my available balance?” and a tool named get_account_summary that shares no words with the query.

Sparse retrieval — BM25

BM25Okapi from rank_bm25, tokenized on whitespace and underscores:

def _tokenize(text: str) -> list[str]:
    text = text.lower()
    text = re.sub(r"[^a-z0-9\s_]", " ", text)
    return [t for t in re.split(r"[\s_]+", text) if len(t) > 1]

get_account_summary tokenizes to ['get', 'account', 'summary'], so the word “summary” in a query scores against it directly — this is also why tool documents are built with underscores expanded to spaces before indexing (see What gets indexed below). BM25 also applies term-frequency saturation: mentioning “transfer” ten times gives roughly a 3× boost, not a 10× one, unlike TF-IDF’s linear scaling.

Zero-score guard. When a query shares no vocabulary with any tool, every BM25 score is exactly zero, and sorting zero-score ties by insertion order would inject arbitrary noise into the fusion step below. The curator detects this and skips BM25 entirely for that query — dense retrieval carries the result alone:

max_bm25 = max((s for s, _ in pool_raw), default=0.0)
if max_bm25 > 1e-9:
    sparse_ranked = [...]   # normal path
# else: every BM25 score is zero — skip sparse retrieval for this query.

This was found during evaluation, not designed upfront — worth testing specifically if you’re combining dense and sparse retrieval elsewhere.

Fusion — Reciprocal Rank Fusion

Dense and sparse scores live on incompatible scales, so fusion uses rank position only, not the raw scores:

RRF_K = 60   # standard constant from the original RRF paper
def _rrf_merge(dense, sparse, k=RRF_K):
    scores = {}
    for rank, (name, _) in enumerate(dense, 1):
        scores[name] = scores.get(name, 0.0) + 1.0 / (k + rank)
    for rank, (name, _) in enumerate(sparse, 1):
        scores[name] = scores.get(name, 0.0) + 1.0 / (k + rank)
    return scores

A tool ranked #1 in both lists scores 1/61 + 1/61 ≈ 0.033; a tool only in dense at rank 25 scores 1/85 ≈ 0.012 — tools both retrievers agree on float to the top, and k=60 flattens the curve enough that one strong single-retriever signal can still outweigh a weak dual-retriever one. No score normalization is required anywhere in this step.

Name-token anchor

RRF handles the general case, but nothing guarantees a tool whose name directly matches the query survives to the top-k on rank alone. The anchor runs after fusion:

ALWAYS_INC_RATIO = 0.60
def _always_include(query: str, tool_name: str) -> bool:
    tokens = [t for t in tool_name.lower().split("_") if len(t) > 3]
    if not tokens:
        return False
    hit_ratio = sum(1 for t in tokens if t in query.lower()) / len(tokens)
    return hit_ratio >= ALWAYS_INC_RATIO

A query containing “hostel leave” against apply_for_hostel_leave hits 2 of 3 meaningful tokens (67%) and clears the 60% bar, so its RRF score is forced to 999 — guaranteed inclusion regardless of where fusion ranked it. The 60% threshold exists specifically to stop long tool names from being force-included by one incidental word.

Compound query splitting

A query like “show my attendance and also raise a grievance” encodes as a single blended vector sitting between two tool clusters, under-ranking both. A deliberately narrow set of split markers triggers decomposition:

_SPLIT_RE = re.compile(
    r"\b(?:and\s+then|and\s+also|after\s+that|additionally|"
    r",\s*then|,\s*and\s+also|then\s+also|as\s+well\s+as|"
    r"along\s+with|then|also|plus)\b",
    re.IGNORECASE,
)

Plain “and” and “or” are intentionally excluded — they’re too common in ordinary single-intent queries (“show my accounts and balances”) to safely treat as an intent boundary. When a split marker is detected: each sub-query retrieves independently with a budget of max(top_k // parts + 2, 7), results are unioned, and the full unsplit query runs once more as a top-up pass to catch cross-intent tools that only make sense given the whole query.

Low-confidence expansion

After fusion, if the strongest dense similarity is still weak, the match itself is probably weak, not just the ranking:

LOW_CONF_SIM = 0.25
CONF_BOOST   = 5
if max_sim < LOW_CONF_SIM:
    effective_k = min(top_k + CONF_BOOST, len(tools))

At the default top_k=20, a low-confidence query gets 25 tools back instead of 20 — a small hedge against a genuinely ambiguous or oddly-phrased query, at the cost of a slightly larger declaration set for that one turn.

Disk cache

The FAISS index and embeddings are persisted to index_dir, keyed by an MD5 hash of every (tool_name, description) pair plus the domain-synonym content:

def _pool_hash(tools):
    blob = json.dumps(sorted(tools), sort_keys=True).encode()
    return hashlib.md5(blob).hexdigest()[:16]

First build for a few hundred tools: a couple of seconds (embedding + FAISS + BM25 construction). Every subsequent process start with the same tool set and synonym file: well under 100ms, loaded straight from disk. Editing a tool description or domain_synonyms.json changes the hash, which invalidates the cache automatically on the next build_index() call — no manual cache-clearing step exists or is needed.

What gets indexed

Each tool is indexed over more than its bare description:

def _build_text(name, desc, extra=""):
    human = name.replace("_", " ")   # underscores -> spaces, so BM25 matches word-by-word
    parts = [human, desc.strip()]
    if extra:
        parts.append(extra.strip())
    return " ".join(p for p in parts if p)

extra is exactly the per-tool string supplied via extra_context in build_index() — see Configuration & Domain Synonyms for how that string is authored and why it’s the single highest-leverage lever for retrieval accuracy on a new domain.