RAG Tool Curator — Technical Documentation

As agentic LLM systems grow to handle hundreds of tools, two problems show up together. The context window fills with tool definitions the model won't use for a given query, and tool-selection accuracy drops as the candidate pool grows. This document describes a tool curator: a retrieval layer sitting between role-based access control and the LLM that narrows a large tool pool down to a small, query-specific shortlist before the model ever sees it.

The curator is a small, swappable component. A two-method interface (build an index, select tools for a query) sits in front of a hybrid retrieval implementation combining BGE dense embeddings, FAISS, BM25, and Reciprocal Rank Fusion, plus several mechanisms aimed at specific failure modes.

It was built and tested against a university system — eight MCP servers, 191 tools, four user roles, and a 281-case test suite — across three generations. A no-curator baseline reached 100% (281/281) once tool descriptions were rewritten into a tighter contract format, up from roughly 80%. A TF-IDF curator reached 83.6% correct (88.4% accuracy score), with failures concentrated in paraphrased queries. The hybrid curator reached 93.2% correct (94.8% accuracy score), cutting the prompt token cost for an initial student query from roughly 11,860 to roughly 1,561 tokens, an 86.8% reduction.

Giving an LLM access to tools usually means listing them, either in the system prompt or as function declarations. For ten or twenty tools this is fine. Past roughly fifty, two problems start to compound.

The first is token cost. Each tool definition runs 100 to 200 tokens once serialised; a student account in the testbed has access to 87 tools, costing around 11,860 prompt tokens before the user types anything. Scale that to a few thousand tools, common on enterprise platforms, and the tool definitions alone can exceed the context window of most deployed models.

The second is that tool-selection accuracy doesn't stay flat as the pool grows. Picking the right tool from 87 candidates in one pass means overlapping vocabulary and similar names introduce noise, and that noise compounds when a query needs two or three tools in sequence.

A third issue sits upstream of both: the wording of the descriptions affects how well the model picks. Early versions of the testbed used documentation-style descriptions, with bulleted "use this tool when..." sections and repeated examples, and with no curator and full tool visibility scored around 80%. Rewriting all 191 descriptions into a shorter, contract-style format, stating what the tool returns and what its parameters are, took the same system to 100% with no other change. Every accuracy number in this document is measured against the refined descriptions, since a curator's job is to approach what a full, well-described tool list achieves, not compensate for badly written ones.

The question this project set out to answer: can a lightweight retrieval layer cut a large, role-filtered tool pool down to roughly fifteen tools per query, without losing the accuracy a full, well-described tool list achieves, in a way that isn't tied to one domain's vocabulary?

Role-based filtering alone

The first layer of curation is access control: a student's view of 191 tools is cut to whatever their role permits — 87 for students, around 65 for faculty, around 27 for executives, and the full 191 for admins. This is necessary but doesn't touch the per-query problem, since eighty-seven tools is still too many for reliable single-query selection and role filtering does nothing to reflect that any given question only needs one or two of them.

TF-IDF keyword retrieval

The first active curator used TF-IDF with cosine similarity: vectorise each tool's name and description at startup, vectorise the query the same way, keep the highest-scoring matches. For queries where wording lines up, this works well: "What is my CGPA?" shares a token with "Returns the student's cumulative grade point average," so get_cgpa scores highly. A large share of real queries fall into this category.

Where it breaks is anywhere the user's vocabulary and the tool's don't overlap, even when the meaning is identical to a person. "Am I at risk of being barred from exams?" should map to an eligibility check, but "barred" appears nowhere in that description, so the score is near zero. "Raise a formal complaint about the projector" should map to a grievance tool, but neither "complaint" nor "raise" appears in "submit a new grievance ticket." These aren't tuning problems; they're a consequence of scoring token overlap with no notion of two words meaning the same thing.

There was also a second-order issue with action tools, the ones that change state. An early TF-IDF version sent only its top-k matches, which badly hurt accuracy on action queries, since those tools didn't always score highest for how users phrased requests. The fix, always including every action tool the role can access, helped, but pushed the effective tool count for a student query from fifteen to around thirty-three.

Dense retrieval on its own

Dense embeddings close the vocabulary gap TF-IDF can't: a semantic model places "raise a formal complaint" and "submit a new grievance ticket" close together with no shared words. But it can be too loose for queries that should have an exact match. "Get my CGPA" should retrieve get_cgpa and arguably nothing else, but a dense model might also surface get_course_performance_summary or get_student_academic_summary, all sitting in the same semantic neighbourhood.

Neither approach alone is sufficient. Lexical matching is precise when wording happens to align and falls apart otherwise; semantic matching closes that gap but loses precision on exact matches. The curator below combines both.

The curator sits between role-based filtering and the LLM. By the time a query reaches it, the tool pool is already filtered to whatever the role permits. The curator's job is to take that pool and the query and return a much smaller shortlist, typically around fifteen tools.

The interface

Everything the rest of the system needs to know is defined in tool_curator/curator_interface.py, an abstract base with two methods:

class ToolCurator(ABC):
    @abstractmethod
    def build_index(self, tools, extra_context=None) -> None: ...

    @abstractmethod
    def select(self, query, tools) -> list[tuple[str, str]]: ...

build_index runs once at startup, after tool definitions are collected and role-filtered. The resulting FAISS and BM25 structures are cached to disk under .tool_index/ — subsequent startups load the cache in ~50 ms rather than the 2–3 s needed to re-encode all tools from scratch. The cache key is an MD5 hash of the tool definitions plus domain_synonyms.json, so editing either file automatically invalidates it. select runs once per query. Neither signature mentions BGE, FAISS, or BM25; tool_curator/__init__.py reads TOOL_CURATOR_MODE from settings and returns whichever implementation matches, so swapping retrieval approaches doesn't touch the orchestrator. The same file exposes load_tool_context(), which reads domain_synonyms.json and passes it to build_index as extra_context.

The production implementation, tool_curator/hybrid_rag_curator.py, is built from seven pieces, each added because something simpler was tried first and didn't hold up.

Seven components

BGE_QUERY_PREFIX = "Represent this sentence: "
q_emb = model.encode(BGE_QUERY_PREFIX + query, normalize_embeddings=True)
doc_emb = model.encode(tool_text, normalize_embeddings=True)

{
  "submit_grievance": "raise complaint file issue lodge protest formal complaint",
  "get_exam_eligibility": "can I write exam barred eligible allowed to sit detained"
}

RRF_K = 60
def _rrf_merge(dense_ranked, sparse_ranked, k=RRF_K):
    scores = {}
    for rank, (name, _) in enumerate(dense_ranked, 1):
        scores[name] = scores.get(name, 0.0) + 1.0 / (k + rank)
    for rank, (name, _) in enumerate(sparse_ranked, 1):
        scores[name] = scores.get(name, 0.0) + 1.0 / (k + rank)
    return scores

ALWAYS_INC_RATIO = 0.60
def _always_include(query, tool_name):
    tokens = [t for t in tool_name.lower().split("_") if len(t) > 3]
    if not tokens:
        return False
    hit_ratio = sum(1 for t in tokens if t in query.lower()) / len(tokens)
    return hit_ratio >= ALWAYS_INC_RATIO
# applied after RRF: matching tools get their score forced to 999

DEPENDENCY_TOOLS = {
    "get_exam_eligibility": ("get_attendance_summary", "get_fee_dues"),
    "register_for_placement_drive": ("check_placement_eligibility",),
    "make_payment": ("get_fee_dues",),
}

Supporting infrastructure

A few pieces around the curator affect reliability without being part of retrieval itself. orchestrator/hallucination_guard.py runs after the model picks a tool: it compares the name against known tools using difflib.SequenceMatcher, silently correcting above 0.82 similarity, returning a hint for the model to retry between 0.60 and 0.82, and failing outright below 0.60. This catches the common case of a model knowing which tool it wants but misspelling the name, like get_attendence_summary.

orchestrator/mutation_guard.py handles state-changing tools: before any such tool runs, it shows a confirmation panel and waits for approval, and injects identity fields (student_id, faculty_id) from the session so users never type their own ID. tool_curator/suggestion_validator.py scans the model's final text for snake_case-looking names that aren't real tools — a check for when the model invents a plausible tool name while suggesting a follow-up; it warns but doesn't block. tool_curator/terminal_display.py is the UI layer and carries no retrieval logic.

One bug worth flagging because it would silently break any system built this way: the Gemini SDK's per-turn send_message(config=...) call replaces the session-level config rather than merging with it, and the session config is where the system instruction — including the user's identity — lives. A per-turn config containing only the curated tools wipes that out, and the model starts asking for the student ID on every query. The fix passes the system instruction explicitly on every call and, as a second layer, prepends a short session marker (role and user ID) directly into the message text.

Finally, the curator's index is cached on disk: a couple of seconds to build from scratch (encoding 191 tools, building FAISS and BM25), about fifty milliseconds to load. The cache key is an MD5 hash of the tool definitions plus domain_synonyms.json, so editing synonyms automatically invalidates the cache.

The curator was built and tested against a fictional university system, an in-memory database (mock_db/university_mock_database.py) behind eight MCP servers exposing 191 tools, structured to resemble a real internal platform.

Four roles see different slices, defined in orchestrator/persona_access_policy.py: students see 87 tools, faculty around 65, executives around 27 (mostly dashboards), admins all 191. About 55 of the 191 tools change state and are flagged as action tools, intercepted for confirmation by mutation_guard.py.

Three generations

Three generations were built and run against the same 281-case suite: a no-curator version, where every tool the role permits goes to the model at session start, serving as the accuracy and token ceiling; a TF-IDF curator from Section 2, with the top-k-plus-action-tools augmentation layered on once top-k alone proved unreliable for action queries; and the hybrid curator described in Section 3 — BGE, FAISS, BM25, RRF, the anchor, compound splitting, and dependency expansion — which the rest of this document calls "the hybrid curator."

The description refinement step

Before any curated version could be measured meaningfully, the no-curator baseline needed measuring first, since it's the reference point. The first attempt scored around 80%, and the cause was how descriptions were written. The original style was documentation aimed at a person:

Retrieves all internal assessment marks
for a student in a specific course.

Use this tool when the user asks about:
- CIA marks for a course
- Internal assessment scores
- How much I scored in CIA-1 or CIA-2

Parameters:
- student_id: Required. e.g. "21CS045"
- course_code: Required. e.g. "CS3503"

Returns: CIA-1, CIA-2, assignment marks,
total internal marks, percentage.

Returns all internal assessment marks
for a student in a specific course.

Parameters:
- student_id: Required. e.g. "21CS045"
- course_code: Required. e.g. "CS3503"

Returns: CIA-1, CIA-2, assignment marks,
total internal marks, and percentage.

Applied across all 191 tool descriptions → no-curator baseline: ~80% → 281/281 (100%)

Most of the original description was a "use this tool when" section restating the description in different words. Every one of the 191 descriptions was rewritten to drop that section, keeping only the contract. With this applied across all 191 tools and nothing else changed, the no-curator baseline went from roughly 80% to 281/281. The extra text seems to compete for the model's attention with the part that actually distinguishes tools, and it also roughly halves every tool's token footprint on top of whatever the curator saves. Every accuracy number in this document is measured against these refined descriptions.

All three versions ran against the same 281-case suite (MCP_Evaluation_Sheet_-_TF_IDF_Tool_Curator.xlsx), each case specifying a query, persona, up to four expected tools, and an expected output summary, split across seven categories:

Each case was run by hand and labelled Correct, Partially correct (some expected tools called), Wrong (the wrong tool called, with irrelevant output), or Skipped (the system said no tool was available when one exists), with Accuracy Score = (Correct + 0.5 × Partially Correct) / Total.

Reading the table: the "No curator, refined descriptions" row at 100% is an accuracy ceiling, not a deployment target — sending all 87 role-permitted tools costs ~11,860 prompt tokens before a word of user input. The curator's goal is to approach that ceiling while cutting token cost by ~87%.

Moving from TF-IDF to the hybrid curator: correct rose 235→262, wrong fell 9→3, skipped fell 10→7, partial fell 27→9, and the accuracy score moved from 88.4% to 94.8%. The remaining seven-point gap to the 100% baseline is the subject of the failure analysis below.

The hybrid curator's token count is roughly 86.8% below the no-curator baseline. The gap between TF-IDF's ~33 tools and the hybrid's ~15 is mostly TF-IDF's blanket action-tool addition versus the hybrid's targeted dependency expansion: one adds all action tools to every query, the other adds only what's actually depended on.

Nineteen of 281 cases didn't come back fully correct: 3 wrong, 7 skipped, 9 partially correct. The patterns matter more than the individual cases.

The single largest move in this project wasn't a retrieval technique: it was rewriting tool descriptions.

Two of the seven components came from testing rather than design, and both are worth remembering. The BM25 zero-score skip stops a tied, effectively random sparse ranking from injecting noise into queries with no keyword overlap — anyone combining dense and sparse retrieval should specifically test that case. Dependency expansion replaced TF-IDF's blanket action-tool inclusion, which worked but nearly doubled the effective tool count for every query regardless of need; the targeted version adds only what a selected tool's workflow depends on, both cheaper and, given the wrong-case count dropping from 9 to 3, more accurate.

Skipped cases come from two genuinely different problems needing different fixes. Vocabulary gaps, where query and description share no meaning even after embedding, get fixed in domain_synonyms.json. Pool crowding, where a direct match gets pushed out by several near-matches in a large pool, is a top_k or anchor-threshold question specific to roles with bigger pools. Reaching for the synonym file every time fixes one and does nothing for the other.

Compound splitting clearly helps but isn't complete: plain "and" doesn't trigger it by design, and three-or-four-intent "give me everything" queries can still run into a tight per-sub-query budget, where the remaining partially-correct cases concentrate. Separately, when two tools differ only in intent rather than subject matter — like the mark_attendance / update_attendance pair from the failure analysis — that distinction needs to be explicit in both descriptions, since the model's choice between them depends entirely on it.

Finally, separate curator failures from model failures when reading results. A "wrong" label doesn't always mean the curator missed the right tool — sometimes it was available and the model picked differently anyway. Lumping both together as "curator accuracy" risks tuning retrieval to fix a problem that actually lives in the descriptions or the model's reasoning.

The retrieval algorithm doesn't know anything about universities. It operates on (name, description) pairs and a query string; everything domain-specific lives outside it, in files written for this testbed that need rewriting for anything else.

The curator's only requirement is (name, description) pairs, which can come from MCP list_tools(), OpenAI tool specs, LangChain registries, or a database of tool metadata, as long as something converts them to that shape.

A healthcare deployment follows the same shape: patient_id/doctor_id as identity fields, prescribe_medication depending on patient history and allergies.

For TOOL_CURATOR_TOP_K: roughly six to eight for a small, easily-distinguished tool set, ten to fifteen for many similar tools (this project's situation), and ten plus dependency expansion for high-risk workflows. BAAI/bge-small-en-v1.5 was the default used throughout; bge-base-en-v1.5 trades speed for better accuracy on paraphrased queries, while all-MiniLM-L6-v2 was noticeably weaker in informal comparisons and wasn't used for any reported result.

If only the retrieval algorithm is needed:

from tool_curator import get_curator, load_tool_context

curator = get_curator(mode="rag", top_k=10,
                       model_name="BAAI/bge-small-en-v1.5", index_dir=".tool_index")

tools = [("get_account_balance", "Returns account balance for a customer account.")]
curator.build_index(tools, extra_context=load_tool_context())
selected_tools = curator.select("Can I send money to Rahul?", tools)

selected_tools is what gets passed to the LLM; role filtering, confirmation, and identity injection remain the calling application's job.

To be precise about the data contract: build_index receives a list of (name, description) tuples plus an optional extra_context dict mapping tool names to synonym strings. It has no return value — it mutates internal index state and writes the cache to disk. select receives the same query string and the same tool list (the role-filtered subset of the full registry) and returns a list of (name, description) tuples in ranked order, typically 10–18 items depending on anchor and dependency additions. The caller is responsible for serialising those tuples into whatever format the target LLM expects (function declarations, tool specs, etc.).

The results in Section 5 came from four things working together: tool names that map onto how users phrase requests, descriptions written as contracts, a synonym file bridging this domain's vocabulary gaps, and dependency rules reflecting how this domain's workflows chain together. FAISS, BM25, and RRF don't care what they're indexing, but reusing only the algorithm without rewriting the other three is likely to underperform what's reported here.

The curator at the centre of this project is a small, swappable component — a two-method interface in front of a hybrid BGE/FAISS/BM25/RRF implementation, tested across three generations of the same 191-tool university system: a no-curator baseline reaching 100% only after description refinement (from roughly 80%), TF-IDF at 83.6% correct (88.4% accuracy score), and the hybrid curator at 93.2% correct (94.8% accuracy score) with an 86.8% token reduction.

The remaining nineteen cases split into patterns, not noise: near-identical tools the model confuses even when both are retrieved correctly, direct queries crowded out in large pools, genuine vocabulary gaps the synonym file hasn't covered, and multi-intent queries asking for more than the current splitting and budgeting handle reliably. Each points at a specific, different fix.

The reusability guide lays out the cost of pointing this curator at a different domain: a few files carry over unchanged, a handful need small edits, and three — the synonym file, the dependency rules, and the role and identity configuration — need rebuilding. The algorithm doesn't know or care what it's retrieving. The numbers above came from that algorithm combined with domain work specific to a university, on top of descriptions deliberately rewritten to be easy to tell apart. Reuse the algorithm freely; just budget for that other work wherever it lands next.