Limitations & Future Work

The curator has been evaluated end-to-end against one deployment — the reference university system. The reusability guide describes how to port it elsewhere, but that hasn't been run against a second real deployment, so generalization currently rests on design reasoning, not a second measured result. The seven internal retrieval mechanisms also haven't been ablated against each other — each component's individual contribution to the 94.8% accuracy score is unknown. Fixing this means running the same eval methodology against a second domain (banking, healthcare) and an ablation pass (removing one mechanism at a time, re-running the suite).

All 281 reference-deployment evaluations were labeled by hand — consistent in that the same rubric was applied throughout, but slow, and with some judgment involved in partial-credit cases. Replacing this with an automated harness (deterministic checks for expected tool names, plus LLM-as-judge for free-text output) is the highest-value next step, both for iteration speed and for testing synonym or dependency changes without a full manual pass. The bundled curator_eval.py automates the retrieval-scoring half of this already — the free-text output judging half remains manual.

The mark_attendance/update_attendance confusion explanation in the failure analysis is inferred from tool similarity, not confirmed against real call logs — a reading, not a measurement. The entire reference system runs against an in-memory mock database; none of the engineering here — the curator interface, the eval harness, the disk cache, the guards — has run against real users or production load. "Production-shaped" is accurate; "production-deployed" would not be.

The curator has no memory across conversation turns. If a model proposes a follow-up tool and the user replies "yes," the curator retrieves on that short reply alone, with no memory of what was just suggested — conversation history sometimes lets the model recover on its own, but this isn't guaranteed. A fix would store the last-suggested tool(s) in session state and force-include them when the next input looks like a short confirmation.

Every pool-crowding skip in the failure analysis came from the largest-pool roles (faculty/admin) — suggesting top_k and the anchor threshold could be set per-role rather than globally, which the current configuration surface doesn't directly support (it's one global parameter per curator instance, not per role). For extreme-paraphrase queries, a lightweight query-rewriting/normalization step ahead of embedding might help more than trying to hand-anticipate every possible phrasing in the synonym file.