Multilingual Normalizer

A provider-agnostic translation stage that keeps an English-trained retriever working on non-English queries — without ever touching the retriever itself.

Why this exists

A modern LLM is inherently multilingual and handles non-English queries natively — a query in Tamil reaches the model directly and gets a correct Tamil response. tool_curator’s dense retrieval breaks this: BAAI/bge-small-en-v1.5 is trained exclusively on English text, so when a non-English query is embedded and compared against English tool descriptions, cosine similarity scores land near zero across the board. The curator would then select tools based on noise, not relevance.

multilingual_normalizer exists solely to close that gap: translate any non-English query to English before it reaches the curator, so retrieval keeps operating on its training distribution. It has no other job — it doesn’t touch tool execution, response generation, or anything downstream of the query string it returns.

The fast-path

Calling a translation API on every query has real latency and real cost, so every query first passes through a local, zero-network heuristic that decides whether it’s already English:

from multilingual_normalizer.fastpath import is_likely_english

is_likely_english("show my account balance")   # True  — fast path, no API call
is_likely_english("naan en kanakku parkanum")   # False — routes to translation

Why not a bare ASCII check. Romanized non-English text is very often fully ASCII — “eard hisabati” is plain ASCII but is not English, and a pure-ASCII check would wrongly fast-path it straight to the curator untranslated. Instead, the heuristic looks at the fraction of 4+ character words that are recognized common English words:

Non-ASCII text (any non-Latin script) is never fast-pathed. Pure-ASCII text with no 4+-character words at all (short codes, IDs, “ok”) is fast-pathed by default, since there’s nothing meaningful to translate. Everything else needs a majority of its 4+-character words in a generic common-English-word list to qualify. That list is deliberately generic — articles, common verbs, common nouns — not domain jargon. If your domain’s vocabulary would otherwise trigger unnecessary translation calls (an English query using words the generic list doesn’t know), extend it per-instance rather than editing the package:

normalizer = get_normalizer(provider="gemini", api_key=..., extra_vocab=frozenset({"cgpa", "hostel", "grievance"}))

BaseNormalizer

class BaseNormalizer(ABC):
    def normalize(self, query: str) -> tuple[str, str]:
        if is_likely_english(query, self._extra_vocab):
            return query, "English"
        if query in self._cache:
            return self._cache[query]
        result = self._translate(query)
        self._cache[query] = result
        return result

    @abstractmethod
    def _translate(self, query: str) -> tuple[str, str]: ...

normalize() is implemented once, in the base class, because the fast-path check and the in-memory session cache are identical regardless of translation backend. Implementing a new provider means writing _translate() only — everything else is already handled. _translate() should fail soft (return (query, "English") on error) rather than raise, so a translation-API outage degrades to “treat as English” instead of crashing the caller’s retrieval step.

GeminiNormalizer — the default backend

from multilingual_normalizer import get_normalizer

normalizer = get_normalizer(provider="gemini", api_key="...", model="gemini-2.5-flash")
english_query, language = normalizer.normalize(query)

Internally this is a single Gemini call with thinking_config=types.ThinkingConfig(thinking_budget=0) — translation is a direct mapping task with no reasoning required, and leaving thinking enabled burns 1–3K thinking tokens per call for nothing. The response is constrained to a small JSON shape ({"english": "...", "language": "..."}), and max_output_tokens=150 keeps a translation call cheap even before the fast-path skips most queries entirely.

Adding your own provider

from multilingual_normalizer.normalizer_interface import BaseNormalizer

class MyNormalizer(BaseNormalizer):
    def _translate(self, query: str) -> tuple[str, str]:
        # call whatever translation backend you want — another LLM, a
        # dedicated translation API, a local model — and return
        # (english_text, detected_language_name).
        ...

Register it in your own factory, or just instantiate it directly — get_normalizer()’s provider= dispatch is a convenience, not a requirement; any BaseNormalizer subclass works anywhere a normalizer is expected.