multilingual_normalizer

Everything importable from the multilingual_normalizer package.

get_normalizer()

get_normalizer(provider: str = "gemini", **kwargs) -> Optional[BaseNormalizer]

Factory. Returns a normalizer instance, or None when provider="none".

BaseNormalizer

class BaseNormalizer(ABC):
    def __init__(self, extra_vocab: frozenset[str] | None = None): ...

    def normalize(self, query: str) -> tuple[str, str]: ...

    @abstractmethod
    def _translate(self, query: str) -> tuple[str, str]: ...

normalize(query) is the only method callers use — it runs the English fast-path, checks the in-memory session cache, and only calls _translate() for queries that need it. Returns (english_query, detected_language); detected_language is "English" for anything that took the fast path.

extra_vocab extends the fast-path’s common-English-word list with domain-specific words that should count as “obviously English” for your use case, reducing unnecessary translation calls without editing the package itself.

Implementing a new provider means subclassing BaseNormalizer and writing _translate() only — see Multilingual Normalizer.

GeminiNormalizer

GeminiNormalizer(
    api_key: str,
    model: str = "gemini-2.5-flash",
    system_prompt: str | None = None,
    extra_vocab: frozenset[str] | None = None,
)

The default shipped BaseNormalizer implementation. system_prompt overrides the default translation instruction if you need to preserve specific terms verbatim (product names, reference-code formats, etc.) — the default prompt has no domain knowledge baked in. Requires the normalizer extra (google-genai); the import is local to this module, so importing multilingual_normalizer itself never requires it.

fastpath.is_likely_english()

from multilingual_normalizer.fastpath import is_likely_english, COMMON_ENGLISH_WORDS

is_likely_english(text: str, extra_vocab: frozenset[str] | None = None) -> bool

The English fast-path heuristic, usable standalone if you want the detection logic without the rest of the normalizer (for logging, metrics, or a custom pipeline). Non-ASCII text always returns False. Pure-ASCII text with no 4+-character words returns True. Otherwise, returns True only when a majority of 4+-character words are in COMMON_ENGLISH_WORDS | extra_vocab.