Expert routing lets dense multilingual search scale across languages while keeping token budgets modest. By directing each input through a small subset of specialized experts, a model can grow its total capacity without inflating the number of parameters touched per query. This mechanism preserves inference cost even as it adds language‑ and modality‑specific lanes.
Before these results, multilingual dense retrieval relied on monolithic encoders that either ballooned compute or fell back to sparse baselines. Off‑the‑shelf multilingual dense models routinely lost to a parameter‑free BM25 baseline on specialist Greek corpora, exposing the inefficiency of naïve scaling [1].
K‑EXAONE 2.0 packs 750 billion parameters while activating only ~37 billion per token. The upcycled Mixture‑of‑Experts architecture therefore triples the total parameter count of its predecessor yet keeps the per‑token footprint comparable to a much smaller model [2].
It achieves perfect retrieval scores across evaluated needle positions and context lengths up to 256 k tokens. Such long‑context competence demonstrates that dense search can span entire documents without exploding token budgets, a direct benefit of expert routing [2].
LoRA‑tuning a 30B‑A3B mixture‑of‑experts reader lifts judged answer correctness from 29.4 % to 66.9 %. The same study notes that a max_len of 512 truncates roughly 87 % of training pairs, highlighting how the MoE reader sidesteps severe context loss while delivering grounded generation [1].
These gains leave open questions about routing overhead and language coverage. K‑EXAONE 2.0 expands multilingual support to ten languages but still falls short of full‑global breadth, and its expert dispatch demands specialized serving stacks; the Greek adaptation focuses on specialist domains, so transferability to other low‑resource tongues remains unproven [2], [1].
Teams building product‑scale multilingual search should replace monolithic dense encoders with MoE backbones such as K‑EXAONE 2.0 and adopt LoRA‑tuned MoE readers for retrieval‑augmented generation, re‑running existing benchmarks (e.g., HERA) under the extended context windows these models make affordable.
