Verified Semantic Cache for Hinglish

BCA (Honours, AI/ML) dissertation · Mohammad Jamal Yusuf, Myshaa Khan, Dhiren · SICSR, Pune · Mentor: Prof. Arun Kumar

A semantic cache reuses an earlier AI answer when a new question means the same thing, saving an LLM call (about 2.3 s and its cost). Plain caches fail on Hinglish and serve wrong answers to near-misses — questions that look similar but need a different answer. This page shows our system's real decision on every question in the held-out test set.

91.6%of Hinglish answers served from the cache are correct (plain cache: 64.8%)

61% → 85%Hinglish questions matched to the right cached question (fine-tuned BGE-M3)

91.6% vs 85.7%Hinglish cache precision vs a vCache-style baseline (ICLR 2026), while serving more from cache (40% vs 37%)

13,220test queries across 2 benchmarks, incl. 10,896 human-written Hinglish

How it works

Question (English / Hinglish / Hindi)→ Fine-tuned BGE-M3 finds the closest cached question→ Fine-tuned MuRIL + reranker verify the match→ Per-language cut-offs (tuned for ≥90% precision)→ Reuse cached answer or call the LLM

Models: fine-tuned BGE-M3 embedder · fine-tuned MuRIL verifier

Test-set explorer

742 customer-support questions from intents the models never saw in training. Click a question to see why the system decided what it did.

Results

Customer-support benchmark (test)	Served from cache	Cached answers correct	Users given a correct answer
Plain semantic cache, Hinglish (per-language threshold)	46.0%	64.8%	83.8%
Our system, Hinglish	40.4%	91.6%	96.6%
Our system, English	49.1%	91.5%	95.8%
Our system, Hindi (Devanagari)	76.4%	80.2%	84.9%

Right cached question ranked first	BGE-M3	Ours (fine-tuned)
Customer support — Hinglish	61.3%	84.9%
Customer support — English	87.7%	89.6%
Customer support — Hindi	80.7%	77.4%
Hinglish-TOP (voice assistant, unseen intents)	52.7%	65.0%

Verifiers (ROC-AUC, Hinglish): fine-tuned MuRIL 0.81–0.82 at 3 ms; BGE reranker 0.75 at 10 ms; LLM judge 0.47 at 830 ms.

Honest limits. The main benchmark is synthetic and small (about 265 test questions per language, so about ±6 points of uncertainty on Hinglish precision). Hindi (Devanagari) stays below 90%. On Hinglish-TOP no policy reaches 90% precision yet (best: 74% correct at 38% served). Negative results we report: storing Hinglish copies of cached questions hurt retrieval (61% → 50%); ranking candidates by the verifier alone hurt; fine-tuning on one domain hurt the other until the training data was mixed.