Bilingual Scheme Retrieval
Hindi↔English CRAG over 3,300+ government schemes, measured not vibed
- 3,300+
- schemes indexed
- 30K+
- retrieval chunks
- 85%+
- eval accuracy
The problem
Welfare schemes are discoverable only if you already know what they are called. The people who most need them search in Hindi, in natural language, and ask three distinct kinds of question: am I eligible, what do I get, and how do I apply.
Approach
Chunk for the question, not the document
Metadata-enriched hybrid chunking produced 30K+ chunks tagged by scheme and section. Top-2 retrieval plus sibling expansion pulls the neighbouring context only when it is needed, which cut token overhead against sending whole scheme documents.
Route by intent
Eligibility, benefits and how-to-apply questions want different sections of a scheme. Detecting intent before retrieval meant the retriever searched a smaller, more relevant slice.
Measure it
An LLM-as-judge evaluation over a generated question set scored 85%+ answer accuracy. Without that harness, every prompt change is a coin flip you cannot see the result of.
Outcome
- 3,300+ schemes queryable in Hindi or English through one interface.
- 85%+ accuracy on a repeatable evaluation set, so prompt and retrieval changes are comparable.
What I'd do differently
The translation layer is the weakest link — scheme names are proper nouns and round-tripping them through translation loses them. Next version pins entities before translating.