Scholarship Assistant
A question answering system for India's scholarship landscape. It tells a student what they qualify for and by when, and it is built around the fact that ordinary RAG gets this domain wrong in a way that is easy to miss and expensive for the student.
The API sleeps on a free tier after 15 minutes, so the first request can take about 50 seconds. Live evaluation scores are published at /api/eval/latest, and query log trends at /api/health.
Eligibility is numbers. "Minimum 80% marks" and "minimum 75% marks" produce nearly identical embeddings, so semantic search returns the wrong threshold and states it as fact. A student then wastes weeks on an application they were never eligible for, or misses a deadline that was never theirs to meet.
Measured on the corpus, plain semantic retrieval put 54 schemes in front of the model, and 44 of them, 81%, were ones the student could not access. Twelve schemes they actually qualified for never made the candidate list at all. Filtering after retrieval cannot fix that, because the right scheme was never retrieved.
Filter in SQL, first
Eligibility rules are extracted into typed columns at ingestion. Six nullable columns hold marks, income, age, course level, category and state, where NULL means no constraint. Comparisons happen in SQL before any retrieval, which is a relational query Chroma's metadata filter cannot express.
Search only what survives
The surviving scheme ids are passed to ChromaDB as a metadata filter, so vector search with bge-small-en-v1.5 runs only inside schemes the student actually qualifies for. The model never sees the other 81%.
Ground it, or refuse
Two independent checks. A regex verifies that every numeric claim in the answer appears in the retrieved text, then an LLM judge reviews the claims. If either fails, the system says what it does not know instead of guessing.
| Measure | Result | Method |
|---|---|---|
| Overall correctness | 0.80 → 0.88 | 40 hand-written questions |
| Threshold questions | 0.83 → 0.92 | 12 profiles sitting at cutoffs |
| Answers with invented figures | 0.075 → 0.00 | Regex, then LLM judge |
| Correct refusals | 0.63 → 0.88 | Unanswerable trap questions |
- Python 3.12
- FastAPI
- PostgreSQL
- ChromaDB
- bge-small-en-v1.5
- ONNX
- Groq
- Gemini
- RAGAS
- React 19
- Vite
- Neon
- Vercel
- Render
React front end on Vercel, FastAPI on Render's 512 MB free tier, PostgreSQL on Neon. Postgres holds the durable copy of every embedding and Chroma is the index built from it, so a restart on ephemeral disk loses nothing. A cross-encoder reranker was built and then disabled in production: two evaluation runs scored 0.88 and 0.93, a five point swing that is two questions of run-to-run variance on a 40 question set, so the improvement could not be told apart from noise at that sample size. Every query is logged with its retrieval scores, abstention reason and latency, and the trends are public at /api/health.
- Move to a multi-level criteria schema, since a single marks threshold per scheme cannot represent cutoffs that vary by course level.
- Build a held-out trap set. Abstention precision is currently 0.40, so the grounding rule blocks more answerable questions than it should.
- Redo the RAGAS run with a profile-aware judge. Standard faithfulness scores this system unfairly, because it legitimately grounds answers in the student profile and the SQL verdict, neither of which appear in the retrieved text.
- Re-extract a disability field, which two schemes require and the current schema has no column for.