PolicyQA
A question-answering service over nine Indian labour-law documents, built so that every answer can be traced back to the clause it came from, and so the pipeline cannot ship if its numbers slip.
Labour-law questions are keyword-sensitive. A section number matters as much as the semantics around it. Pure vector search misses exact references; pure keyword search misses paraphrase.
So the answer had to be hybrid, and it had to be verifiable: a legal answer without a citation is worse than no answer at all.
Hybrid retrieval
BM25 keyword search runs alongside vector semantic search over ChromaDB, and the two candidate lists are merged by Reciprocal Rank Fusion.
Cross-encoder rerank
ms-marco-MiniLM rescores the fused candidates pairwise against the query, cutting to the top five passages actually worth sending to the model.
Grounded generation
Google Gemini answers from those five passages only, with citations back to source clauses; Langfuse traces every call for inspection.
| Measure | Result | Method |
|---|---|---|
| Retrieval hit rate | 0.955 | 22-question set |
| Answer faithfulness | 0.941 | LLM-as-judge |
| Citation accuracy | 0.964 | LLM-as-judge |
| Deploy gate | Pass/fail | CI quality gate |
- Python
- LangChain
- Google Gemini
- ChromaDB
- BM25
- Cross-encoder
- FastAPI
- Streamlit
- Docker
- Cloud Run
- Langfuse
Containerised with Docker and deployed to Google Cloud Run: a FastAPI backend, a Streamlit front end, and automated evaluation as a quality gate that blocks the deploy on metric degradation.
- Grow the eval set past 22 questions and stratify it by document, so per-source regressions surface individually.
- Swap in a fine-tuned reranker and A/B it against ms-marco on the same harness.
- Add answer-level abstention when top-passage scores fall below threshold, rather than answering weakly.