01 The user, and the 11 PM problem
Meet Riya — 17, Class 12 in Jaipur, targeting JEE. She studies 5–6 hours a day, and Chemistry is her weakest subject. Not from lack of effort — from lack of on-demand help.
When she gets stuck at 11 PM, she has two bad options: Google (which returns NCERT-level or random coaching blogs, in the wrong style for her syllabus) or WhatsApp (a 12–24 hour wait for a batchmate). Both break her study flow at exactly the moment she needs to keep going. I watched this exact pattern for a decade of teaching — ChemIQ is built for that moment.
26 chapters can't fit in a context window, and a bare LLM hallucinates facts that aren't in the notes. Fine-tuning would need re-running every week as the notes grow. RAG retrieves from the latest knowledge base at query time and grounds every answer in the teacher's actual content — the only approach that fits the problem.
02 The P0s: trust is the feature
The launch-blocking user stories weren't about answering questions — plenty of tools do that. They were about being trustworthy enough to rely on:
- Step-by-step answers, any time — the core value: unblock self-study at 11 PM.
- A source citation on every answer — chapter + section, visible below the response, no exceptions. A missing citation is logged as a system failure.
- An honest "not in the repository" — if retrieval similarity is below 0.50, the system must refuse to answer from base LLM knowledge. This hard guardrail is what stops a plausible hallucination from being taken as fact by a student who can't check it.
03 How it's built — a dual pipeline
Two pipelines share one vector store: an offline ingestion pipeline (notes → chunks → embeddings) that runs on each chapter update, and a live query pipeline that must return a grounded answer in under three seconds.
Intent + topic
Query is classified to Chemistry-only and tagged to a chapter namespace before any retrieval runs.
Hybrid search
BM25 + vector (RRF fusion) over box-level chunks, so exact terms and concepts both match.
Quality gate
Cohere Rerank cuts top-10 to top-3 — the critical step that keeps near-miss chapters out.
Cited answer
LLM answers only from retrieved chunks, in a fixed format, with a source-citation card.
Trace + learn
Every query logged to Arize: chunks, faithfulness score, feedback — feeding the improvement loop.
The <3-second latency budget
The SLA is a hard product requirement — a student on mobile at 11 PM has zero tolerance for lag. Every component has an explicit time budget, and the P95 target lands at ~2.25s with 750ms of headroom. Two PM calls made that real: streaming is mandatory (1.8s of generation feels instant when the answer writes itself out), and a reranker-timeout fallback drops to raw top-3 if Cohere exceeds 300ms — because a slightly weaker answer in time beats a perfect one that breaks the SLA.
04 What breaks — a 9-failure analysis
The heart of the PRD isn't the happy path — it's cataloguing how RAG fails on real chemistry, how each failure is detected in Arize, and the PM decision it forces. A few that shaped the build:
| Failure | What breaks | PM decision it forced |
|---|---|---|
| Chunking | A solved problem spans 8 steps across 3 paragraphs; paragraph chunking splits the answer | Box-level chunking — each colour-coded box is one atomic chunk |
| Embedding | "Fe²⁺ is oxidised" and "iron loses electrons" score low despite identical meaning | Add BM25 hybrid; budget for a larger embedding model only if recall gains >15% |
| Wrong chapter | "EAN rule?" returns Redox chunks because they share the word "electrons" | Metadata topic-filtering — one query classifier that also fixes re-ranking |
| Prompt bloat | Oversized context makes the LLM lose focus; faithfulness drops | Faithfulness guardrail as a product gate — visible yellow warning below 0.75 |
| Model cost | GPT-4o costs 10× mini with no factual-quality gain | Two-model routing; never use a costlier model than the task needs |
Four more failures — OCR quality, re-ranking mismatch, data drift, and user-behaviour (single-word queries) — are handled the same way in the full PRD.
05 Proving it works
For a trust-critical tool, "how do we know the answer is right?" is the product. So evaluation is fixed, not vibes:
Every thumbs-down becomes a labelled signal, sorted into three buckets (retrieval wrong / explanation confusing / missing content) and reviewed every Monday. The month's top-20 failures graduate into the offline eval set — so each failure makes the system harder to break the same way twice. That closed loop is the real engine.
06 Why it defends and how it pays
The PRD doesn't stop at the build. The moat is four compounding layers — a proprietary data set of the teacher's own notes, switching cost from personalised study history, a data-and-social network effect through JEE peer groups, and ultimately a multi-teacher ecosystem with supply-side lock-in.
Monetisation is sequenced deliberately: Free (validate PMF) → Student freemium (₹99/mo, paywall triggered at the exact 11 PM moment of highest need) → Teacher SaaS & B2B coaching (a single institute deal worth 250+ individual subscriptions). The teacher — me — is both the first customer and the proof of concept.
07 What building this taught me
- Evaluation is the product, not a finishing step. RAG failure analysis now goes into the PRD before a line of code — because for a tool students trust, correctness is the feature.
- Domain depth is the moat, not the model. Anyone can call an LLM. Almost no one has ten years of structured, grounded chemistry notes and knows exactly where students get stuck.
- Latency and cost are product decisions, not engineering afterthoughts. The streaming call, the reranker fallback, the two-model routing — each was a PM tradeoff about what the student actually feels.
The artifacts
ChemIQ is an ongoing personal build; the live demo is kept private while the notes corpus is expanded chapter by chapter.