ChatRAG — Ciel: An Enterprise RAG Assistant
Executive Summary: The Discovery Journey
"A chatbot that reads your documents is easy to demo. A chatbot that never lies about what it read — that's the actual job."
Building Ciel during my internship at SADEC Technology JSC went through 3 stages I didn't expect:
- The Assumption: I believed embedding search alone would find the right passages.
*Discovery: Pure vector search missed exact filename and table lookups — I had to blend it with BM25 keyword search as a hybrid retriever. - The Confusion: Follow-up questions in a conversation kept breaking retrieval — the second question alone didn't carry enough meaning to search well.
*The Pivot: I added a query-rewriting step that folds the conversation history into the question before it ever gets embedded. - The Realization: In an internal-tools setting, a confident wrong answer is worse than no answer.
*Result: I made every answer context-strict with inline citations, so the model can only speak from what it actually retrieved.
1. The Problem
Employees at the company were losing time digging through shared drives and asking coworkers the same recurring questions about internal documents — policies, reports, spreadsheets, scanned paperwork. I was asked to build a chatbot, internally nicknamed "Ciel," that could sit on top of those documents and answer in plain language, in Vietnamese, English, Japanese, or Chinese, without ever making an answer up.
The constraint that shaped everything else: this had to work for non-technical staff, respect department-scoped permissions, and be trustworthy enough that people would actually rely on it instead of double-checking the source file every time.
Figure 1: Ciel answering a question with clickable [N] citations pointing back to the exact source document.
2. Architecture & Tech Choices
The stack is a FastAPI backend streaming answers over SSE to a React + Vite + TypeScript frontend. A few decisions mattered more than the others:
- Supabase (Postgres + pgvector): I wanted structured metadata (departments, permissions, audit logs) and vector embeddings living in the same database instead of stitching together a separate vector store, to keep permission checks and retrieval consistent.
- Redis pub/sub: Document ingestion — parsing, OCR, chunking, embedding — takes real time. Redis lets the frontend show live progress instead of a spinner that lies.
- Local or cloud LLMs: Some documents are sensitive, so I made the model backend swappable — Ollama (default
gemma3:4b) running fully on-prem, or a cloud provider (Groq, OpenAI, Gemini, Anthropic, Cerebras) when quality matters more than data locality. - multilingual-e5-base embeddings: chosen specifically so retrieval quality doesn't fall apart across Vietnamese, English, Japanese, and Chinese queries against mixed-language documents.
Figure 2: High-level architecture — ingestion pipeline, hybrid retrieval, and the streaming answer path.
3. Engineering Deep-Dive
- Hybrid, filename-aware search: BM25 + vector similarity + a keyword fallback, so asking for "the Q3 budget file" works even when the phrase never appears verbatim inside the document.
- Table-aware retrieval: spreadsheets and tables get chunked and prompted differently from prose, since flattening a table into plain text destroys the thing that makes it useful.
- BGE cross-encoder reranking: the first-pass retrieval casts a wide net; a reranker then re-scores the candidates so the passages that actually go into the prompt are the most relevant ones, not just the nearest by cosine distance.
- OCR via PaddleOCR: a meaningful share of "documents" in a real company are scans and photos of paperwork, not clean PDFs.
- JWT auth with department-scoped permissions: retrieval never surfaces a document a user isn't cleared to see, and every query and admin action is audit-logged.
4. Key Results
- Grounded answers: every response carries clickable
[N]citations back to the source document — a hallucinated claim has nowhere to point to, which makes wrong answers easy to catch. - Multi-turn that actually works: the query-rewriting step means a user can ask a vague follow-up ("what about last month?") and still get the right documents pulled in.
- Works on real company documents: mixed PDFs, Word, Excel, CSV, and scanned images, in four languages, without a separate pipeline per format.
[AanSensei: nếu có số liệu thật — vd số tài liệu index được, thời gian phản hồi trung bình, % câu hỏi trả lời đúng qua đánh giá nội bộ — chèn vào đây sẽ mạnh hơn nhiều so với mô tả định tính.]
5. Conclusion
This internship pushed me past the "cool demo" version of RAG that most tutorials stop at. The interesting engineering wasn't the LLM call — it was everything around it: making retrieval actually find the right passage, making multi-turn conversation not fall apart, and making the system honest about the limits of what it knows. Building Ciel taught me that in an enterprise setting, trustworthiness is a feature you have to engineer for, not a side effect of a good model.
6. Future Directions
Directions I'd want to push this further:
- Feedback-driven reranking: use thumbs-up/down on answers to fine-tune the reranker on the company's own documents instead of relying purely on a general-purpose model.
- Usage analytics: a lightweight admin dashboard showing which documents get queried most and where retrieval confidence is consistently low — a map of where the knowledge base has gaps.
ChatRAG — Ciel
Source Code