ChatRAG · Ciel: An Enterprise RAG Assistant
"We're building a RAG system. Kind of like an internal chatbot with some role-based permission thing." That's roughly how my director and coworkers explained the project to me on my first day.
Context & Goal
SADEC Technology JSC's problem was simple to state and expensive to live with: employees were spending real work hours digging through shared drives and asking coworkers the same recurring questions. Where's the policy, what does this report say, which version of this file is current. The company's goal was an internal assistant, later nicknamed "Ciel," that could sit on top of its documents and answer in plain language, in Vietnamese, English, Japanese, or Chinese, without ever making an answer up.
The non-negotiable part of the brief: it had to work for non-technical staff, respect department-scoped permissions, and be trustworthy enough that people would actually rely on it instead of double-checking the source file every time.
Figure 1: The full Ciel interface, sidebar navigation, conversation, and a cited answer together.
Figure 2: A closer look at a cited answer, with clickable [N] references pointing back to the exact source document.
The Ideation Phase
Before any backend code, we weighed three ways to get an LLM to "know" the company's documents: fine-tune a model on internal data (expensive, and stale the moment a policy changes), give an LLM raw file access with no structure (fast to prototype, no way to guarantee it isn't making things up), or retrieval-augmented generation. RAG means indexing the documents, retrieving the relevant pieces at query time, and forcing the model to answer only from what it retrieved. RAG won because company documents change monthly, not yearly; an approach that needed retraining every time HR updated a policy was a non-starter.
Early on it was also clear that a single vector search wouldn't be enough. Early tests kept missing exact filename and table lookups that plain embedding similarity just isn't built for. That's what pushed the design toward a hybrid retriever (BM25 + vector + keyword fallback) from the start, instead of bolting it on later.
Building It & What Went Wrong
A few things broke before they worked:
- Multi-turn conversations kept breaking retrieval. A vague follow-up question, taken on its own, doesn't carry enough meaning to search well. Fixing this meant adding a query-rewriting step that folds the conversation history into the question before it ever gets embedded.
- Tables didn't survive being flattened into plain text. Spreadsheets and tables needed their own chunking and prompting strategy. Treating them like prose lost the exact thing that made them useful.
- Scanned paperwork was its own category of pain. A meaningful share of "documents" in a real company are photos and scans, not clean PDFs, which is why OCR (PaddleOCR) ended up load-bearing rather than a nice-to-have.
- A confident wrong answer is worse than no answer. In an internal-tools setting, that meant every response had to be context-strict with inline citations, so the model can only speak from what it actually retrieved. Reranking (BGE cross-encoder) made sure the passages that make it into the prompt are the actually relevant ones, not just the nearest by cosine distance.
Figure 3: High-level architecture, covering the ingestion pipeline, hybrid retrieval, and the streaming answer path.
The Person Who Guided Me
Phi, a senior engineer at SADEC, was the person I turned to whenever an idea looked good on paper but fell apart against real documents. He reviewed the retrieval design, pushed back on shortcuts that would've looked fine in a demo and broken in production, and was a big part of why this ended up as a system I'd trust with real company data rather than just a working prototype.
Results
- Grounded answers: every response carries clickable
[N]citations back to the source document. A hallucinated claim has nowhere to point to, which makes wrong answers easy to catch. - Multi-turn that actually works: the query-rewriting step means a user can ask a vague follow-up and still get the right documents pulled in.
- Works on real company documents: mixed PDFs, Word, Excel, CSV, and scanned images, in four languages, without a separate pipeline per format.
Looking Back & Ahead
This internship pushed me past the "cool demo" version of RAG that most tutorials stop at. The interesting engineering wasn't the LLM call. It was everything around it: making retrieval actually find the right passage, making multi-turn conversation not fall apart, and making the system honest about the limits of what it knows. Building Ciel taught me that in an enterprise setting, trustworthiness is a feature you have to engineer for, not a side effect of a good model.
Directions I'd want to push this further:
- Feedback-driven reranking: use thumbs-up/down on answers to fine-tune the reranker on the company's own documents instead of relying purely on a general-purpose model.
- Usage analytics: a lightweight admin dashboard showing which documents get queried most and where retrieval confidence is consistently low, mapping out where the knowledge base has gaps.
ChatRAG · Ciel
Source Code