Fully private AI for a 50-person firm: prove the answers before you build the app

In an October 2026 Flareworks engagement, a 50-person home-repair and appliance group in Singapore set one condition before sharing any company documents: it needed to know where company material would be stored and processed. It also wanted a narrow test before any large commitment. (The same group appears in Every AI proposal has a second spec, which reviews the vendor proposal it had received.)

The problem behind the request is ordinary. A technician on a job with a question about a price, a procedure or a manual messages a manager. Counting the search through documents, the wait for someone free to reply and the back-and-forth, an answer takes about 20 minutes. The design bar was set at roughly ten times better on speed or accuracy, enough that asking the system beats asking the boss. So the first service level was set before any software was chosen: an answer in 10 to 30 seconds, with the source document named.

Two meanings of "private"

"Private AI" covers two different setups, and proposals often blur them:

• Private storage: documents sit on company-controlled systems, but each question and the passages found for it are sent to an outside AI provider to write the answer.
• Private processing: documents, the search index, the questions and the answers all stay on company-controlled hardware, with an open model running locally.

The firm's condition pointed to the second. That ruled out the usual shortcut of a hosted knowledge-base product connected to ChatGPT, Claude or Gemini, and it set the design problem: get production-grade answers from a model small enough to run on one office machine.

Why RAGFlow, compared with three alternatives

Four open-source projects can run fully on company hardware. They differ in what they are built around, and the comparison below is about product focus, not a performance ranking.

	RAGFlow	Dify	Onyx	AnythingLLM
Built around	Turning messy documents into searchable evidence	Building AI apps, workflows and agents	Search across many connected company apps	A simple, local chat over your documents
Document handling	"Deep document understanding": OCR, layout and table recognition built in	A knowledge pipeline you assemble from extraction and chunking steps	Indexes "50+ applications" through connectors	PDF, TXT, DOCX and more, with citations
Inspect and fix chunks before go-live	Yes, chunks are shown for "human intervention"	Chunk preview before processing	Not a focus	Not a focus
Licence	Apache 2.0	Dify Open Source License (Apache 2.0 with added conditions)	MIT (Community Edition)	MIT
Pick it when	Document accuracy is the hard part	The end goal is process automation	Knowledge already lives in many tools	You want the lightest setup

RAGFlow was chosen for two reasons. The hard part of this firm's problem is the documents: scanned forms, rate cards, manuals and spreadsheets, where a table read wrongly gives a technician the wrong price. And it is cheap to try and cheap to leave. There is no licence fee or upfront spend, it runs in Docker, the original files are never altered, and the AI models plug in through Ollama, so they can be swapped without touching the documents.

The part of the study that makes switching cheapest is not the platform at all. It is the set of 100 questions with known answers. The same questions can score Dify, or any other system, on the same documents tomorrow. They have already been used to score four other model families without changing the setup.

What fully private costs

Running everything in-house moves the cost from a monthly AI bill to hardware, and the hardware sets the speed.

Where it runs	Where questions are processed	What it leaves out
A Mac Mini or MacBook with 24–32 GB of memory, on-site	On the machine	Uptime depends on the office: if the machine is off, the bot is off
A cloud server with a hosted AI model	By the model provider	Not private processing
A dedicated cloud GPU server	On the rented server	Private, but the affordable options found were in Europe, not Singapore

Answer speed on a local model is set mostly by memory bandwidth. The tests ran on a 2021 MacBook Pro (M1 Max, 32 GB). By bandwidth, an entry-level Mac Mini would be about three times slower, and its 16 GB is not enough for the model and the search database together; a Mac Mini with the Pro chip should match the test machine. The fuller cloud-versus-laptop measurements are in Private AI, measured. On a GPU server the model choice would change: gemma4 12B, the most accurate model tested, was too slow on the laptop and is the first one to re-test there.

A feasibility study before the app

The study is phased so that answers are proven before anything is built around them:

1. Feasibility test on synthetic documents, local models (done). A fictional company whose documents mirror the industry's formats, 100 questions with known answers, scored on three model sizes. The next two sections describe it.
2. Field test on real documents. Technicians ask during real jobs and site surveys, and each answer is checked against the 20-minute baseline of asking a manager.
3. Version control and approvals. Deciding which document is current, and who signs off a change.
4. Integrations and interface. Telegram bots per department, then anything else the field test shows staff need.

The phasing assumes trust is lost faster than it's won. The same group had already dropped a team tool because technicians wouldn't log in, so a system that gets early answers wrong is unlikely to get a second try. Interfaces and integrations are the visible part of a project, so they tend to get built first. Here they come last, after the answers are shown to hold up.

The test setup

Everything ran on the one laptop, with nothing sent outside it:

Layer	Choice	Why
Platform	RAGFlow 1.0 (release candidate 2) in Docker: the app, Elasticsearch, MySQL, MinIO and two small queue services	One install covers parsing, search, chat and citations
Parsing	RAGFlow's built-in DeepDoc models (OCR, layout, tables), 512-token chunks	Runs locally, no per-page cost
Search	Hybrid: keyword 0.7, vector 0.3, top 6 passages above a similarity of 0.2	Model numbers, rates and error codes need exact keyword matches; meaning needs vectors
Embeddings	bge-m3 through Ollama	A local multilingual model, so documents are never sent out to be embedded
Answers	qwen3.5 (2B, 4B and 9B) through Ollama, temperature 0.1, reasoning mode off	Same family at three sizes, so speed and accuracy can be traded off on one scale
Rules	System prompt: answer only from the documents, quote exact figures, say "This is not stated in the company documents" when the answer is missing, and prefer the latest effective date when versions conflict	Turns the service levels into instructions the model follows
Citations	RAGFlow's default: each answer links to the passages it used	No extra setup needed

Two settings cost the most time, and both were on by default. Ollama's context window was 4,096 tokens, while a prompt with six passages ran to about 4,800, so Ollama quietly cut the passages off and the model answered without them; raising the window to 16,384 fixed it. And qwen3.5's reasoning mode doubled answer times and sometimes returned empty answers, so it was switched off.

Why the test contained what it did

The test company is fictional, but its 18 documents were built in the formats a trades and appliance business actually keeps: Excel rate cards and zone tables, typed PDF procedures with ruled and borderless tables, Word and text policies, and a scanned two-column permit form saved as an image-only PDF. Each format is there because it fails in a different way, and the questions were written to find those failures:

• 30 base questions, each tied to one document. 13 simple lookups, 12 that need a table or a calculation, 3 on the scanned permit, and 2 whose answers are in no document. Each one names its expected answer and source, so a miss points at a specific document.
• 56 paraphrases, two per answerable question. Technicians don't ask in the document's words. They ask the same thing in shorthand, in different terms, or half-formed. If an answer only comes back when the question matches the document's wording, it will fail in the field.
• 14 cross-document questions. Real answers often need two or three documents, such as a rate from one sheet and a surcharge from another. One of them contains a genuine contradiction: the forklift procedure says hot work must stay 5 m from a charging bay, the hot-work procedure says 15 m.
• A superseded document. A 2025 rate card was added beside the 2026 one, to see whether the system gives the current figure, both, or the old one.
• Unanswerable control questions. These test the generation step on its own. Retrieval always returns the closest passages, even when none of them holds the answer, so these questions show whether the model refuses or fills the gap.

Each question was scored twice, so a failure can be placed:

• Retrieval: is a passage containing the answer in the top six? This measures the index and the parsing, independent of the model.
• Answer: does the reply contain the expected facts? Automatic keyword checks miss correct answers that are paraphrased, so every miss was read by hand and the reviewed number is the one reported.

The models ran one at a time on the same 100 questions, with a rule to stop a run if five answers took over 40 seconds. A first attempt to run three models in parallel was thrown out, because contention, not the models, set the speed.

The service levels the study had to meet

Each requirement was written as a target that can be measured every month:

Requirement	Service level	What the test found
Faster than asking a manager	95% of answers within 30 seconds	9B model: 15.8 s average, 18.8 s at the 95th percentile
Answers staff can rely on	At least 90% correct on the firm's own acceptance questions	96 of 100 on the 9B, 93 on the 4B
No guessing (groundedness)	Zero invented answers on the unanswerable control questions, which test the answer-writing step	9B and 4B refused every one; the 2B answered one with invented details
Every answer checkable	Every factual answer cites the source document	RAGFlow's default: answers link to the passages used
Current information only	A retired document stops appearing in answers the same day	Conflicting versions: 2 of 5 correct, then 5 of 5 after the fix below

The accuracy target is 90%, not 99%, on purpose. Every answer names its source and every miss is reviewed monthly, so the misses stay visible. The target is set where it can be checked monthly, not where it sounds best.

Groundedness rests on two layers. Search returns up to six passages, but anything scoring below a similarity of 0.2 is dropped, so a question about something the documents never mention often reaches the model with little or nothing attached. The prompt then tells the model to reply "This is not stated in the company documents" rather than fill the gap. The planned next step is raising the threshold to about 0.3, so off-topic questions are refused before the model is called at all, then re-running the 100 questions to check that no answerable ones are lost.

Availability is deliberately not on the list. Power, the network and the machine being switched on belong to the firm, and the proposal says so rather than promising uptime nobody on Flareworks' side controls.

Too small made things up, too large was too slow

The first internal target was 5 seconds per answer. Only the smallest model, qwen3.5 at 2 billion parameters, met it: 4.3 seconds on average. It also invented a payroll calculation, answered a trap question about a towing contractor instead of refusing and misread a zone table, for about 78 of 100. The 4B model (9.6 s, 93 of 100) is the smallest that stayed grounded, and the 9B (15.8 s, 96 of 100) is the default.

The opposite end failed too. A larger 27-billion-parameter model, qwen3.8, hit the stop rule with a 95th-percentile time of 55.7 seconds, well past the 30-second target. Smaller models invent answers and larger ones are too slow for this hardware, which leaves the 4B and 9B as the range that meets both service levels.

Four other model families ran on the same setup and the same questions, to check that the result was not specific to qwen. None displaced it:

Model	Questions answered	Correct (reviewed)	Speed	Result
granite4.2 8B	100	about 83	20.7 s average, 28.2 s at the 95th percentile	Slower and less accurate than the qwen3.5 4B
gpt-oss 20B	90, then stopped	84 (93%)	22.9 s average, 39.9 s at the 95th percentile	Accurate, but too slow and too large for the target hardware
gemma4 12B	75, then stopped	73 (97%), missing only 2 scanned-permit questions	33.7 s average, 40.3 s at the 95th percentile	The most accurate model tested, but more than twice as slow as the 9B
lfm2.5 8B	100	about 73	8.4 s average, 16.0 s at the 95th percentile	Fast, but invented values, and its reasoning text could not be switched off

Each alternative was either slower than the 9B or less accurate than the 4B, and the two accurate ones were stopped by the 40-second rule. For now, qwen3.5 gives the best balance of speed and accuracy on this hardware.

The 5-second target was dropped. The 30-second target replaced it because it is still dozens of times faster than the 20-minute baseline, and the 9B and 4B models met it without inventing answers. Load matters too: with three people asking at once, the 9B slowed from 16 to 45 seconds, so a shared machine needs headroom built into the target.

Most misses came from the documents, not the model

Word files, spreadsheets, plain text and typed PDFs with ruled tables answered every question correctly. The misses came from two documents:

• A PDF table without borders. The parser guessed the column boundaries from whitespace and read a lorry rate from the after-hours column.
• A scanned two-column form. The characters came through almost perfectly, but the layout collapsed, so each value landed next to the wrong label. Every model named the wrong fire watcher, including gemma4 12B, whose only misses in 75 questions were two questions on this form.

A bigger model can't repair text that was read in the wrong order, so the fix belongs at parsing. RAGFlow sets a parser per dataset and lets each document override it, which allows routing by document type:

• Word, text, Excel and typed PDFs with ruled tables: the default DeepDoc parser, which is fast and answered every question on these formats.
• Scans and awkward PDFs: the vision-model parser already built into this release, which sends each page as an image to a local vision-capable model that reads a form the way a person does. It costs about 30 to 60 seconds per page, once at upload, and nothing at question time. It has not been tested here yet, so it is the next experiment, not a promise.
• A heavier fallback: Docling can run as a separate server with a stronger table model, at the cost of several more gigabytes on the machine.

Running two parsers over the same file and merging the results is not supported, and is not worth building: two chunkings to reconcile and double the upload time, for little gain over choosing the right parser per document. The cheapest fix still comes first: tables stay in Excel, scanned forms get a short typed note stored beside them, and RAGFlow's chunk view is used to check each awkward document before it goes live.

Two versions of the same rate card

With a 2025 and a 2026 rate card both live, the assistant gave the current figure on only 2 of 5 rate questions. It doesn't know which version is newer. Three fixes each reached 5 of 5: a prompt rule to use the latest effective date and name the document used, a metadata filter, and simply disabling the old card. That is why "a retired document stops appearing the same day" is a service level, and why keeping one current version per topic is part of the monthly upkeep.

What RAGFlow does not solve

Three gaps sit outside what the platform does, and the proposal states them up front:

• Which version is authoritative. RAGFlow can find both a manual that says a product has a 24-month warranty and a spreadsheet that says 12, and cite both. It cannot decide which applies to a given customer, product revision or purchase date. If phase 3 needs that, it is a structured record with a person approving changes, not a search problem.
• Read-only access. The free edition shares datasets with the whole workspace, with edit rights. The design response is one administrator, one dataset and one Telegram bot per department, and staff who never log in.
• Who can message the bot. A Telegram bot answers anyone who knows its name. Settings reduce the exposure; restricting it to named staff is a small custom build.

Room for agents later

Keeping integrations until last doesn't close them off. RAGFlow exposes an API and an MCP endpoint, the open standard many AI agents use to call tools. The MCP endpoint is read-only: it can search the knowledge base and list datasets and assistants, but it can't add or change documents. For an agent that is a useful limit, because the administrator stays the only route by which company knowledge changes.

An agent's custom tool can call that endpoint for read access. One use case discussed for a later phase is an agent that reads both the knowledge base and the CRM to suggest the next step for a client, with a person approving it before anything is sent. How many AI agents does a small business need? covers why that role should be defined before the agent is built.

What has been proven so far

On the fictional company, the 100-question set of base questions, paraphrases and cross-document questions scored:

Model	Correct (reviewed)	Average answer time	95th percentile	Unanswerable questions
qwen3.5 9B	96 of 100	15.8 s	18.8 s	All refused
qwen3.5 4B	93 of 100	9.6 s	11.6 s	All refused
qwen3.5 2B	about 78 of 100	4.3 s	5.3 s	Answered one with invented details

Conflicting documents were handled once the prompt rule or the removal of the old version was in place. All of it ran on one laptop with nothing sent outside it. That shows the approach can give grounded, cited answers within the service levels on a realistic mix of document formats, including questions worded the way staff ask them.

It does not prove the firm's own documents will behave the same, which is what the field test is for. The study's next step uses real documents and questions technicians actually ask, measured against the same targets, before any interface work begins.