cerebrollm

Ask a question of your own Markdown documents. You get an answer with a quote that the tool has re-read in the source file, or the words Not in the sources.

Status: version 0.2.0, early research. The aim of at least 90% right outcomes with under 2% confidently wrong runs is not yet met by local models; the measured results are below.

What it does

cerebrollm runs on your machine. It indexes a folder of Markdown files in a local PostgreSQL database. For a question it finds the best passages (keyword and vector search, then a reranker), asks a local model to quote the one line it relies on, and checks that quote against the source file and the sha256 recorded at indexing time. It prints the answer only when a quote is confirmed. It also serves the same search and check to a cloud model through a read-only MCP server.

It is for people who run local models, or who let a cloud model read their documents, and who need every answer traced to a file.

Animated walk-through in nine frames: cerebrollm on your computer between you, your notes and local models; its parts; then one question, link by link: you ask, the index returns 20 passages, the reranker keeps 8, a local model answers and quotes one line, cerebrollm re-reads that file, and you get the answer with its source or Not in the sources; last, the output gate.
Figure: one question through cerebrollm, built from the three figures on this page. It leaves out indexing and error exits.
Context diagram. Inside the boundary Your computer: cerebrollm, your Markdown notes and local models run by Ollama. You ask cerebrollm in the terminal. An optional AI app with a hosted model calls search and verify over MCP on a dashed two-way line; the passages go back to the app and its model's provider.
Figure: who and what cerebrollm talks to. Arrows point from the caller; what travels on a dashed line reaches a hosted model's provider. Solid lines stay on your computer only while Ollama uses its default loopback address; cerebrollm refuses any other address unless allow_remote = true is set under [ollama]. It leaves out the AI app's own traffic to its provider.
Parts diagram. The cerebrollm program (commands and a read-only MCP server) reads and re-checks the notes folders, stores and searches a PostgreSQL index of passages, vectors and file fingerprints, calls Ollama to embed and answer, and runs a local reranker that keeps the best 8 of 20 passages.
Figure: the parts of cerebrollm and what each link does, read against engine/cerebrollm/. It leaves out the configuration file and the secret scan that runs while indexing.
Flow chart of cerebrollm ask. A question goes through search for 20 passages, rerank to 8, generate with a local model and verify of each citation against the source file. A gate then asks whether any citation verified OK. Yes prints the answer and the verify codes. No prints exactly Not in the sources. Empty answers and searches with no passages also abstain. A failure in search, rerank or generation prints an error and exits 2.
Figure: the five steps of cerebrollm ask and its output gate, checked against engine/cerebrollm/answer.py. It leaves out indexing and the MCP server.

Measured results

One machine (Windows 11, WSL2 Ubuntu 24.04, RTX 4070 SUPER 12 GB), public documentation as the corpus. Source files are paths in the repository.

QuestionResultSource fileState
Does the top 8 hold a line that answers the question? (60 questions)Default search 45/60; search top 20 then rerank 46/60; any of the top 20 48/60eval/runs/retrieval/R-summary.mdmeasured
Does retrieval help a small local model (phi4)?Without retrieval 1/60; with it 47/60 (46 questions gained, 0 lost)eval/RESULTS.mdmeasured
Does retrieval help a strong hosted model (Claude Haiku)?47/60 without, 48/60 with: no effect showneval/RESULTS.mdmeasured
How often does a citation verify?Haiku 347 of 350 answers; phi4 184 of 350eval/RESULTS.mdmeasured
Which embedding model? (150 questions)qwen3-embedding:0.6b, hybrid hit@5 0.927 against 0.907 and 0.893research/model-study/RESULTS.mdmeasured
Which generator fits a 12 GB card?gpt-oss:20b at effort medium: code 39/40, tools 12/12, 46.2 tokens/sresearch/model-study/RESULTS.mdmeasured, one machine
Does ask reach 90% right outcomes with under 2% confidently wrong runs?Not met by any local model; met only by Claude Sonnet 5.5 with plain search (table below)eval/RESULTS-v2.mdmeasured
Qwen3.6 with multi-token prediction on and off, 8,000-token promptOn: code 37/40, tools 12/12, 52.9 tokens/s, 41.3% on the GPU. Off: code 36/40, tools 12/12, 51.3 tokens/s, 43.9% on the GPUresearch/model-study/r2-wsl/r2-table-M3.mdmeasured, one machine; context ladder not measured yet

v2 evaluation: 35 public items

20 answerable questions on public release notes dated after 2026-06 and 15 unanswerable questions. An item counts as right when more than half of its runs are right: a correct answer, or "not in the sources" for an unanswerable question. A run is confidently wrong when it answers an answerable item wrongly or answers an unanswerable one. Plain search is the top 8 of the default search with the citation instruction; ask is cerebrollm ask.

ModelPlain search: right outcome of 35, confidently wrong runsask (engine 0.1.0): right outcome of 35, confidently wrong runs
Claude Sonnet 5.5 (hosted)34 (97.1%), 0 of 105not run
gemma4:12b33 (94.3%), 4 of 175 (2.3%)22 (62.9%), 0 of 105
gpt-oss:20b32 (91.4%), 17 of 175 (9.7%)28 (80.0%), 7 of 104 (6.7%)
phi431 (88.6%), 10 of 175 (5.7%)21 (60.0%), 0 of 105

The aim is at least 90% right outcomes and under 2% confidently wrong runs. Only Claude Sonnet 5.5 with plain search meets it. ask removes most confidently wrong runs and gives up right outcomes in exchange. With 35 items, even zero confidently wrong items bounds the rate only below 8.6% (rule of three, 3 of 35), so this set cannot show a rate under 2%. Grader: gpt-oss:20b, checked against gemma4:12b on 454 answers (432 agree). Source: eval/RESULTS-v2.md. Fifty raw records that quote private documents are withheld; the scores are as committed.

The answer-quality grades come from a model grader that agreed with a person on 19/20 and 20/20 of 40 answers; its error on the other answers is not measured.

Install

Tested on Ubuntu 24.04, native or under WSL2. Step 1 needs Ollama running at 127.0.0.1:11434.

  1. Clone:
    git clone https://github.com/sourav-patra/cerebrollm && cd cerebrollm
  2. System packages, once, with sudo:
    sudo bash engine/setup/root-ubuntu.sh
    Check: /usr/lib/postgresql/16/bin/postgres --version prints 16.x.
  3. Virtual environment and the socket-only database:
    bash engine/setup/user.sh
    Check: the last line is USER SETUP DONE.
  4. Models:
    ollama pull qwen3-embedding:0.6b
    ollama pull gpt-oss:20b
    Check: ollama list shows both.
  5. Index your notes and ask. The README lists the configuration rule your folder needs and the one-time reranker download:
    engine/bin/cerebrollm index
    engine/bin/cerebrollm ask "a question your notes answer"

With pip install cerebrollm you get the same commands without the clone; steps 1, 3 and 4 still apply.

Limits