A private search engine for AI agents over 13,000 Obsidian notes
Obsidian, a 66 MB local embedding model, sparse search, RRF and a shareable agent skill, combined into private retrieval without sending the knowledge base to an external embedding API.
The knowledge base is an Obsidian vault of more than 13,000 Markdown notes: product decisions, customer calls, workshop transcripts, old application essays, technical incidents, journals, and years of half-finished thinking. It is a personal knowledge base that I also use for my business, not a client deployment. The same idea, a knowledge base its owners keep, was taught to two insurance advisors in a second-brain workshop.
The problem was no longer capturing information.
It was recovering the right evidence when it was needed.
I mostly work through Claude and Codex rather than clicking around Obsidian. Both agents could run a keyword search. But keywords are literal. A question about a “career highlight” may be answered in notes that say selected, led, shipped, won, or made a full-time-level contribution—without using the phrase career highlight once.
I wanted one command I could give either agent:
$obsidian-hybrid-retrieval Find the situations where I performed unusually well.
And I wanted it to be local, inspectable, inexpensive, and honest about its evidence.
The architecture
The system has five layers.
1. Markdown remains the source of truth
The notes stay inside my Google Drive Obsidian vault. I did not move the knowledge base into a hosted vector database or create a second cloud copy that could quietly drift from the original.
That matters operationally. When the system returns a result, the final citation is still a path to a note I can open, edit, diff, or delete.
2. Sparse retrieval handles exact evidence
The skill expands a natural-language question into likely evidence terms and searches filenames, headings, phrases, and note bodies.
For Where did Mindreader have a win?, that may include:
traction · revenue · paying customers · retention · conversion · case study
Sparse search is good at customer names, dates, revenue figures, product names, and exact quotations. It is also fast enough to sweep the vault directly.
3. Smart Connections supplies the local semantic layer
The Smart Connections plugin had already indexed the vault with TaylorAI/bge-micro-v2.
The model is unusually small: roughly 66 MB, producing a 384-number vector for each indexed block. It runs locally through Transformers.js. There is no external embedding bill and no need to send every private journal entry, customer note, or strategy memo to an embedding API.
The local vector index lets differently worded ideas sit near each other. Commercial traction, people paid for it, and customer win can be related even when their words do not overlap.
4. RRF combines rankings without comparing incompatible scores
Sparse and dense retrieval produce different kinds of scores. A lexical score of 14.7 cannot be meaningfully averaged with a cosine similarity of 0.83.
Reciprocal Rank Fusion uses position instead:
RRF(document) = Σ weight(retriever) / (k + rank(document))
With the default k = 60, a note ranked first by sparse search and third by dense search receives:
1 / 61 + 1 / 63 ≈ 0.03226
A document that appears in both lists usually rises above one found by only one retriever. The skill starts with equal weights, but the fusion script can expose sparse and dense weights as an evaluation parameter. Factual searches may deserve more lexical weight; exploratory pattern searches may deserve more semantic weight.
5. The agent judges evidence, then opens the source
Retrieval only produces candidates.
The language model must still distinguish:
- a completed result from a target;
- a client win from a demo that never closed;
- an original customer quote from generated landing-page copy;
- three notes repeating one event from three separate achievements.
The skill explicitly requires the agent to open the original Markdown before presenting a claim as fact.
Making it reusable across agents
I packaged the workflow as a skill rather than leaving it as a prompt in one conversation.
The skill contains:
- the canonical vault path;
- the retrieval sequence;
- a deterministic sparse-search helper;
- an RRF fusion helper;
- evidence-quality rules;
- a required answer format with source links and confidence labels.
That makes the behavior portable. Claude and Codex can share the same instructions and scripts instead of each improvising a new search strategy.
The sequence is deliberately simple:
Ask in natural language
I describe what I am trying to recover, not the filenames I happen to remember.
Generate independent rankings
Sparse retrieval finds explicit evidence. Smart Connections provides local semantic candidates when its live lookup is available.
Fuse and rerank
RRF combines independent rankings. The agent then judges intent, negation, provenance, and duplicate evidence.
Verify the source
The agent opens the underlying Markdown and returns clickable citations rather than an unsupported synthesis.
Three searches that changed the answer
The value was not merely finding more files. It was finding a better relationship between them.
1. “Where did Mindreader have a win?”
A basic keyword sweep returned hundreds of documents: pitch drafts, success-story copy, pricing experiments, enterprise proposals, and hypothetical revenue plans.
The semantic and evidence-ranking layers changed the answer. They connected:
- the track-record note documenting 3,000+ users and 100+ paying customers;
- the revenue reconstruction reporting approximately SGD 29,000 in historical sales;
- subscription analysis showing a small but real legacy revenue base;
- retirement notes showing two customers cared enough about their accumulated data to request exports.
Just as importantly, the system excluded CIMB, EXO, and several polished prototypes as “wins” because the source notes said those deals did not close.
The useful result was not a larger list. It was a defensible distinction between traction, residual value, and impressive-looking attempts.
2. “Where have I performed unusually well?”
The answer was distributed across an old Chinese write-up, NOC reflections, Stanford applications, workshop postmortems, and product records.
Read separately, they looked like unrelated chapters:
- building a k-means segmentation system for 500,000 dating-app users;
- leading Stanford MS&E students while still an intern;
- winning third prize and People's Choice at a Berkeley hackathon;
- turning live workshops into product feedback, testimonials, and a guided application.
Together they revealed a repeatable operating pattern: I do well when a problem is ambiguous, crosses technical and human boundaries, and needs to become something other people can actually use.
That is more useful for a portfolio than “I know Python and AI.” It identifies the situation in which those skills create leverage.
3. “What did Nicholas's workshop feedback really imply?”
One transcript mentioned a preflight checklist: participants should know what to download before the live session.
On its own, that sounds like a small logistics request.
Connected with earlier workshop notes, it pointed to a larger design problem:
- mobile devices were the default;
- account and navigation friction consumed live teaching time;
- participants struggled when prompts or playbooks required copying and scrolling;
- the event path needed to be rehearsed from link to final output.
The stronger product lesson was not “send a checklist.” It was move setup work out of the live experience and protect the workshop's attention budget.
That thread became input for a better 90-minute workshop format, not merely an extra PDF.
The privacy and cost trade-off
“Local” needs a precise definition.
My Markdown remains local. The embedding model runs locally. The generated vectors are stored locally. After the model is downloaded, creating and comparing embeddings does not require an external model API.
That removes one category of exposure and recurring cost.
It does not make the entire computer magically private. Google Drive still syncs the vault. Claude or Codex may read note excerpts when I ask an agent to reason over them. Device security, sync configuration, agent permissions, and secret hygiene still matter.
The privacy improvement is narrower and real: I do not need to send the entire vault to a third-party embedding service just to make semantic retrieval work.
What works today—and what is still being tightened
Inside Obsidian, Smart Connections owns the local model and vector index.
The shareable agent skill always has a reliable sparse-search path. When a callable Smart Connections lookup is available, it adds the dense ranking and runs RRF. When that endpoint is unavailable, the skill says so and falls back to sparse retrieval plus language-model reranking.
It does not pretend a dense query happened.
That fallback is intentional. A retrieval system should degrade visibly, not silently produce a weaker answer while preserving the label “hybrid search.”
The next engineering step is a small local bridge that keeps Smart Connections' index warm and exposes one stable lookup method to every approved agent. That will remove duplicated model loading and make dense + RRF the normal agent path rather than the accelerated path.
What I learned
The embedding model was not the difficult part.
The difficult parts were:
- deciding which vault was canonical;
- avoiding thousands of slow Google Drive file opens per query;
- preserving provenance through ranking and synthesis;
- treating generated drafts differently from direct evidence;
- making degraded behavior explicit;
- packaging the workflow so another agent follows the same rules.
The durable asset is not a vector database.
It is the retrieval contract: where truth lives, how candidates are combined, what the agent must verify, and when the system is allowed to say “I do not have the dense layer available.”