A laptop with 16–32 GB of memory can answer questions about your own notes, repositories and half-finished ideas without sending a byte anywhere. The model is the easy part: how much memory you give it, and how well it finds the right pages of your work, decide whether it's useful or just a confident guesser.
This reflects the local-model landscape as of September 2026. Model families, default settings and tool licences change every few months, so check the current version of anything you install.
The gist
Run a quantized 7–14B model locally and spend your effort on memory sizing and retrieval, not on model shopping.
- Size to memory, with room for context. Pick the largest chat model whose quantized weights plus conversation cache fit in fast memory, not the largest one that merely loads.
- Retrieve; don't fine-tune or stuff the prompt. Chunk along the material's seams, keep metadata, add keyword search.
- Choose one embedding model and keep it. Switching means re-indexing everything.
- Make every answer cite a file. An uncited claim is where to look for an invention.
- Audit what "local" means. Turn off telemetry and cloud fallbacks, keep Ollama on localhost, and don't wire your index into a cloud assistant.
The mental model: three layers you can swap
A private artificial intelligence (AI) assistant over your own material is three separate pieces, and each one can be replaced without touching the others:
- A runtime and a chat model — the program that loads a large language model (LLM) into memory and generates text. Ollama, llama.cpp and LM Studio are runtimes; Qwen, Gemma and Llama are models.
- Retrieval — a small embedding model plus an index of your files, whose only job is to find the few passages that matter for a question.
- An interface — wherever you actually ask: a chat window, your editor, your notes app, or an agent.
The piece people expect to matter most — the chat model — knows nothing about your files and never will. It was trained on the public internet and then frozen. Everything it "knows" about your billing refactor or your novel's second act, it reads at question time, from passages the retrieval layer found and pasted into the prompt. That is why the rest of this post spends as long on memory and retrieval as on models.
Why local, and what you give up
Running models on your own hardware makes sense when the material shouldn't leave it. Four reasons come up again and again:
- Confidentiality. Client code under a non-disclosure agreement (NDA), unpublished writing, product plans, health or legal notes — intellectual property (IP) you're not allowed, or not willing, to hand to a third party.
- No policy to trust. A cloud provider's no-training or zero-retention promise is a contract. A model on your laptop needs no promise, because nothing is sent.
- Offline and stable. It works on a plane, and the model only changes when you pull a new one.
- Flat cost. Once the hardware is paid for, a thousand questions cost the same as one.
The trade is real, though. A model that fits in laptop memory is noticeably weaker than a frontier cloud model at long, multi-step reasoning, and you become the operator: you choose, update and debug everything yourself.
| Local model | Cloud API (application programming interface) | |
|---|---|---|
| Your data leaves the machine | ||
| Lookup, summaries, drafting over your files | ||
| Long multi-step reasoning | ||
| Cost per question | ||
| Works offline | ||
| Setup and upkeep |
The sweet spot for a local setup is exactly the task this post is about: questions whose answer is in your material — "what did I decide about X", "where do we handle Y", "summarise these notes". Retrieval does the heavy lifting there, and a mid-sized model is enough to read the passages and write a grounded answer.
Resources: it all comes down to memory
The one number that decides what you can run is fast memory: the model's weights and its working memory for the conversation must both fit in memory the processor can read quickly, or generation slows to a crawl. The next five parts build that budget one piece at a time.
Weights: parameters × bits
A model's size is its parameter count times the bits used to store each parameter. Most models are published at 16 bits per weight, but runtimes ship them quantized — rounded to fewer bits — with a small loss in quality. Quantization is what makes local models practical at all.
llama.cpp's single-file GGUF format names its quantization levels with codes like Q4_K_M (about 4-bit) and Q8_0 (about 8-bit). In llama.cpp's own measurements Q4_K_M averages 4.89 bits per weight and Q8_0 8.50[1]. Q4_K_M is the usual default: the quality drop from 16-bit is small, and the size drop is more than threefold.
| Parameters | Q4_K_M (≈4.9 bits) | Q8_0 (≈8.5 bits) | 16-bit original |
|---|---|---|---|
| 4 B | ≈2.4 GB | ≈4.3 GB | ≈8 GB |
| 8 B | ≈4.9 GB | ≈8.5 GB | ≈16 GB |
| 14 B | ≈8.6 GB | ≈14.9 GB | ≈28 GB |
| 32 B | ≈19.6 GB | ≈34 GB | ≈64 GB |
| 70 B | ≈42.8 GB | ≈74 GB | ≈140 GB |
(B = billion parameters. Real files differ by a few percent, because some layers are kept at higher precision.)
The context tax: the KV cache
The second cost is the conversation itself. While generating, the model keeps a key-value (KV) cache — a key vector and a value vector for every attention head in every layer, for every token in the context — so it doesn't recompute the whole prompt for each new word. Its size per token is 2 × layers × KV heads × head size × bytes per value, times the number of tokens in the context[2].
For Llama 3.1 8B (32 layers, 8 KV heads of size 128, 16-bit values) that comes to 128 KiB per token:
| Context window | KV cache |
|---|---|
| 4,000 tokens | ≈0.5 GiB |
| 32,000 tokens | ≈4 GiB |
| 128,000 tokens | ≈16 GiB |
At 128,000 tokens the cache is three times the size of the model. This matters for retrieval, because every passage you paste into the prompt spends context — and it's why runtimes default to modest context windows. Ollama now picks the default from available graphics memory: 4,000 tokens below 24 GiB, 32,000 up to 48 GiB, 256,000 above that. You raise it with the OLLAMA_CONTEXT_LENGTH variable or a per-request num_ctx, and ollama ps shows what was actually allocated[3].
Where the memory lives: Apple Silicon, a graphics card, or the CPU
Where that budget comes from depends on the machine, and the three options behave very differently:
| Apple Silicon (unified memory) | NVIDIA graphics card (VRAM) | CPU only (system RAM) | |
|---|---|---|---|
| Memory the model can use | |||
| Speed of reading it | |||
| What happens when it doesn't fit | |||
| Sweet spot | Laptops with 32 GB+ | Desktops, 8–14B fast | Embedding models, small chat models |
A Mac shares one pool of random-access memory (RAM) between the central processing unit (CPU) and the graphics processing unit (GPU), so a 64 GB MacBook can hold models no consumer graphics card can. By default macOS lets the GPU claim only about two-thirds to three-quarters of that pool; sudo sysctl iogpu.wired_limit_mb=<megabytes> raises the limit until the next reboot[4]. On a Windows or Linux machine, the graphics card's own video memory (VRAM) is the ceiling: even NVIDIA's top consumer card has 32 GB[5]. Apple's machine-learning framework MLX[6] and the Metal backend in llama.cpp are what let runtimes use the Mac's GPU.
Speed is bandwidth, not compute
Once a model fits, how fast it writes depends on how fast memory can be read. Producing each token means reading every active weight once, so generation is bound by memory bandwidth rather than arithmetic[2]. A useful ceiling: tokens per second ≈ bandwidth ÷ model size.
Apple's M4, M4 Pro and M4 Max move 120, 273 and 546 GB/s[7]. For a 4.9 GB 8B model that gives ceilings of roughly 24, 55 and 110 tokens per second; real numbers land below that, but the ratio holds. It also explains the appeal of mixture-of-experts (MoE) models such as Qwen3-30B-A3B: all 30 billion parameters must fit in memory, but only about 3 billion are read per token[8], so it answers at the speed of a small model with the knowledge of a larger one.
What your machine can realistically run
Putting weights, context and bandwidth together:
| Memory | Chat model that fits comfortably | Usable context | Verdict |
|---|---|---|---|
| 8 GB | 3–4B at Q4 | ≈4,000 tokens | |
| 16 GB | 7–9B at Q4 | 8,000–16,000 tokens | |
| 32 GB | 12–14B at Q4, or a 20–30B MoE | ≈32,000 tokens | |
| 64 GB+ | 27–32B dense at Q4 | 32,000+ tokens |
Budget disk space too: a handful of models to compare adds up to tens of gigabytes.
Picking two models, not one
With the budget known, you choose two models: a chat model that reads and writes, and an embedding model that turns text into vectors for search.
The chat model should be the largest that fits with room left for context — a 14B model squeezed in with a 2,000-token window is worse at this job than an 8B with 16,000, because it can't see enough of your material. The open-weight families worth trying first:
Apache 2.0 lets you use a model commercially with almost no strings. Meta's licence adds conditions, such as a separate licence above 700 million monthly users, that won't touch a personal setup but are worth reading before you build on it at work. Try two or three candidates on your questions over your files — published benchmarks rarely predict which one handles your notes best.
The embedding model is tiny by comparison — embeddinggemma has 300 million parameters, and nomic-embed-text, bge-m3 and qwen3-embedding are in the same class[14]. It runs happily on the CPU. Choose it once and stick with it: vectors from different embedding models aren't comparable, so switching means re-indexing everything.
With Ollama, fetching one of each is two commands:
ollama pull qwen3:8b
ollama pull nomic-embed-text
Making it know your material: retrieval
The chat model can't learn your files, so the system looks them up for every question. This is retrieval-augmented generation (RAG)[15]: split your material into chunks, store a vector for each, and at question time paste the closest chunks into the prompt.
A vector is a list of numbers that places a passage in "meaning space": passages about the same thing land close together, even when they share no words. The whole mechanism fits in a page of Python using Ollama's client library, which is worth reading once to demystify what the desktop apps do:
import pathlib
import numpy as np
import ollama
EMBED, CHAT = "nomic-embed-text", "qwen3:8b"
NOTES = pathlib.Path("~/notes").expanduser()
# 1. Chunk: one chunk per "## " section of every Markdown file
chunks = []
for path in NOTES.rglob("*.md"):
for section in path.read_text(encoding="utf-8").split("\n## "):
if section.strip():
chunks.append((path.name, section[:2000]))
# 2. Embed every chunk once (Ollama returns unit-length vectors)
vectors = np.array(ollama.embed(model=EMBED, input=[text for _, text in chunks])["embeddings"])
def ask(question, k=5):
# 3. Retrieve: the k chunks closest to the question
q = np.array(ollama.embed(model=EMBED, input=question)["embeddings"][0])
best = np.argsort(vectors @ q)[::-1][:k]
context = "\n\n".join(f"[{chunks[i][0]}]\n{chunks[i][1]}" for i in best)
# 4. Generate: answer only from those chunks
reply = ollama.chat(model=CHAT, messages=[
{"role": "system", "content": "Answer only from the notes provided. "
"Cite the file name in brackets. If the notes don't say, say so."},
{"role": "user", "content": f"{context}\n\nQuestion: {question}"},
])
return reply["message"]["content"]
print(ask("What did I decide about the billing refactor?"))
Every retrieval tool elaborates on those four steps. The elaborations that make the biggest difference:
- Chunk along the material's own seams. Headings for prose, functions or classes for code, one entry per idea for a journal. A chunk cut mid-argument retrieves badly and reads worse.
- Keep metadata with each chunk — file path, date, project — so answers can cite their source and you can filter ("only 2026 notes").
- Combine vectors with keyword search. Embeddings capture meaning but blur exact strings: function names, ticket numbers and people's names are better found by plain keyword matching, and most tools offer this "hybrid" mode.
- Store the index somewhere boring. A few thousand notes fit in memory, as above; beyond that, a single-file store such as the sqlite-vec extension for SQLite[16] or LanceDB keeps everything in one local file.
Why not fine-tune the model on your files?
Fine-tuning — continuing a model's training on your own text — sounds like the way to make it "know" your material, but it's the wrong tool here. It teaches style and format better than facts, needs far more compute than inference, can't tell you which file an answer came from, and goes stale the moment you edit a note. When researchers compared the two for injecting knowledge, retrieval consistently beat unsupervised fine-tuning, for both familiar and entirely new facts[17].
Why not paste everything into a long context?
For a single document, you should: if it fits in the context window, just include it. For a whole vault, no. The KV cache makes long contexts expensive in memory, and models use information at the start and end of a long prompt much better than information buried in the middle[18]. Five well-chosen chunks beat fifty mediocre ones.
Three setups for three kinds of material
The three layers stay the same whatever you're working with; what changes is which interface fits the material. These are the tools that come up most, grouped by the layer they fill:
| Tool | Layer | What it is | Retrieval over your files |
|---|---|---|---|
| Ollama | Runtime | Command-line tool and local server | |
| LM Studio | Runtime + chat | Desktop app for GGUF and MLX models[19] | [20] |
| Open WebUI | Interface | Self-hosted web chat for Ollama | [21] |
| AnythingLLM | Interface + retrieval | Desktop app organised into workspaces | [22] |
| Continue | Interface (editor) | Visual Studio Code and JetBrains extension | [23] |
| Obsidian plugins | Interface (notes) | Copilot for Obsidian, Smart Connections | |
| Model Context Protocol server | Bridge | Exposes your index as a tool to any compatible app |
LM Studio has been free for work use since July 2025[24], and serves an OpenAI-compatible endpoint on port 1234, so most tools that speak to OpenAI can be pointed at it instead. Open WebUI moved to its own licence in April 2025, which forbids removing its branding above 50 users — irrelevant for one person, worth knowing before rolling it out to a team[25].
Docs and notes
A folder of Markdown, PDFs (Portable Document Format files) and exported pages is the easiest case. AnythingLLM is the shortest path: create a workspace, drop the folder's files into it, pick Ollama as the model provider, and it chunks, embeds and indexes with its own built-in embedder and LanceDB store. Open WebUI does the same through "Knowledge" collections you attach to a chat with #. If the notes live in Obsidian, Copilot for Obsidian talks to Ollama or LM Studio directly[26], and Smart Connections builds its embeddings locally by default[27], so the vault never has to leave the app.
Code projects
For repositories, the interface belongs in the editor. Continue can use Ollama for chat, autocomplete and — with nomic-embed-text in the embedding role — local embeddings of your code[23]. Code rewards the hybrid search described earlier: identifiers are exact strings, so keyword matching finds InvoiceScheduler where embeddings return "something about billing". Keep chunks at function or class level, and include the file path in every chunk so answers point at real locations.
An ideas journal
Ideas are short, many and loosely connected, so the job shifts from lookup to association: "what else have I written that relates to this?" A small model is plenty — even 4B — because each note is short and the value is in surfacing connections rather than composing long answers. Index each entry as its own chunk with its date, and ask questions across time: "how has my thinking on pricing changed since spring?"
One index, many interfaces
The Model Context Protocol (MCP), an open standard Anthropic introduced in November 2024[28], lets you write the retrieval layer once — a small server that searches your index — and plug it into any client that speaks the protocol: a desktop chat app, an editor, an agent. That keeps you free to change interfaces without re-indexing. It comes with one catch: the retrieved passages go wherever the client's model runs.
Documenting with it, not just asking it
The same setup that answers questions can write documentation from your material — the other half of making private work usable. Local models are good at first drafts over bounded input: a README section from a module's source, a summary of a folder of meeting notes, an architecture decision record (ADR) from a scattered thread of ideas, docstrings for a file.
The one-shot version needs no retrieval at all, because the input is a single file:
ollama run qwen3:8b "Write a README section for this module: what it does, its public functions and their inputs. Use only what is in the code. $(cat src/billing/invoice.ts)"
Three habits keep the output honest:
- Give it one unit at a time. A module, a note, a week of journal entries — small enough to fit the context window with room to spare, so nothing gets cut and nothing gets invented to fill gaps.
- Make it cite. Ask for the file or section each claim came from, as the Python example does. A claim without a citation is the first place to look for an invention.
- Treat the source as the truth, the docs as generated. Regenerate documentation when the code or notes change rather than hand-patching the output, and review it like any other diff.
Pitfalls that make a private setup fail
Most failures come from retrieval and plumbing, not from the model:
- Confident answers from missing context. If retrieval misses the right chunk, the model still answers — from its general training, fluently and wrongly. Instruct it to say when the notes don't cover something, and check its citations.
- A stale index. Retrieval only knows what was indexed. Re-index on a schedule or on file changes, or yesterday's decision won't exist.
- A silently short context window. If the prompt plus retrieved chunks exceeds the window, part of it never reaches the model. Check
ollama psand raise the context length when you add more chunks. - "Local" that isn't. The model can be local while something else phones out: an app configured with a cloud model as fallback, a cloud embedding provider left as the default, or telemetry. AnythingLLM sends anonymous usage data by default until you turn it off in its privacy settings or set
DISABLE_TELEMETRY[22]; older versions of the Chroma vector database did the same until 1.5.4[29]. And an MCP server plugged into a cloud-hosted assistant sends every retrieved passage to that cloud — correct behaviour, but the opposite of private. - A server open to the network. Ollama binds to
127.0.0.1:11434by default, reachable only from your machine; settingOLLAMA_HOST=0.0.0.0exposes it, with no authentication, to anyone on the network[30]. Keep it on localhost, or put it behind a proxy that checks credentials. - Licences you didn't read. Model and tool licences differ, as the tables above show; check them before anything leaves personal use.
Where this leaves you
A private assistant over your own work is well within reach of an ordinary developer laptop, and most of the effort goes into two decisions that have nothing to do with which model is fashionable this month. Size the model to your memory with room left for context, and invest in retrieval — sensible chunks, metadata, hybrid search, an index kept fresh. Get those right and an 8B model on a 16 GB machine will find last spring's decision, summarise a repository and draft its README without your material ever leaving the desk.
References
- ggml-org, llama.cpp quantize tool: bits per weight for Llama-3.1-8B — GitHub
- NVIDIA, Mastering LLM Techniques: Inference Optimization — NVIDIA Technical Blog
- Ollama, Context length — Ollama docs
- ggml-org, Metal: only 2/3 or 3/4 of unified memory usable (discussion #2182) — GitHub
- NVIDIA, GeForce RTX 5090 — specifications
- Apple, MLX: an array framework for Apple silicon — GitHub
- Apple, Apple introduces M4 Pro and M4 Max — Apple Newsroom, October 2024
- Qwen Team, Qwen3: Think Deeper, Act Faster — Qwen blog
- Ollama, qwen3.5 — Ollama library
- Google, Gemma 4 model card — Google AI for Developers
- Meta, Llama 3.1 Community License Agreement
- Mistral AI, Mistral-Small-3.2-24B-Instruct-2506 — Hugging Face
- Ollama, gpt-oss — Ollama library
- Ollama, Embeddings — Ollama docs
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — arXiv, 2020
- Alex Garcia, sqlite-vec: a vector search SQLite extension — GitHub
- Ovadia et al., Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs — arXiv, 2023
- Liu et al., Lost in the Middle: How Language Models Use Long Contexts — arXiv, 2023
- LM Studio, Welcome to LM Studio Docs — LM Studio
- LM Studio, Chat with Documents — LM Studio docs
- Open WebUI, Retrieval Augmented Generation (RAG) — Open WebUI docs
- Mintplex Labs, AnythingLLM README (telemetry and vector database) — GitHub
- Continue, Embedding model role — Continue docs
- LM Studio, LM Studio is free for use at work — LM Studio blog, July 2025
- Open WebUI, License — Open WebUI docs
- Logan Yang, Copilot for Obsidian: LLM providers — GitHub
- Brian Petro, Smart Connections — GitHub
- Anthropic, Introducing the Model Context Protocol — November 2024
- Chroma, Open-source Chroma (telemetry note) — Chroma docs
- Ollama, FAQ: How do I configure Ollama server? — Ollama docs