A laptop with 16–32 GB of memory can answer questions about your own notes, code and unfinished ideas, and none of your data leaves the computer. Choosing the model is the easy part. Two other things decide whether the answers are useful: how much memory the model has, and how well it finds the right pages of your files. Without them, it just guesses, and it sounds sure even when it is wrong.
This describes local AI models and tools as they were in September 2026. Models, default settings and tool licences change every few months, so check the current version of anything you install.
The gist
Run a mid-sized model (7 to 14 billion parameters, compressed so it fits on a laptop) on your own computer. Most of the work isn't choosing the model. It's making sure the model fits in your computer's memory, and making sure it finds the right pages of your files.
- Choose the model by your memory, and leave room for the conversation. Pick the biggest model that fits in your computer's fast memory together with the conversation. A model that only just fits will start, but it runs very slowly once you ask it something.
- Let the system look things up; don't retrain the model or paste everything in. Cut your files into small pieces at natural breaks (headings, paragraphs, functions). Keep a note of which file each piece came from. And search for exact words as well as for meaning.
- Pick one search model and never change it. The search uses a small second model that turns your files into numbers. If you change it, you have to process all your files again.
- Make every answer say which file it came from. If part of an answer names no file, check it first: the model may have made it up.
- Check that "local" really means nothing leaves your computer. Turn off the usage data apps send back, and any setting that sends questions to an online AI when the local one can't answer. Make Ollama reachable only from your own computer. And don't connect your files to an online assistant.
The mental model: three parts you can replace
A private artificial intelligence (AI) assistant for your own files has three separate parts. You can replace any one of them without changing the others:
- A runtime and a chat model. The runtime is the program that loads a large language model (LLM) into memory. The model writes the answers. Ollama, llama.cpp and LM Studio are runtimes. Qwen, Gemma and Llama are models.
- Search. A small embedding model (the search model from the gist) plus an index of your files, like the index at the back of a book. Its only job is to find the few passages that answer your question.
- An interface. This is where you type your question: a chat window, your code editor, your notes app, or an agent (an AI tool that does tasks for you).
Most people think the chat model is the most important part. But it knows nothing about your files, and it never will. It learned from the public internet, and then it stopped learning. When it answers about your project or your writing, it reads the answer from passages the search found and added to your question, at the moment you ask. That is why this post spends as much time on memory and search as on models.
Why run it on your own computer, and what you lose
Running a model on your own computer makes sense when your files must not leave it. There are four common reasons:
- Privacy. Client code under a non-disclosure agreement (NDA), unpublished writing, product plans, health or legal notes. This is intellectual property (IP) you may not share with another company, or don't want to.
- No promise to trust. An online AI company can promise not to keep your data or train on it, but that is only a contract. A model on your laptop needs no promise, because nothing is sent.
- Offline and stable. It works on a plane. The model only changes when you download a new one.
- Fixed cost. Once you have paid for the computer, a thousand questions cost the same as one.
But there is a real price. A model small enough for a laptop is clearly weaker than the best online models at problems that need many steps of thinking. And you have to run everything yourself: you choose, update and fix every part.
| Local model | Online AI service | |
|---|---|---|
| Your data leaves the machine | ||
| Lookup, summaries, writing drafts from your files | ||
| Hard problems with many steps | ||
| Cost per question | ||
| Works offline | ||
| Setup and maintenance |
A local setup works best for exactly the task this post is about: questions whose answer is in your files. For example: "What did I decide about X?", "Where in the code do we handle Y?", "Summarise these notes." The search does most of the work, and a mid-sized model is good enough to read the passages it finds and write an answer based on them.
Resources: memory decides everything
One number decides which models you can run: fast memory. Both the model and its working memory for the conversation must fit in memory the processor can read quickly. If they don't, the model becomes extremely slow. The next five parts add up that memory, one piece at a time.
The model's size: parameters × bits
A model is made of billions of numbers called parameters (also called weights). The computer stores each number using a few bits, the 0s and 1s of computer memory. More bits make each number more exact, but they take more memory. So a model's size is its number of parameters times the bits used for each one.
Most models are published with 16 bits per parameter. But the programs that run them usually download a quantized version: each number is rounded to fewer bits. It's like saving a photo as a smaller JPEG. The model loses a little quality, but it becomes much smaller. This compression is what makes it possible to run models on a laptop at all.
llama.cpp saves models in a single-file format called GGUF, and names each compression level with a code: Q4_K_M means about 4 bits per number, and Q8_0 means about 8. In llama.cpp's own measurements, Q4_K_M uses 4.89 bits per number on average, and Q8_0 uses 8.50[1]. Q4_K_M is the usual choice: it loses only a little quality compared with 16 bits, and the file is more than 3 times smaller.
| Parameters | Q4_K_M (≈4.9 bits) | Q8_0 (≈8.5 bits) | 16-bit original |
|---|---|---|---|
| 4 B | ≈2.4 GB | ≈4.3 GB | ≈8 GB |
| 8 B | ≈4.9 GB | ≈8.5 GB | ≈16 GB |
| 14 B | ≈8.6 GB | ≈14.9 GB | ≈28 GB |
| 32 B | ≈19.6 GB | ≈34 GB | ≈64 GB |
| 70 B | ≈42.8 GB | ≈74 GB | ≈140 GB |
(B = billion parameters. Real files are a few percent bigger or smaller, because some parts of the model are kept at more bits.)
The conversation's memory: the KV cache
The second cost is the conversation itself. Models read text in tokens: small pieces of text, each about ¾ of a word. Everything the model can see at once is its context: your question, the passages the search found, and the conversation so far. The maximum size of that is the context window.
While the model writes, it keeps notes on every token in the context, so it doesn't have to read everything again for each new word. These notes are called the key-value (KV) cache, and they take memory for every token.
For readers who want the formula: the size per token is 2 × layers × KV heads × head size × bytes per value, times the number of tokens in the context[2]. For Llama 3.1 8B (32 layers, 8 KV heads of size 128, 16-bit values), that is 128 KiB (about 0.13 MB) per token:
| Context window | KV cache |
|---|---|
| 4,000 tokens | ≈0.5 GiB |
| 32,000 tokens | ≈4 GiB |
| 128,000 tokens | ≈16 GiB |
At 128,000 tokens, the cache is three times bigger than the model itself. This matters for search: every passage added to your question uses up space in the context. It's also why the programs that run models use a small context window by default. Ollama now picks the default from how much graphics memory you have: 4,000 tokens below 24 GiB, 32,000 up to 48 GiB, and 256,000 above that. You can make it bigger with the OLLAMA_CONTEXT_LENGTH setting, or with num_ctx for a single request. ollama ps shows how much it actually reserved[3].
Where the memory is: a Mac, a graphics card, or the main processor
A computer has two kinds of processor. The central processing unit (CPU) is the main one. The graphics processing unit (GPU) is built for graphics, and it runs AI models much faster. Where the model's memory comes from depends on your machine:
- A Mac with Apple Silicon has one shared memory (random-access memory, RAM) that the CPU and the GPU both use. Apple calls it unified memory. So a 64 GB MacBook can hold models that no graphics card sold to normal buyers can. By default, macOS lets the GPU use only about two-thirds to three-quarters of it. The command
sudo sysctl iogpu.wired_limit_mb=<megabytes>raises that limit until you restart[4]. Apple's machine-learning framework MLX[6], and the Mac support in llama.cpp (called Metal), are what let these programs use the Mac's GPU. - A Windows or Linux computer with an NVIDIA graphics card: the card has its own memory, called video memory (VRAM), and that is the maximum. Even NVIDIA's best card for normal buyers has 32 GB[5].
- Only the CPU: the model uses the normal RAM. There is a lot of it, but it is slow to read.
| Mac (shared memory) | NVIDIA graphics card (VRAM) | CPU only (normal RAM) | |
|---|---|---|---|
| Memory the model can use | |||
| Speed of reading it | |||
| What happens when it doesn't fit | |||
| Best for | Laptops with 32 GB+ | Desktops, 8–14B models, fast | Search models, small chat models |
Speed depends on memory, not on calculating
Once a model fits, how fast it writes depends on how fast the memory can be read. To write each token, the computer has to read every active parameter once. So the limit is how much data per second the memory can deliver (its bandwidth), not how fast the processor can calculate[2]. A useful maximum: tokens per second ≈ bandwidth ÷ model size.
Apple's M4, M4 Pro and M4 Max chips read 120, 273 and 546 GB per second[7]. For an 8B model of 4.9 GB, that gives a maximum of about 24, 55 and 110 tokens per second. Real speeds are lower, but the faster chips stay faster by about the same amount. This also explains why mixture-of-experts (MoE) models are popular. An MoE model is made of many smaller "expert" parts, and it uses only a few of them for each token. Qwen3-30B-A3B, for example, has 30 billion parameters, and all of them must fit in memory. But it reads only about 3 billion for each token[8]. So it answers as fast as a small model, and knows as much as a bigger one.
What your computer can really run
Here are the model, the conversation and the speed, all together:
| Memory | Chat model that fits with space to spare | Usable context | Result |
|---|---|---|---|
| 8 GB | 3–4B at Q4 | ≈4,000 tokens | |
| 16 GB | 7–9B at Q4 | 8,000–16,000 tokens | |
| 32 GB | 12–14B at Q4, or a 20–30B MoE | ≈32,000 tokens | |
| 64 GB+ | 27–32B normal (not MoE) at Q4 | 32,000+ tokens |
Plan for disk space too: a few models to compare add up to tens of gigabytes.
Choosing two models, not one
Now that you know how much memory you have, you choose two models. A chat model reads and writes. An embedding model (the search model) turns text into lists of numbers, so the search can find it.
The chat model should be the biggest one that fits, with room left for the conversation. A 14B model that only fits with a 2,000-token context window is worse for this job than an 8B model with 16,000 tokens, because it can't see enough of your files at once. These are the open-weight model families worth trying first. Open-weight means you can download the model and run it yourself, and a family is a group of models from one maker, in different sizes:
The Apache 2.0 licence lets you use a model in a business with almost no conditions. Meta's licence has more rules; for example, you need a separate licence if your product has more than 700 million users a month. That won't affect you for personal use, but read it before you use the model at work. Try two or three models with your questions on your files. Public tests that score models rarely tell you which one works best with your notes.
The embedding model is very small compared with the chat model. embeddinggemma has 300 million parameters, and nomic-embed-text, bge-m3 and qwen3-embedding are about the same size[14]. It runs well on the CPU. Choose it once and keep it. Each embedding model makes its own kind of numbers, and numbers from two models can't be compared. So if you switch, you have to process all your files again.
With Ollama, downloading one of each takes two commands:
ollama pull qwen3:8b
ollama pull nomic-embed-text
Making it know your files: retrieval
The chat model can't learn your files, so the system looks them up for every question. This method is called retrieval-augmented generation (RAG)[15]. In plain words, the model writes its answer with the help of text that was looked up first. It works in three steps: cut your files into pieces (chunks), store a list of numbers (a vector) for each piece, and when you ask a question, add the closest pieces to it.
A vector places a passage on an imaginary map of meaning. Passages about the same thing sit close together, even when they use different words. The whole method fits in one page of Python, using Ollama's library. It's worth reading once, to see that the desktop apps aren't doing anything magic:
import pathlib
import numpy as np
import ollama
EMBED, CHAT = "nomic-embed-text", "qwen3:8b"
NOTES = pathlib.Path("~/notes").expanduser()
# 1. Cut: one piece per "## " section of every Markdown file
chunks = []
for path in NOTES.rglob("*.md"):
for section in path.read_text(encoding="utf-8").split("\n## "):
if section.strip():
chunks.append((path.name, section[:2000]))
# 2. Turn every piece into numbers, once (Ollama scales them, so we can compare them directly)
vectors = np.array(ollama.embed(model=EMBED, input=[text for _, text in chunks])["embeddings"])
def ask(question, k=5):
# 3. Search: find the k pieces closest to the question
q = np.array(ollama.embed(model=EMBED, input=question)["embeddings"][0])
best = np.argsort(vectors @ q)[::-1][:k]
context = "\n\n".join(f"[{chunks[i][0]}]\n{chunks[i][1]}" for i in best)
# 4. Answer: using only those pieces
reply = ollama.chat(model=CHAT, messages=[
{"role": "system", "content": "Answer only from the notes provided. "
"Cite the file name in brackets. If the notes don't say, say so."},
{"role": "user", "content": f"{context}\n\nQuestion: {question}"},
])
return reply["message"]["content"]
print(ask("What did I decide about the billing refactor?"))
Every search tool is built on those four steps, and adds more on top. These additions make the biggest difference:
- Cut at natural breaks. Use headings for text, functions or classes for code, and one entry per idea for a journal. A piece cut in the middle of an idea is hard to find, and hard to understand when it's found.
- Keep information about each piece: file path, date, project. Then answers can say which file they came from, and you can filter ("only notes from 2026").
- Search for exact words too. Vectors capture meaning, but they miss exact words: function names, ticket numbers and people's names are easier to find with a normal word search. Most tools can do both at once, which they call "hybrid" search.
- Store the index in a simple place. A few thousand notes fit in memory, like in the code above. For more, a single-file database keeps everything in one local file, for example the sqlite-vec extension for SQLite[16], or LanceDB.
Why not train the model on your files?
Fine-tuning means training a model some more, on your own text. It sounds like the way to make it "know" your files, but it's the wrong tool here, for four reasons:
- It teaches style and format better than facts.
- It needs much more computing power than simply running the model.
- It can't tell you which file an answer came from.
- It's out of date the moment you edit a note.
When researchers compared the two ways of giving a model new knowledge, search always did better than this kind of training, both for facts the model had seen before and for completely new ones[17].
Why not paste all your files into the question?
For a single document, you should: if it fits in the context window, just include it. For a whole collection of notes, no, for two reasons. First, the KV cache makes a long context use a lot of memory. Second, models use information at the start and the end of a long text much better than information in the middle[18]. Five well-chosen pieces work better than fifty average ones.
Three setups for three kinds of files
The three parts stay the same whatever you're working with. What changes is which interface suits your files. These are the tools you'll hear about most, grouped by the part they provide:
| Tool | Layer | What it is | Retrieval over your files |
|---|---|---|---|
| Ollama | Runtime | A program you use by typing commands, which other apps on your computer can connect to | |
| LM Studio | Runtime + chat | Desktop app for GGUF and MLX models[19] | [20] |
| Open WebUI | Interface | A web chat for Ollama that you run yourself | [21] |
| AnythingLLM | Interface + retrieval | Desktop app, with a separate space per project | [22] |
| Continue | Interface (editor) | Visual Studio Code and JetBrains extension | [23] |
| Obsidian plugins | Interface (notes) | Copilot for Obsidian, Smart Connections | |
| Model Context Protocol server | Connector | Lets any app that supports it search your index |
LM Studio has been free for work use since July 2025[24]. It also copies the way OpenAI's service works, at a local address (port 1234), so most tools built for OpenAI can use it instead. Open WebUI changed to its own licence in April 2025: with more than 50 users, you may not remove its name and logo. That doesn't matter for one person, but know it before you give it to a team[25].
Docs and notes
A folder of Markdown files, PDFs (Portable Document Format files) and exported pages is the easiest case. AnythingLLM is the easiest way: create a workspace, drag the files into it, and choose Ollama as the model provider. It then cuts the files into pieces, makes their search numbers, and builds the index, using its own search model and a LanceDB database. Open WebUI does the same with "Knowledge" collections, which you add to a chat by typing #. If your notes are in Obsidian, Copilot for Obsidian connects to Ollama or LM Studio directly[26], and Smart Connections makes its search numbers on your computer by default[27], so your notes never leave the app.
Code projects
For code, the interface belongs in your code editor. Continue can use Ollama for chat, for autocomplete, and (with nomic-embed-text as its search model) for searching your code locally[23]. Code needs the exact-word search from earlier. Names in code must match exactly, so a word search finds InvoiceScheduler, where a meaning search returns "something about billing". Cut the code into one piece per function or class, and include the file path in every piece, so that answers point to the right place in your code.
An ideas journal
Ideas are short, there are many of them, and they are only loosely connected. So the task changes: instead of finding a fact, you look for related ideas ("what else have I written about this?"). A small model is enough, even a 4B one, because each note is short, and the value is in finding links between ideas, not in writing long answers. Save each entry as its own piece, with its date, and ask questions across time: "How has my thinking on pricing changed since spring?"
One index, many interfaces
The Model Context Protocol (MCP) is an open standard that Anthropic introduced in November 2024[28]. With it, you build the search part once, as a small server that searches your index. Then you connect it to any app that supports MCP: a desktop chat app, a code editor, an agent. So you can change interfaces without processing your files again. But there is one risk: the passages the search finds go wherever the app's model runs. If the app uses an online model, your passages go online too.
Using it to write documentation, not only to answer questions
The same setup that answers questions can also write documentation from your files. That's the second way to make private work useful. Local models are good at first drafts from a limited amount of text: for example, a README section (the file that describes a project) from a module's code, a summary of a folder of meeting notes, an architecture decision record (ADR, a note that records a technical decision) from ideas spread across many places, or docstrings (the comments that explain code) for a file.
The simplest version needs no search at all, because the input is a single file:
ollama run qwen3:8b "Write a README section for this module: what it does, its public functions and their inputs. Use only what is in the code. $(cat src/billing/invoice.ts)"
Three habits stop the model from inventing things:
- Give it one thing at a time. A module, a note, a week of journal entries: small enough to fit the context window with space left over. Then nothing gets cut off, and the model doesn't invent anything to fill the gaps.
- Make it say where each fact came from. Ask for the file or section behind each statement, like the Python example does. A statement with no source is the first place to look for something invented.
- Your code and notes are the truth; the docs are generated. When the code or notes change, generate the documentation again instead of fixing it by hand, and check it like any other change.
Common mistakes that break a private setup
Most problems come from the search and the connections between the parts, not from the model:
- Sure-sounding answers when the right passage is missing. If the search doesn't find the right piece, the model still answers, from what it learned on the internet. The answer sounds good, but it's wrong. Tell the model to say so when the notes don't cover something, and check which files it names.
- An out-of-date index. The search only knows files that were processed. Process them again at fixed times, or whenever a file changes. Otherwise yesterday's decision doesn't exist.
- A context window that is too small, without any warning. If your question plus the passages found are bigger than the window, part of them never reaches the model. Check
ollama ps, and make the window bigger when you add more pieces. - "Local" that isn't really local. The model can run locally while something else secretly connects to the internet: an app set up to use an online model when the local one can't answer, an online search model left as the default, or usage data sent back to the app's maker. AnythingLLM sends anonymous usage data by default, until you turn it off in its privacy settings or set
DISABLE_TELEMETRY[22]. Older versions of the Chroma database did the same, until version 1.5.4[29]. And an MCP server connected to an online assistant sends every passage it finds to that online service. That's how it's meant to work, but it's the opposite of private. - A server anyone on the network can reach. By default, Ollama listens only at
127.0.0.1:11434, which only your own computer can reach. SettingOLLAMA_HOST=0.0.0.0makes it reachable by anyone on the network, with no password[30]. Keep the default, or put a program in front of it that asks for a password (a proxy). - Licences you didn't read. Model and tool licences are different, as the tables above show. Check them before you use anything for more than yourself.
Conclusion
A normal developer laptop can easily run a private assistant for your own work. Most of the effort goes into two decisions, and neither is about which model is popular this month. Choose the model by your memory, with room left for the conversation. And spend your effort on search: sensible pieces, information about each piece, exact-word search as well as meaning search, and an index that stays up to date. Get those right, and an 8B model on a 16 GB computer will find a decision you made last spring, summarise a code project and write its README, and your files never leave your computer.
References
- ggml-org, llama.cpp quantize tool: bits per weight for Llama-3.1-8B — GitHub
- NVIDIA, Mastering LLM Techniques: Inference Optimization — NVIDIA Technical Blog
- Ollama, Context length — Ollama docs
- ggml-org, Metal: only 2/3 or 3/4 of unified memory usable (discussion #2182) — GitHub
- NVIDIA, GeForce RTX 5090 — specifications
- Apple, MLX: an array framework for Apple silicon — GitHub
- Apple, Apple introduces M4 Pro and M4 Max — Apple Newsroom, October 2024
- Qwen Team, Qwen3: Think Deeper, Act Faster — Qwen blog
- Ollama, qwen3.5 — Ollama library
- Google, Gemma 4 model card — Google AI for Developers
- Meta, Llama 3.1 Community License Agreement
- Mistral AI, Mistral-Small-3.2-24B-Instruct-2506 — Hugging Face
- Ollama, gpt-oss — Ollama library
- Ollama, Embeddings — Ollama docs
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — arXiv, 2020
- Alex Garcia, sqlite-vec: a vector search SQLite extension — GitHub
- Ovadia et al., Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs — arXiv, 2023
- Liu et al., Lost in the Middle: How Language Models Use Long Contexts — arXiv, 2023
- LM Studio, Welcome to LM Studio Docs — LM Studio
- LM Studio, Chat with Documents — LM Studio docs
- Open WebUI, Retrieval Augmented Generation (RAG) — Open WebUI docs
- Mintplex Labs, AnythingLLM README (telemetry and vector database) — GitHub
- Continue, Embedding model role — Continue docs
- LM Studio, LM Studio is free for use at work — LM Studio blog, July 2025
- Open WebUI, License — Open WebUI docs
- Logan Yang, Copilot for Obsidian: LLM providers — GitHub
- Brian Petro, Smart Connections — GitHub
- Anthropic, Introducing the Model Context Protocol — November 2024
- Chroma, Open-source Chroma (telemetry note) — Chroma docs
- Ollama, FAQ: How do I configure Ollama server? — Ollama docs