← All postsBlog

Letting an AI agent work on your notes without uploading them

By Antone Heyward, maker of Pellet · 8 September 2026

The useful version of “AI in your notes” isn’t a chat box in the corner. It’s an agent that can read your vault, find the six commitments buried in last week’s meeting notes, and make them tasks with the right dates, while you watch. That’s tedious by hand and easy to describe, which is exactly the shape of work agents are good at.

The question is where your notes have to be for that to happen. The default answer in most products is “on our servers”. I think that’s the wrong default for a personal knowledge base, and this post is about the alternative.

What an agent is actually good for here

Concrete jobs, all of which I’ve used:

• “Add a task for every unticked checkbox in this week’s daily notes, due Friday.”
• “Find every note that mentions the Monterrey site and give me a one-paragraph summary with links.”
• “What’s overdue that I haven’t touched in two weeks?”
• “Make a note for each person in this meeting who doesn’t have one, and link them from the meeting.”
• “Move everything tagged #warehouse that’s done into the archive.”

None of these need a model to be clever. They need it to read a lot of files, apply a rule, and write a few changes. The value is in the reading, and the reading is the part you should be careful about.

Why the notes should stay on your machine

A second brain is the most personal dataset you own: what you’re working on, who you’re dealing with, what you decided and why, what you’re worried about. Sending it to a service to get help with it changes the deal you made when you wrote it down.

There’s a practical argument too. A cloud agent works on a copy, and copies drift. An agent that works on the files on your disk is working on the truth, and every change it makes is right there in the same folder, under the same backups and the same version history as your own edits.

So the design I want is: the notes stay where they are, the agent connects to them locally, and the connection has a fence around it.

How Pellet does it

Pellet can open a small connection on your own computer that AI tools and scripts can use to read and write your notes, tasks and calendar. It speaks MCP, a standard several AI tools already understand, so which tool you point at it is your choice. Everything involved is on your own computer: the tool, the connection, and your files. See Letting an AI agent use your vault.

The parts I’d want anyone to check before trusting a feature like this:

• It’s off until you turn it on, and switching it off closes it immediately.
• It only listens to your own computer. Nothing on the internet can reach it, with or without the address.
• You can require a token, kept in your operating system’s keychain, so other software on the machine can’t use the connection by accident.
• What the agent can do is yours to limit. A read-only switch that hides every writing operation rather than merely refusing it. A folder allowlist, which makes notes outside those folders invisible to the agent, not just unwritable. Per-operation switches to turn off, say, deleting notes or touching your calendar.
• Mistakes are recoverable. Deleting goes to Pellet’s trash, not to nowhere, and file history keeps a version of any note an agent overwrites.

And the reach is real: around fifty operations covering tasks (including completing a repeating task correctly), full-text search inside note bodies, notes, boards, subtasks, checklists, your connected calendars, daily notes, templates, the archive and the trash.

What “local” does and doesn’t mean

Being precise here, because it’s easy to overclaim.

The connection is local. The files never go to Pellet, because there’s no Pellet server to go to. What the AI tool on the other end of that connection does with what it reads is that tool’s business and yours. If you connect a cloud-hosted model, the text it reads is sent to that model’s provider; if you connect a model running on your own machine, nothing leaves it. Pellet doesn’t choose for you, and the Help page says so in the same words.

That’s the honest shape of the thing: Pellet keeps your notes local and gives you the switches; you decide which agent gets to look, at what, and with what powers.

A sensible first setup

1. Turn the connection on in Settings › AI & MCP, with a token.
2. Switch on read-only and try a few questions: what’s overdue, summarise this project’s notes, find everything about a client.
3. When you trust it, turn read-only off but keep the folder allowlist tight: one project folder, the daily notes, nothing else.
4. Widen from there, one switch at a time.

Start with reading. Most of the value is in questions you’d never have bothered to answer by hand, and reading can’t break anything.

Why this is a PKM feature and not an AI feature

The agent is only as useful as the vault it reads. A folder of well-linked notes with tasks that carry their context gives it something to work with; a pile of untitled scraps doesn’t. Everything else on this blog, about second brains, PARA and GTD, is about building the vault. The agent is the part that finally makes a large vault feel small.

If you want to try it, the connection needs the desktop app, and Pellet is free to download.

aiagentspkmlocal-first