I've spent two months building Deiko, a Mac app for talking to coding agents, and today it's open source under MIT. Here's what it does and how the interesting parts work.
The two problems
Coding agents can't see what you're looking at. So you type a paragraph describing a screen they never saw, take a screenshot, attach it, then explain which part of it you meant.
And they forget. Every new chat with Claude Code, Cursor or Codex starts from zero, so on Friday you explain Tuesday's bug again, including the fix that already didn't work.
Showing instead of typing
Double-tap Right Option and talk. While you talk, you point at things with the cursor, or hold Left Option and draw around a region. Deiko records your voice, crops the screen around each point, reads the accessibility label under the cursor (a button's title, a cell's value) and notes the window and page titles.
When you stop, a small Node pipeline turns that into a brief. The interesting step is alignment: each word you said is matched to what you were pointing at when you said it, so "this price" and "that button" resolve to the right crop. Secrets are redacted before anything is written, and if a credential is visible in a screenshot, that screenshot is withheld and the brief says so.
A coin appears. Drop it on your agent's window and the brief is pasted and sent.
Remembering, without a vector database in the cloud
Every brief gets filed into the piece of work it continues, or starts a new one. Filing runs in two steps:
- On your Mac: a keyword score (BM25) over each task's text, blended with a meaning score from EmbeddingGemma running on-device through ONNX. That gives a shortlist.
- A small judge: the shortlist, with each task's title and recent summaries, goes to Jev, a small model that answers whether the brief is the same work, points back to earlier work ("remember the pricing fix? in that task…"), or is new.
Clear matches join, close calls ask "Same work?" on the card, and nothing you placed by hand is ever moved. The next brief on that task carries a note of where it stands and what was decided.
Agents that report back
Deiko ships a local MCP server with five tools: search_briefs, list_tasks, get_task, get_brief and save_outcome. Agents can look up earlier work when they need it, and when they finish a brief they save what they did, decided and left open, which feeds the task's note.
Agents don't always remember to save, so for Claude Code there's a Stop hook: if the agent is about to finish the turn a brief arrived in without saving, it's asked once to do so. It never asks twice in a row, and it stays quiet in chats that aren't a Deiko brief.
Privacy you can check
Screenshots, OCR and accessibility text never leave the Mac. Your voice goes to Whisper (with your own Groq key, through Deiko's relay, or not at all with Apple's on-device recogniser), and filing sends titles and summaries to Jev through the relay, which keeps nothing. Until today you had to take my word for that. Now the relay's code is in the repo.
What's in the repo
-
apps/macos: the Swift menu-bar app -
packages/core: the Node pipeline and the MCP server -
packages/alignment: word-to-pointer alignment (TypeScript) -
services/relay: the relay (AWS Lambda) -
evals: the filing, search and memory evals I use to check changes
Try it
curl -fsSL https://deiko.app/install.sh | sh
macOS 14+ on Apple silicon. Free, with two hours of hosted transcription a month; your own Groq key or on-device transcription works with no limit.
Repo: https://github.com/maddy30445r/deiko
There are a few good first issues open for Hacktoberfest (Zed support, README translations, a Homebrew tap). Issues and feedback are very welcome, especially about where the gesture gets in your way.
Security note: it asks for Accessibility, Screen Recording and Microphone access and captures only while a session runs (a red bar shows). Screenshots never leave your Mac. It isn't notarized by Apple yet, so read the install script first: https://github.com/maddy30445r/deiko/blob/main/scripts/install.sh
