I don’t know how to play an instrument, so obviously I built one as an app. Literally, everyone in my family can sign or play an instrument, and I’m the odd one out.
And I know what you’re thinking, “Who cares? With AI, you can build almost anything.” I’m more excited about the technique I chose to build the app with my agent. Specifically I used context from the agent session that built the app to find and fix the most important gap in its Playwright tests.
Here’s how I did it.
Step 1: Install Entire
Entire captures the prompts, transcripts, tool calls, and decisions behind agent-generated code, seamlessly connecting that underlying context to your Git commits through lightweight checkpoints.
On macOS:
brew tap entireio/tap
brew install --cask entire
Check out these instructions to install on your operating system.
Step 2: Create the project
I created an empty directory (or you can ask your agent to do this)
mkdir music-app
cd music-app
Step 3: Enable Entire
Before handing off any work to the agent, I initialized Entire directly within the repository because I wanted to capture my agent sessions:
entire enable -y
You can also target a specific agent (I personally use Codex):
entire enable -y --agent codex
This sets up the background hooks Entire relies on to capture agent activity, binding that session context directly to the commits generated along the way.
Step 4: Turn the vague idea into a plan
Rather than starting with a rigid technical spec, I simply shared my initial idea:
I'm not entirely sure about the app i want to build..but i want to build a music app that enables me to play instruments even though idk how..this should use computer vision and it should be able to work with real instruments or just like "air" instruments as in there's no instrument there..but i am moving fingers and sounds are being made..and it should like im making real music. idk if this should be sonic pi..but i know i should use media pipe for it. lets start working on a plan together
Working together, the agent helped refine this into an “Air Jam” concept: a browser app where MediaPipe tracks hand gestures, a custom gesture engine interprets them, and Tone.js handles the audio output.
Wanting it to double as a learning tool, I followed up with:
can it still show keys and chords etc..like still be a learning tool in some way
That added a crucial new dimension. Beyond being a fun novelty, the app could now render notes, highlight active scales and chord tones, and eventually break down the music theory behind what was being played.
To lock this down, I asked the agent to document everything:
ok lets put this plan into a markdown file
It generated PRODUCT_PLAN.md, detailing the vision, architecture, development stages, MVP targets, and explicit success criteria.
Phase 1 focused on laying the foundation:
- Bootstrap the browser application.
- Request camera access and show mirrored video.
- Integrate MediaPipe Hand Landmarker.
- Draw hand landmarks and confidence information.
- Define eight virtual strings.
- Detect a fingertip crossing one string.
- Trigger a basic synth note.
- Add filtering, hysteresis, and note cooldowns.
- Measure tracking rate and gesture-to-sound latency.
Most importantly, it defined a clear exit condition: Deliberate movements reliably produce one corresponding sound with few false triggers.
Step 5: Set up atomic agent commits
I prefer keeping commits bite-sized, which is especially helpful when an agent is touching multiple parts of a codebase. To enforce this, I added a repo rule:
also in an agents.md write a rule that says every time we make a change to a file, make a commit
Since I didn't want that configuration change committed just yet, I quickly clarified:
no dont make any commits..just add the note
From then on, the agent created neat, focused commits as features were built. And thanks to Entire, every single commit remained tied to the exact session context that created it.
Step 6: Build Phase 1
With the roadmap in place, we dove straight into implementation:
lets start with phase 1..gesture to sound experiment
The agent systematically assembled the foundation:
- Webcam access
- MediaPipe hand tracking
- Eight virtual strings
- Gesture-to-note conversion
- Tone.js audio
- Velocity-sensitive notes
- Gesture smoothing and cooldowns
- Landmark and string visualizations
- Camera-free note buttons
- Live performance diagnostics
And just like that, it worked. I could wave a finger across a virtual string and play a fully functional air harp.
Step 7: Generate Playwright tests
Before expanding the app, I wanted a solid browser test suite to protect what we’d built. I kept my request broad:
write some playwright tests
The agent generated five passing Playwright tests covering:
- The initial interface
- Camera-free note buttons
- Starting and stopping the synthetic camera
- Camera permission errors
- The mobile layout
On paper, everything was green. But a passing test suite doesn't automatically mean you're testing what actually matters. Because I hadn't written the code or designed the test architecture myself, I wasn't entirely convinced these five tests actually protected the core user experience.
Step 8: Start a new agent session
By this time, my original session had consumed roughly 87 percent of its context window.
I generally avoid pushing agents when their context gets that crowded. It puts them right into what I call the "dumb zone." While the agent technically retains the conversation history, its ability to prioritize key details begins to degrade.
My usual fix is to compact the history or spin up a fresh session. I opted for a fresh start here, not as a staged setup for this article, but as a natural next step after an extended session spent planning, building, debugging, and testing.
Step 9: Compare the tests with the original intent
While the new agent could easily read the code, I wanted it to evaluate the application using the rich context Entire had captured from our earlier planning, implementation, and testing sessions.
I passed it this prompt:
look at the existing Playwright tests and compare to my entire sessions and checkpoints. Do they actually test the main user experience from beginning to end, or do they only test separate pieces of it? Tell me what important behavior is still untested, and show me what you found in the sessions that led you to that conclusion.
The agent pulled up the Playwright suite, cross-referencing it with our earlier Entire checkpoints. Codex summarized its findings:
We tested the camera. We tested the notes. But we never tested that moving your hand actually plays a note.
And there it was. It hadn't actually tested whether waving your hand played a note, which is the fundamental purpose of the app.
Digging deeper into the suite revealed why:
- The camera tests used synthetic video, but the MediaPipe mock always returned zero hands. It proved the video feed could toggle, but never actually passed hand data through the system.
- The note tests triggered audio by clicking UI buttons directly. They proved notes could sound, but skipped testing whether a tracked fingertip crossing a virtual string would trigger one.
Entire provided the agent with concrete historical context to back up this realization:
- The original planning checkpoint explicitly required that deliberate movements produce exactly one sound.
- The implementation checkpoint defined the core feature around that full camera-to-gesture-to-note feedback loop.
- The test-writing checkpoint showed that the initial five tests were narrowly scoped to basic UI, buttons, camera toggles, errors, and layout.
Step 10: Add the missing Playwright test
With the gap exposed, I gave a final instruction:
generate tests for the missing gap
The agent updated the MediaPipe mock to supply realistic 21-point hand coordinates, simulating an index fingertip sweeping across a virtual string.
The new test verified that:
- MediaPipe successfully returned a tracked hand.
- The fingertip crossed the target string.
- The corresponding C4 note activated.
- The note counter incremented exactly once.
- Settling after the gesture produced no duplicate triggers.
- No hidden browser errors occurred during the gesture loop.
That brought our suite from five tests to six. But more than just increasing the count, that sixth test mapped directly back to the success criteria we set during our very first planning session.
What makes this test genuinely better?
It’s easy to fake a bug or throw in a superficial assertion to craft a good story, but that wasn't the case here. This test is fundamentally better because it:
- Protects the primary user journey.
- Exercises the full stack of components end-to-end instead of in isolation.
- Directly reflects our original definition of success.
- Uses deterministic synthetic landmarks so it can run reliably in CI without a physical webcam.
- Ensures strict reliability by validating that one gesture yields exactly one note.
To be clear, it doesn't guarantee that a real-world MediaPipe model will capture every single hand type or that audio physically plays through a user's speakers. Those require dedicated smoke runs or hardware testing. But it does prove that when realistic hand coordinates hit the app, the entire gesture-to-note pipeline executes flawlessly.
I don’t use Entire to save my chat transcript and scroll through later on. Instead, I gave it to my agent to use as the historical evidence it needs to make better engineering decisions.
Please note that I wrote some of this by hand but for much of it I asked my agent to look at my session context and turn it into a blog. Because I have other work to do, and I'm trying to catch up!!
