Running six PDCA (Plan-Design-Do-Check-Act) cycles with Claude Code on a color extraction tool taught me that high alignment rates are often vanity metrics. In my case, half the cycles were essentially wasted because I relied too heavily on whether the code matched the design.
The biggest realization was that alignment and effectiveness are two different axes. In one cycle, gap analysis showed 100% alignment between the design document and the implementation; however, zero cases were actually fixed. The AI followed the design perfectly, but the design failed to solve the problem. Now, I track two separate metrics: Axis A (how well the design was met) and Axis B (whether the hypothesis in the plan actually worked).
I also learned that post-processing cannot fix failures in the upstream process. I spent two cycles trying to fix missing colors by adjusting filters, which failed because the clustering stage wasn't generating those colors in the first place. Now, any plan to modify a filter must first verify that the upstream process is producing the target. Similarly, introducing weights to vivid pixels—intended to improve clustering—actually pulled the center toward outliers, increasing the error in the hardest cases from 20 to 45.
Synthetic data proved to be another trap. While actual images missed colors in eight out of fourteen cases, a synthetic verification tool caught only one out of eight. The synthetic data was too clean, lacking the gradients and compression noise found in reality. I now require a check to ensure synthetic data statistics are within 10% of real-world data before adopting it as an MVP.
Interestingly, not every task requires a design document. For simple UI features with clear requirements in the plan, jumping straight to implementation still yielded 98% alignment. This lesson mirrored a Mac Mini review project where five rounds of AI script audits failed for the same reason: generalizing a single constant without conditions.
Ultimately, PDCA isn't about making the AI write documents; it is a mechanism to force objective judgment. Alignment tells you if the AI followed its own rules, but only reality tells you if it worked.
Originally published at Homelab Notes — notes from one Mac mini running local LLMs and 24/7 automation.
