The One Thing That Makes AI Coding Actually Work
Last Tuesday, I wasted four hours chasing a bug that didn’t exist. The task seemed trivial: add a last_login_ip field to our user profile API response. It’s a simple data plumbing job, perfect for an AI coding agent. I gave it the product requirements doc, pointed it at the user_service.py and the corresponding UserProfile Pydantic model, and let it run.
Twenty minutes later, a pull request appeared. The code was clean. It fetched the IP from the right table, added the last_login_ip: Optional[str] field to the model, and even included a new unit test in test_user_service.py to confirm the field was present and correctly formatted in the API response. All checks were green. I gave it a quick once-over, saw the new test passing, and merged it. Job done. Or so I thought.
An hour later, our monitoring alerts for the Android app started screaming. A spike in 500 Internal Server Error responses, but only for users on app version 3.2 or older. My first thought was a bad deployment. I frantically rolled back the change. The errors stopped. Okay, so it was my change. But why? The API response was perfectly valid according to its own contract. The unit test proved it. It took me and another engineer the rest of the afternoon to piece it together. The older Android clients used a hand-rolled parsing library (don’t ask) that was brutally strict. It expected a fixed set of keys in the user profile response. When our API started sending a new, unexpected last_login_ip key, the client-side parser choked and threw a fatal error, which our backend was logging as a generic 500. The agent’s code was 100% correct, but the solution was 100% wrong.
The Plausible Fallacy
My four-hour wild goose chase is a perfect example of what I’m starting to call The Plausible Fallacy. AI-generated code is often locally correct - it passes its own tests and fulfills the explicit instructions - but it is globally, systemically ignorant. The agent did exactly what I asked, but it had no concept of the unstated contract with a legacy mobile client. It delivered a perfect answer to the wrong question.
This isn’t just a me-problem; it’s a universal groan echoing across the industry. I was scrolling through a Reddit thread on r/ExperiencedDevs last week titled “Is anyone else getting tired of babysitting their AI?” and the top comment was a litany of these exact failures. One engineer spent a day debugging why a new API endpoint was slow, only to find the agent-generated code was making N+1 database queries inside a loop. The code worked, it just didn’t work well. It was plausibly correct, but architecturally naive. Another comment with 40+ replies detailed a similar story where an agent “helpfully” refactored a function, changing its signature and breaking a dozen downstream consumers that weren’t in the provided context.
The theme is always the same: we are drowning in plausible but incomplete work. As one person on a Hacker News thread put it, “The agent is a savant at the micro level and a toddler at the macro level.” We are shipping code that passes the test, but fails the business.
Why This Keeps Happening
The root cause of my Android app fiasco wasn’t the model’s intelligence or lack thereof. It’s not about which agent I used, what prompts I wrote, or whether I paid $20 a month or $500. The problem is structural. AI coding agents, in their current form, are built to do one thing: satisfy the immediate prompt.
They are local optimization machines. Their entire world is the code and context you provide in a single session. They have no durable, persistent understanding of your system’s architecture, its implicit contracts, or its non-functional requirements. They see the file, not the system. They see the function, not the fallout.
It’s not that the models are bad. They are astonishingly good at code generation. The problem is that we are asking them to perform the duties of a senior engineer - who holds a complex mental model of the entire system in their head - while only giving them the information of a junior intern on their first day. The agent didn’t fail me; my process did. I asked it to write a chapter without giving it the outline of the book.
Why Prompting Won’t Fix It
I can already hear the objections. “Julian, you just needed a better prompt! You should have told it, ‘Add the IP field, but be EXTREMELY careful to consider all downstream consumers, including legacy mobile clients that might have strict parsing.’”
Honestly, this is a dead end. We are trying to solve a systemic problem with a conversational patch. It’s like trying to enforce a building’s fire code by shouting “remember the sprinklers!” at the construction workers from the street. You might get a sprinkler in that one room, that one time, but you haven’t fixed the process.
Trying to guarantee systemic correctness through prompting is like fighting gravity. You can win for a moment with a heroic, perfectly crafted prompt, but gravity - the agent’s innate tendency to solve the most local, immediate problem - always wins in the end. A prompt is an ephemeral suggestion. It is not an enforceable, durable contract. Relying on it for correctness means you are still the system of record. You are still the one holding the entire architectural model in your head, with the added cognitive load of trying to translate it into the right magic words every single time. This isn’t automation; it’s just a new and more elaborate way to be the bottleneck.
Tired: “My AI needs to be smart enough to understand my whole system.” Wired: “I need a system that enforces my architecture on my AI.”
The Fix
The solution to this is embarrassingly simple, and frankly, a little bit boring. It isn’t a more powerful model, a clever new prompting technique, or a fancier VS Code extension.
The one thing that makes AI coding actually work is external verification.
That’s it. A system, separate from the agent, that holds the ground truth of the requirements and programmatically enforces them. A system that doesn’t care how clever the agent is or how clean the code looks. It only cares about one thing: is there objective, verifiable proof that all requirements - functional and non-functional - have been met?
Everything else is an optimization. The model you use, the tool, the prompt… those are just ways to speed up the generation of the candidate solution. But the generation is not the work. The verification is the work. Without a system of enforcement, you’re just generating plausible problems faster than ever before.
What This Looks Like in Practice
This is the entire premise of what we’re building at Ceetrix. We are not building another agent. We are building the external verification system that makes any agent safe and reliable to use for professional software development.
If I had run my last_login_ip task through Ceetrix, the story would have been a non-event. The PRD, which lives in our Document Editor, would have included the requirement: “The change must be backward compatible with mobile client v3.2.” Using Spec Chain Enforcement, that specific requirement would be immutably linked to the design, the implementation tasks, and the test tasks. It’s an unbreakable chain of accountability.
When my agent submitted its “perfect” PR, it would have slammed headfirst into our Gate System (G0-G12). Gate G9, our automated Coverage Checking gate, would have cross-referenced the work against the spec chain. Our Coverage Gap Visibility would light up, showing that while the implementation task was done, there was no proof that the backward-compatibility requirement had been met. The task would be blocked. Cold.
The system’s analysis of the feature’s Impact Dimensions would have flagged the change for its high user_proximity and potential impact on reversibility, automatically creating mandatory Test Task Types. It would have required a new e2e test task to be created and completed, specifically simulating a login from an older client. To complete its task, the agent has to provide Task Completion Evidence - not just code, but a rationale and artifacts proving the test passed. There is a QA Step in Workflows that can’t be skipped. There is even Exit Gate Enforcement that would prevent me from just abandoning the half-finished work.
My four-hour debugging nightmare would have been replaced by a two-minute automated gate failure notification, pointing out the exact requirement that was missing proof of completion. That’s the difference between generating code and engineering a solution.
Have your say: What’s the most painful “plausibly correct, systemically wrong” bug an AI has introduced into your codebase? I’d love to hear the war stories. And when you’re ready to build on a foundation of verification, try Ceetrix.