A pentester on a whitebox engagement with three days left and forty thousand lines of unfamiliar code pastes a batch of files into an LLM overnight to speed up review. By morning there's a ranked list: twelve "likely vulnerable" functions, one flagged in bold as SSRF via a URL parameter passed straight into an internal HTTP client. It reads like a finding. It's already in the report draft. The only problem is it isn't true: the parameter gets checked against an allowlist two functions earlier, in a file the model never saw because it was in a different batch.
That's the actual risk with AI in a pentest workflow, and it isn't "the model will replace testers." It's that a model reviewing code in isolated chunks has no persistent view of the whole call graph, and it still answers with full confidence, because confidence is a property of how it writes, not of whether it checked. Ask it whether a parameter is sanitized upstream and it will guess plausibly instead of admitting it can't see the rest of the file tree. That guess reads exactly like a confirmed finding unless you already know to distrust the tone.
The workflow that actually holds up treats the model as a hypothesis generator, never as the tester of record. Feed it the high-volume, mechanical parts: parsing a large gobuster or Burp export, summarizing an unfamiliar framework's auth middleware, drafting the first pass of a report section. Let it flag candidates fast across more surface area than a human could read cold in the same time. But every flagged item stays unconfirmed until you've done the part the model can't: traced the actual call path yourself, captured a real request and response that proves the behavior, or reproduced it in a lab. A finding that hasn't been independently reproduced doesn't go in the report, however well the model explained it.
The same rule applies to the specifics models like to supply on their own: CVE IDs, version strings, CVSS scores. All three get invented convincingly often enough that treating an unverified one as fact is how a report ends up citing an advisory that doesn't exist. Check it against the real advisory or the real banner, every time, before it's in writing.
None of this makes AI useless in a pentest. It makes it exactly as useful as a very fast, very confident junior who hasn't learned yet that "I think so" and "I checked" are different sentences. The AI for Penetration Testing book is built around that distinction: 312 pages of ethics-first red team workflows covering where AI genuinely speeds up recon, code review and reporting, and where a human still has to verify before it's real: https://resources.codelivly.com/product/ai-for-hackers-red-team-edition/
