Your public API documentation contains a test suite you never wrote. Every curl example, every hard-coded response body, and every header shown in your README is an executable claim about how the system should behave, and the ugly part is that those claims rot far faster than the code they describe. A stable endpoint can change a default, tighten an auth rule, or move a parameter, while the friendly snippet on the docs page keeps telling readers the old story. So before a user files a bug about your docs, you can point a free model at that documentation, ask it to turn only the concrete request examples into a tiny JSON test plan, and replay that plan against a free server to see which examples are still true.
That is the whole idea in one sentence: treat your own docs as a low-effort regression suite and let a free model do the tedious extraction. This article uses MonkeyCode's free model access and free server option as its product context. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am not assuming a particular model name, quota, runtime, or benchmark result here; the only product facts I rely on are the two in that sentence, and the workflow remains useful with any model that can return strict JSON.
Why a model at all? Because documentation is written for humans, not for a test runner. The interesting examples are sprinkled across prose, tabs, shell variables, and half-finished snippets, so a regular expression alone misses most of them. A model can read a paragraph like If you are using the hosted API, issue a GET to /v1/projects and pass your token in the Authorization header; the response includes a projects array and reduce it to the one machine-checkable fact hiding inside it: method, path, headers, expected status, and a string to look for in the response. That extraction is not a clever AI task; it is a structured-reading task, which is exactly where a free model is most useful.
The artifact I use is intentionally boring. A small Python script pulls the relevant documentation text, sends it to the model with a strict prompt, and stores each returned object as a case. The prompt matters more than the model because you are asking for a very constrained contract and rejecting anything outside it:
PROMPT = '''Read the following API documentation and return a JSON array.
For each concrete request example, include exactly these keys:
name, method, path, headers, body (or null),
expected_status, expected_contains (or null).
Return only the JSON array. If a request is not concrete,
skip it. If a value is a placeholder, return null for that key.
'''
def call_model(docs):
# Use any chat client that returns text. The contract is:
# PROMPT + documentation text in, JSON array out.
# Keep temperature low and validate the JSON before use.
...
The prompt says to skip placeholders rather than guess them, and that one instruction prevents most of the damage. A hallucinated endpoint is annoying, but an endpoint like /v1/<YOUR_TOKEN>/users is worse because it looks almost legitimate. So the runner rejects any case with nulls in critical fields and prints a warning instead of sending a doomed request.
The replay half is plain HTTP. You loop over the extracted cases, build a request with the standard library, and compare the real response to the expected status and the expected body fragment. If the docs say 200 and the server returns 404, or the familiar projects key is gone, the case fails and you have a concrete documentation bug to fix. You can point this at 127.0.0.1:8000 while developing, but the free server option is a better fit because it lets you run the same suite against a throwaway deployment without keeping a local service alive.
def replay(cases, base_url):
for case in cases:
req = urllib.request.Request(
base_url + case['path'],
method=case['method'],
headers=case.get('headers') or {},
)
if case.get('body') is not None:
req.data = json.dumps(case['body']).encode()
try:
with urllib.request.urlopen(req, timeout=5) as resp:
body = resp.read().decode()
ok = resp.status == case.get('expected_status', 200)
ok = ok and (case.get('expected_contains') or '') in body
print('PASS' if ok else 'FAIL', case['name'])
except Exception as exc:
print('FAIL', case['name'], exc)
The most common failure mode is not a wild hallucination; it is a docs example that was already wrong three releases ago. You might have a GET endpoint that used to require an api_key query parameter, and the README still shows it. The model extracts that faithfully, the runner sends the old parameter, the server ignores it, and the request still returns 200 because the endpoint was made more permissive. A stricter check would compare the request the server actually recognized, but that is the point: this workflow is a cheap smoke signal, not a full contract validator. If you need formal guarantees, maintain an OpenAPI spec and derive tests from that instead; if you need a fast way to notice when your human-readable docs drift away from reality, the doc extractor is enough.
This approach is genuinely useful when the documentation contains many concrete examples: a public API with a getting-started page, a webhook guide with payload samples, or an internal service with a README written by the person who no longer owns it. It is less useful when your docs are sparse, automatically generated, or written as broad prose without a single executable line. In those cases the model has nothing to extract, and running the script will only tell you what you already know: the documentation is thin.
You should not use this pattern against production for write-heavy examples. The script follows whatever method the docs show, so a POST that creates a real invoice will create a real invoice if you point it at the wrong base URL. Keep it read-only, run it against a sandbox or the free server option, and review the extracted cases before they touch the network. A human approving twenty lines of JSON is fast; undoing a batch of test records in a shared database is not.
The nice thing about holding the model to a strict JSON schema is that you do not have to trust it blindly. Anything outside the schema gets rejected, any placeholder gets skipped, and any write operation should be blocked by a short allowed-methods check before replay. The model is doing the boring part of reading docs; the test runner is doing the actual verification. If you already have a free model endpoint and a free server option available, try wiring this extractor into a pre-release checklist. The cheapest bug to fix is the one your own documentation caught before your users did.
