The Model May Draft the Happy Path; Humans Own Every Absence Claim
A model drafting documentation from a repository describes what exists far better than what does not, because the code is a positive artifact: routes, flags, and types are visible, while "this endpoint rejects batched writes" exists only in someone's head. That asymmetry is the practical failure you have to design around, and it does not go away with a bigger model or a longer prompt. The control that works in my experience is narrow and mechanical: let the model draft the happy path freely, then force every absence, limit, and guarantee claim into a review queue where a named human signs it. This article describes that two-layer control, gives an illustrative scanner you can adapt, and states plainly where the approach breaks down. The script below was written for this article and has not been executed against a real docs tree, so treat the pattern list as a starting point rather than a validated classifier. Why negative claims fail differently Positive claims about an AP
A model drafting documentation from a repository describes what exists far better than what does not, because the code is a positive artifact: routes, flags, and types are visible, while "this endpoint rejects batched writes" exists only in someone's head. That asymmetry is the practical failure you have to design around, and it does not go away with a bigger model or a longer prompt. The control that works in my experience is narrow and mechanical: let the model draft the happy path freely, then force every absence, limit, and guarantee claim into a review queue where a named human signs it.
This article describes that two-layer control, gives an illustrative scanner you can adapt, and states plainly where the approach breaks down. The script below was written for this article and has not been executed against a real docs tree, so treat the pattern list as a starting point rather than a validated classifier.
Why negative claims fail differently
Positive claims about an API surface degrade gracefully: a stale field name is visibly wrong to anyone who opens the source file. Negative claims degrade silently, because nothing in the codebase contradicts a sentence like "the worker does not retry on 5xx" until an incident proves it. A model asked to be helpful will also fill gaps in a plausible direction, adding cautions that sound responsible but were never verified. Those invented caveats are worse than missing ones, since a reader may plan an architecture around a limitation that does not exist.
The categories that need human ownership are small but high-leverage: absences ("does not", "unsupported"), scope limits ("only works with", "at most"), deprecation timelines, and behavioral guarantees ("idempotent", "thread-safe", "atomic"). Everything else, including most of the prose, examples, and parameter walkthroughs, can stay model-drafted with normal review.
Step 1: Write a drafting contract the model must follow
Before any prose generation, give the model an explicit ownership contract and require it to flag the sentences it cannot ground. The contract below is illustrative text you can paste into a system prompt; the key part is that the model must emit a placeholder rather than a confident sentence.
You may draft: summaries, parameter descriptions, example requests,
and navigation prose. Ground each factual statement in a file you
can cite.
You may NOT finalize any sentence that asserts:
- an absence ("does not", "unsupported", "no support for")
- a scope limit ("only works", "requires", "at most")
- a deprecation ("deprecated", "removed in")
- a guarantee ("idempotent", "thread-safe", "atomic")
For those, emit the sentence followed by:
Do not guess a name and do not remove the placeholder.
The placeholder is deliberately invalid, so it can never pass a merge check, and it is also greppable. Models under-report their own uncertainty, however, so the prompt layer is a convenience rather than a guarantee. That is why the second layer is mechanical.
Step 2: Scan the draft for claims the model may not own
The scanner ignores what the model claimed about itself and looks only at the text. It skips fenced code blocks and inline code so that flags like --no-cache do not register as absence claims, then classifies each remaining sentence into the four categories.
#!/usr/bin/env python3
"""negclaim.py - emit a review queue of claims a model may not own.
Illustrative: written for this article, not validated against a corpus.
"""
import json
import re
import sys
from pathlib import Path
PATTERNS = [
("absence", r"\b(does not|doesn't|do not|don't|not supported|unsupported|cannot|can't|no support for)\b"),
("limit", r"\b(only works|requires?|at most|up to|limited to|maximum of|no more than)\b"),
("deprecat", r"\b(deprecated|removed in|will be removed|sunset|end[- ]of[- ]life)\b"),
("guarantee", r"\b(thread[- ]safe|idempotent|guaranteed|atomic|safe to|always returns)\b"),
]
COMPILED = [(name, re.compile(rx, re.IGNORECASE)) for name, rx in PATTERNS]
SIGN = re.compile(r"", re.IGNORECASE)
FENCE = re.compile(r"^(```
|~~~)")
INLINE_CODE = re.compile(r"`[^`]*`")
def iter_sentences(path):
in_fence = False
for lineno, raw in enumerate(path.read_text(encoding="utf-8").splitlines(), 1):
if FENCE.match(raw.strip()):
in_fence = not in_fence
continue
if in_fence or not raw.strip():
continue
for sent in re.split(r"(?<=[.!?])\s+", INLINE_CODE.sub("CODE", raw)):
if sent.strip():
yield lineno, sent.strip()
def main(argv):
findings = []
for target in argv:
for lineno, sent in iter_sentences(Path(target)):
hits = [name for name, rx in COMPILED if rx.search(sent)]
if not hits:
continue
sign = SIGN.search(sent)
findings.append({
"file": target, "line": lineno, "categories": hits,
"text": sent,
"signed_by": sign.group(1) if sign else None,
"signed_on": sign.group(2) if sign else None,
})
for f in findings:
print(json.dumps(f, ensure_ascii=False))
unsigned = [f for f in findings if not f["signed_by"] or f["signed_by"] == "UNASSIGNED"]
print(f"# {len(findings)} claims, {len(unsigned)} unsigned", file=sys.stderr)
return 1 if unsigned else 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
```
A signed sentence looks like this in the source Markdown, and the signature is intentionally human-readable rather than cryptographic:
``{% endraw %}{% raw %}`markdown
The batch endpoint does not accept more than 100 records per request.
<!-- own: avery.lin 2026-11-09 -->
```
Using a name and a date instead of a content digest is a tradeoff worth naming: editing a signed sentence without re-reviewing it will not invalidate the signature automatically. If you need that property, hash the sentence and compare on every run, at the cost of re-signing after every style edit.
## Step 3: Put the queue in CI and keep it cheap
Wire the scanner into the pull request job so that an unsigned claim fails the build rather than relying on reviewer attention.
``{% endraw %}{% raw %}`yaml
- name: Negative-claim review queue
run: python3 tools/negclaim.py docs/*.md > negclaims.jsonl
```
Draft regeneration and the scan are the two places where free tier resources change the loop's economics rather than its correctness. According to the operator, MonkeyCode offers free model access and a free server option, and that combination lets you re-draft a page several times per change and still run the scanner as a small hosted job. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I have not benchmarked either offering, so verify throughput, limits, and suitability for your repository before depending on them in a required check.
## Step 4: Decide each queued claim explicitly
Reviewers should not rewrite the queue from scratch; they should answer four questions per item and either sign, correct, or delete the sentence.
1. Does the repository or a linked primary source actually support the claim, or is it plausible interpolation?
2. Is the scope precise enough to survive a reader's worst-case interpretation?
3. Does the claim belong in this page at all, or should it live in a changelog, migration guide, or code comment?
4. Who is accountable if the claim turns out to be false, and is that person signing it today?
The fourth question is the one that matters, because a name with a date turns an anonymous prose risk into a specific commitment.
| Claim type | Model may draft | Human must own |
|---|---|---|
| Feature description | Yes, with a citable source | Spot review |
| Example request/response | Yes | Verify the example runs |
| Absence or non-support | No | Named signer, dated |
| Scope limit or precondition | Sentence only | Named signer, dated |
| Deprecation timeline | No | Owner plus release reference |
| Concurrency or safety guarantee | No | Named signer after testing |
## Limitations and who should not use this
The scanner is a regex heuristic, so it over-fires on ordinary prose and misses claims phrased outside its vocabulary. Teams whose docs are generated from an OpenAPI schema or a typed config often already encode absences in machine-readable form, and adding a second prose-level queue just creates duplicate review work. Single-page projects with one maintainer get little from a signature ritual, and teams without a culture of blocking merges should fix that first. This control also does nothing for positive claims that are wrong, since it only protects the sentences people rarely check.
## How to check whether the queue earns its keep
Before adopting the check as required, sample about fifty flagged sentences from your own docs and label each as a true claim or noise by hand. Compute precision per category and drop any pattern family that mostly produces noise, since a queue reviewers learn to ignore is worse than no queue. Then run the scanner against a branch where you deliberately added one false absence claim and confirm CI fails, which costs a few minutes and settles whether the wiring works. Record the date and the sample size next to the pattern list so the next maintainer knows how old the calibration is.
If you run this against a real docs tree, the most useful feedback is which categories over-fired and which absence claims the patterns missed entirely, because that list is what makes the second layer worth trusting.
