VLP Case Study

Alex Gill
Aug 4
2 min read

Updated: Aug 4

Does adding more detail to a software prompt help an AI reviewer identify more real code discrepancies? We ran a controlled three-project case study to find out.

The experiment

The same browser Snake product was generated three times from nested specifications: minimal with 8 frozen requirements, moderate with 21 frozen requirements, and detailed with 35 frozen requirements. Tooling, model, generation wrapper, review procedure, and decision rubric stayed fixed; only assigned prompt detail varied.

The result

Minimal: 20 findings, 0 genuine corrections, and 20 rejected concerns — a 0 correction rate.
Moderate: 20 findings, 0 genuine corrections, and 20 rejected concerns — a 0 correction rate.
Detailed: 20 findings, 0 genuine corrections, and 20 rejected concerns — a 0 correction rate.

All 60 observed MonkeyPaw concerns were accepted as compatible with the assigned PRD or as behavior on which that PRD was silent. None met the pre-registered rubric for a genuine code correction.

Coverage still moved

Minimal manual coverage: 87.5% (7 of 8 requirements).
Moderate manual coverage: 95.24% (20 of 21 requirements).
Detailed manual coverage: 97.14% (34 of 35 requirements).

The static manual audit found no unmet or contradictory frozen requirement. One browser-runtime requirement in every tier was not statically verifiable. This audit is distinct from MonkeyPaw corrections and does not override the pre-registered verdict measures.

Pre-registered verdicts

Claim 1: More detail means less discrepancy

Unsupported for this run. Detailed and minimal both had 0 genuine corrections and a 0 correction rate, so detailed was not strictly lower on either required measure.

Claim 2: Minimal prompts create the most review work

Unsupported for this run. Every tier tied at 20 rejected concerns and 0 genuine corrections, so minimal was not the sole leader on either required measure.

The correction-based comparison hit a floor: every tier had zero corrections. Raw finding volume hit a ceiling: MonkeyPaw 0.1.0 has a 20-question cap, and all three reviews saturated it. Equal raw counts therefore do not establish equal latent discrepancy.

What the result means for VLP

The case study suggests a practical distinction for Verifiable Literate Programming: more explicit requirements can improve static coverage without automatically improving reviewer precision. Useful validation needs traceable prompt-to-code evidence and a deterministic decision policy, not finding volume alone.

Limits of the case study

This was one stochastic generation per tier: three projects are a case study, not a statistical benchmark.
MonkeyPaw reviewed snake.js; HTML and CSS obligations were checked only through validation and the static manual audit.
Runtime loading, rendering, controls, timing, collision behavior, and end-to-end playability were not executed.
The results apply only to these prompts, projects, versions, and model configuration; they do not support a population-wide conclusion.