Living Report

Practical engineering · Original scripted fixtures · Product experiments not performed

An agent wrote tests: do they detect the defect?

Download run_demos.py and contracts.json into one folder before running these commands. Methodology and limitations.

Status: finished instructional workflow with an executed scripted synthetic demonstration. No coding-product experiment has run. The weak test below was deliberately authored for this fixture; no claim is made that a product wrote it.

Use this when accepting a small agent patch whose tests appear green. The next step is to challenge those tests with a defect that violates a separately stated requirement. A passing command alone cannot establish that the tests detect it.

Contract before implementation

For a subtotal expressed in integer cents, shipping costs 500 cents below 10000, and zero at or above 10000. The independent cases in fixtures/contracts.json include 0, 9999, 10000, 10001 and 25000 cents. Inputs outside this stated domain are not studied here. Preserve that contract before asking a tool for code.

The intentionally flawed implementation uses subtotal > 10000. The weak test checks only a subtotal of 12000. That test passes while the boundary is wrong.

Reproduce

Requirements: Python 3.11+; no package install or model/API call. From a checkout:

python3 run_demos.py tests --output /tmp/lr-tests-demo

Choose a new output directory each time; the script refuses to overwrite one. Read result.json, baseline.py, shipping.py and repair.diff in that directory. The script executes a predeclared repair; it is not an autonomous bugfix.

The recorded run observed the weak test passing on the defect. The separate oracle rejected the 10000-cent case: actual shipping 500, expected zero. After the scripted > to >= repair, all five contract cases passed. The contract's SHA-256 is recorded with the result, so a later rewritten oracle is detectable.

Apply it to your agent task

  1. State observable requirements and a few counterexamples before the run.
  2. Keep the acceptance oracle outside the tool's authorized edit set. Give the tool task context and ordinary development tests, not a writable acceptance gate.
  3. Save the baseline, prompt, model/tool version, patch and complete test output.
  4. Run the oracle against the defective baseline, then the proposed patch. A red baseline and green patch are useful evidence for the cases actually covered.
  5. Inspect whether tests changed the requirement, deleted a failing check, mocked away the defect, or asserted only that a function returns a value.
  6. Reject or investigate a patch if the oracle never detected the known defect. Record interventions and failed attempts, including abandoned runs.

Acceptance and limits

Accept this demonstration only if the known boundary fails before the repair, passes after it, and the oracle bytes remain unchanged. Five cases establish a bounded behavior check, not general correctness, security or product quality.

No subscription comparison, tokens or model time were measured. API spend for the scripted demonstration is zero because no API is called. Human review time is unknown. Before a real tool comparison, use the separate experiment protocol; do not turn this fixture result into a recommendation for a coding product.