readme: lead with the runner; the primitives follow, under going it alone.
commit c290ea86e2f4501c45aaae647caed08db3fe155e · parent 8dfec62210
· browse this tree
Ximianth, 2026-10-24 15:04:53 UTC
diff --git a/README.md b/README.md index 998ef26..43ae9b3 100644 --- a/README.md +++ b/README.md @@ -18,11 +18,19 @@ which makes it easy to update fixtures when output changes on purpose. ```bash source path/to/lib/testmark.bash -testmark_parse doc fixtures.md -testmark_check doc greeting/output < <(mytool greet world) -testmark_write doc +exercise_mytool() { + mytool --read-stdin +} + +testmarktest_rule_basic '*/input' exercise_mytool +testmarktest_run fixtures.md ``` +That's a whole test file. +Every hunk in `fixtures.md` named `SOMETHING/input` is a case: +it's fed to `exercise_mytool`, and what that prints is checked against the hunk named `SOMETHING/output`. +Adding a case is adding two blocks to the doc. + It needs bash 4.4 or later. @@ -52,6 +60,50 @@ Later, when the tool's output changes on purpose, it's the same: regen, read the The diff is the review. +Writing a test +-------------- + +A test file makes rules, then runs them over its docs. + +**A rule** is a glob and a function: `testmarktest_rule_basic GLOB FN`. +Every hunk whose name matches GLOB triggers a case, and FN exercises it. +The case is the hunk's name minus its last segment, so `greeting/input` is the case `greeting`, and its output goes in `greeting/output`. +In the glob, `*` matches `/` too, so `'*/input'` finds `a/b/input` as well. + +**The function** is where your tool gets run. +We call ours `exercise_SOMETHING`, by convention. +Each case runs it in a fresh bash, so it's a clean shell: + +- stdin is the trigger hunk's body, exactly. +- `set -euo pipefail` is on. A failing command stops the case, and the failure shows up in its output. +- The working directory is the test script's, so a relative path like `./mytool` works, wherever the test is run from. +- It sees exported variables (like `PATH`), but no other variables, and no other functions. + If it calls helpers, pass them to `testmarktest_provide helper1 helper2` first. + +**What gets checked** is everything the function prints, stdout and stderr together, in order. +Two things get added, only when they're needed, so the fixture for a tool that works is just its output: + +- If the function exits nonzero, a last line `exit status: N`. A case can't pass while its tool fails, unless the fixture says it should. +- If the output doesn't end in a newline, it gets one, then a line `\ No newline at end of output`. + +Output with CRLF line endings fails the case; normalize it in the function, with `tr -d '\r'`. + +**`testmarktest_run DOC...`** runs every case, prints a `PASS` or `FAIL` line for each, and writes the docs if `TESTMARK_REGEN` is set. +Doc paths are relative to the test script. +It fails if any case failed, and also if a rule matched no hunk at all, or a doc matched no rule: +a typo in a glob can't pass by running nothing. + +A doc can hold several kinds of cases. Give each kind its own rule: + +```bash +testmarktest_rule_basic '*/script' exercise_script +testmarktest_rule_basic '*/input' exercise_mytool +``` + +The output hunk must exist already, even if empty: regen fills in a block that's there, and a missing one fails the case. +That's why the workflow starts with an empty block. + + A whole example --------------- @@ -61,69 +113,47 @@ A whole example #!/bin/bash ## Checks todos.sh against every case in todos.md. ## Run with TESTMARK_REGEN=1 to update todos.md to match what todos.sh does now; then review the change with `git diff`. -set -euo pipefail -cd "$(dirname "$0")" -source ../lib/testmark.bash - -work=$(mktemp -d) -trap 'rm -rf "$work"' EXIT - -testmark_parse doc todos.md -mapfile -t names < <(testmark_names doc) -fails=0 -for name in "${names[@]}"; do - case $name in - */input) - case=${name%/input} - testmark_body doc "$name" >"$work/notes.txt" - if testmark_check doc "$case/output" < <(./todos.sh "$work/notes.txt" | sed "s|$work|WORK|g"); then - echo "PASS $case" - else - echo "FAIL $case" - fails=$((fails + 1)) - fi - ;; - esac -done -testmark_write doc -((fails == 0)) +source "$(dirname "$0")/../lib/testmark.bash" + +exercise_todos() { + ./todos.sh /dev/stdin +} + +testmarktest_rule_basic '*/input' exercise_todos +testmarktest_run todos.md ``` -What to copy from it: +**Normalize, then check exactly.** +If your tool's output has anything that changes from run to run, like a temp dir or a timing, replace it in the function: -- **One case per name.** Hunks named `CASE/input` and `CASE/output` make a table; - adding a case is adding two blocks to the doc, with no change to the test. -- **Normalize, then check exactly.** Anything that changes from run to run, like a temp dir, gets replaced before the check (here, with `sed`), - so the check can stay exact. - Prefer this over grepping for a fragment of the output: - an exact check can't pass by accident, it shows the reader what the output really looks like, and regen can write it. -- **`< <(...)` into the check.** Not a pipe; see the rules below. -- **`testmark_write` at the end.** That's what makes `TESTMARK_REGEN=1` write anything. - With nothing to write, it does nothing, so call it unconditionally. +```bash +exercise_build() { + mytool build | sed 's/in [0-9.]*s$/in Ns/' +} +``` -A check doesn't see the tool's exit status. -When it matters, put it in the output: `< <(mytool || echo "exit: $?")` adds a line only when the tool fails. +Then the check can stay exact. +Prefer this over grepping for a fragment of the output: +an exact check can't pass by accident, it shows the reader what the output really looks like, and regen can write it. -Rules to know -------------- +Functions +--------- -**State changes happen in the shell that parsed.** -`testmark_check`, `testmark_patch`, and `testmark_write` refuse to run in a subshell, where what they stage would be lost. -So feed a check with `testmark_check doc NAME < <(cmd)`, not `cmd | testmark_check doc NAME`; -and don't call these inside `$(...)`, or inside a test helper that runs its command there. +| function | does | +| --- | --- | +| `testmarktest_rule_basic GLOB FN` | makes each hunk matching GLOB a case, exercised by FN, with the hunk on stdin, and checked against the case's `output` hunk | +| `testmarktest_provide FN...` | sends more functions into every case, for exercise functions that call them | +| `testmarktest_run DOC...` | runs every case in the docs, reports each, and writes the docs under `TESTMARK_REGEN` | -**Output must be empty, or end in a newline, with no CRLF.** -A hunk's body can't hold anything else, so `testmark_check` rejects it outright, rather than failing forever. -Normalize it first. +Errors come back as a nonzero status, with a `testmark: ...` line on stderr. +The doc comments in [`lib/testmark.bash`](lib/testmark.bash) are the full reference. -**A check never creates a hunk.** -Regen rewrites the body of a hunk that's there; a missing one is an error in both modes. -That's why the workflow starts with an empty block. +Going it alone +-------------- -Functions ---------- +The runner is built on primitives you can call yourself, when you want something it doesn't do: | function | does | | --- | --- | @@ -135,8 +165,25 @@ Functions | `testmark_patch HANDLE NAME [INFO] <BODY` | stages a new body, and optionally a new info string | | `testmark_write HANDLE` | writes what's staged into the file, in place | -Errors come back as a nonzero status, with a `testmark: ...` line on stderr. -The doc comments in [`lib/testmark.bash`](lib/testmark.bash) are the full reference. +[`examples/test-todos-alone.sh`](examples/test-todos-alone.sh) does what the runner does for `todos.md`, by hand. +With the primitives, these are yours to get right: + +- **`testmark_write` at the end.** That's what makes `TESTMARK_REGEN=1` write anything. + Without it, regen stages everything, then throws it away, without a word. +- **Counting failures.** Run every case, and exit nonzero if any failed. +- **The exit status.** A check doesn't see it. Put it in the output: `< <(mytool || echo "exit: $?")` adds a line only when the tool fails. + +**State changes happen in the shell that parsed.** +`testmark_check`, `testmark_patch`, and `testmark_write` refuse to run in a subshell, where what they stage would be lost. +So feed a check with `testmark_check doc NAME < <(cmd)`, not `cmd | testmark_check doc NAME`; +and don't call these inside `$(...)`, or inside a test helper that runs its command there. + +**Output must be empty, or end in a newline, with no CRLF.** +A hunk's body can't hold anything else, so `testmark_check` rejects it outright, rather than failing forever. +Normalize it first. + +**A check never creates a hunk.** +Regen rewrites the body of a hunk that's there; a missing one is an error in both modes. Requirements