Merge branch 'high-road' into usable. Bash has been a mistake. The fraction of this branch that has been attempts to make error handling comprehensible beggars belief.
commit 90280a0b4247c516c4f83c30db3be6570aac64cc · parent 423732f80f · parent 29c582aab1
· browse this tree
warpfork, 2026-10-25 11:38:12 UTC
I still want to attempt to finish out what I originally envisioned for this project, but I no longer believe bash should be used. Ever. This has been *literally* maddening. I am *less sane* than when this project started. Learning all the ways in which bash absolutely fucks with its own error handling has given me so many new gray hairs and new doubts about how insane software can be that it's moved my desire to retire forward by about 4 decades. I want to scream. I want to report bugs. But this just... is bash. Someone thought any of this made sense. I... I lack words. I still contain anger. And disappointment. But I lack enough words.
diff --git a/Makefile b/Makefile index 72d5216..cf4999f 100644 --- a/Makefile +++ b/Makefile @@ -30,3 +30,4 @@ check: ## Verify-only and fast; run by the pre-commit hook on every commit. test: ## Run all tests. tests/run.sh examples/test-todos.sh + examples/test-todos-alone.sh diff --git a/README.md b/README.md index 998ef26..435921a 100644 --- a/README.md +++ b/README.md @@ -18,11 +18,19 @@ which makes it easy to update fixtures when output changes on purpose. ```bash source path/to/lib/testmark.bash -testmark_parse doc fixtures.md -testmark_check doc greeting/output < <(mytool greet world) -testmark_write doc +exercise_mytool() { + mytool --read-stdin +} + +testmarktest_rule_basic '*/input' exercise_mytool +testmarktest_run fixtures.md ``` +That's a whole test file. +Every hunk in `fixtures.md` named `SOMETHING/input` is a case: +it's fed to `exercise_mytool`, and what that prints is checked against the hunk named `SOMETHING/output`. +Adding a case is adding two blocks to the doc. + It needs bash 4.4 or later. @@ -52,6 +60,51 @@ Later, when the tool's output changes on purpose, it's the same: regen, read the The diff is the review. +Writing a test +-------------- + +A test file makes rules, then runs them over its docs. + +**A rule** is a glob and a function: `testmarktest_rule_basic GLOB FN`. +Every hunk whose name matches GLOB triggers a case, and FN exercises it. +The case is the hunk's name minus its last segment, so `greeting/input` is the case `greeting`, and its output goes in `greeting/output`. +In the glob, `*` matches `/` too, so `'*/input'` finds `a/b/input` as well. + +**The function** is where your tool gets run. +We call ours `exercise_SOMETHING`, by convention. +Each case runs it in a fresh bash, so it's a clean shell: + +- stdin is the trigger hunk's body, exactly. +- `set -euo pipefail` is on. A failing command stops the case, and the failure shows up in its output. + Bash's own error messages there are labeled with the function's name, nothing more, so they read the same wherever the test runs. +- The working directory is the test script's, so a relative path like `./mytool` works, wherever the test is run from. +- It sees exported variables (like `PATH`), but no other variables, and no other functions. + If it calls helpers, pass them to `testmarktest_provide helper1 helper2` first. + +**What gets checked** is everything the function prints, stdout and stderr together, in order. +If the function exits nonzero, a last line `exit status: N` is added, so a case can't pass while its tool fails, unless the fixture says it should. +On success nothing is added: the fixture for a tool that works is just its output. + +Line endings don't count. +CRLF reads as LF, and a missing final newline is added, in hunks and in output alike. + +**`testmarktest_run DOC...`** runs every case, prints a `PASS` or `FAIL` line for each, and writes the docs if `TESTMARK_REGEN` is set. +Doc paths are relative to the test script. +It fails if any case failed, and also if a rule matched no hunk at all, or a doc matched no rule: +a typo in a glob can't pass by running nothing. + +A doc can hold several kinds of cases. Give each kind its own rule: + +```bash +testmarktest_rule_basic '*/script' exercise_script +testmarktest_rule_basic '*/input' exercise_mytool +``` + +Nothing adds hunks. +The output hunk must exist already, even if empty: regen fills in a block that's there, and a missing one fails the case. +That's why the workflow starts with an empty block. + + A whole example --------------- @@ -61,65 +114,28 @@ A whole example #!/bin/bash ## Checks todos.sh against every case in todos.md. ## Run with TESTMARK_REGEN=1 to update todos.md to match what todos.sh does now; then review the change with `git diff`. -set -euo pipefail -cd "$(dirname "$0")" -source ../lib/testmark.bash - -work=$(mktemp -d) -trap 'rm -rf "$work"' EXIT - -testmark_parse doc todos.md -mapfile -t names < <(testmark_names doc) -fails=0 -for name in "${names[@]}"; do - case $name in - */input) - case=${name%/input} - testmark_body doc "$name" >"$work/notes.txt" - if testmark_check doc "$case/output" < <(./todos.sh "$work/notes.txt" | sed "s|$work|WORK|g"); then - echo "PASS $case" - else - echo "FAIL $case" - fails=$((fails + 1)) - fi - ;; - esac -done -testmark_write doc -((fails == 0)) -``` - -What to copy from it: - -- **One case per name.** Hunks named `CASE/input` and `CASE/output` make a table; - adding a case is adding two blocks to the doc, with no change to the test. -- **Normalize, then check exactly.** Anything that changes from run to run, like a temp dir, gets replaced before the check (here, with `sed`), - so the check can stay exact. - Prefer this over grepping for a fragment of the output: - an exact check can't pass by accident, it shows the reader what the output really looks like, and regen can write it. -- **`< <(...)` into the check.** Not a pipe; see the rules below. -- **`testmark_write` at the end.** That's what makes `TESTMARK_REGEN=1` write anything. - With nothing to write, it does nothing, so call it unconditionally. - -A check doesn't see the tool's exit status. -When it matters, put it in the output: `< <(mytool || echo "exit: $?")` adds a line only when the tool fails. +source "$(dirname "${BASH_SOURCE[0]}")/../lib/testmark.bash" +exercise_todos() { + ./todos.sh /dev/stdin +} -Rules to know -------------- +testmarktest_rule_basic '*/input' exercise_todos +testmarktest_run todos.md +``` -**State changes happen in the shell that parsed.** -`testmark_check`, `testmark_patch`, and `testmark_write` refuse to run in a subshell, where what they stage would be lost. -So feed a check with `testmark_check doc NAME < <(cmd)`, not `cmd | testmark_check doc NAME`; -and don't call these inside `$(...)`, or inside a test helper that runs its command there. +**Normalize, then check exactly.** +If your tool's output has anything that changes from run to run, like a temp dir or a timing, replace it in the function: -**Output must be empty, or end in a newline, with no CRLF.** -A hunk's body can't hold anything else, so `testmark_check` rejects it outright, rather than failing forever. -Normalize it first. +```bash +exercise_build() { + mytool build | sed 's/in [0-9.]*s$/in Ns/' +} +``` -**A check never creates a hunk.** -Regen rewrites the body of a hunk that's there; a missing one is an error in both modes. -That's why the workflow starts with an empty block. +Then the check can stay exact. +Prefer this over grepping for a fragment of the output: +an exact check can't pass by accident, it shows the reader what the output really looks like, and regen can write it. Functions @@ -127,18 +143,22 @@ Functions | function | does | | --- | --- | -| `testmark_parse HANDLE FILE` | parses FILE, and names it HANDLE for the other calls | -| `testmark_names HANDLE` | prints the hunk names, one per line, in document order | -| `testmark_body HANDLE NAME` | prints a hunk's body, exactly | -| `testmark_info HANDLE NAME` | prints a hunk's info string (`text` in ```` ```text ````) | -| `testmark_check HANDLE NAME <OUTPUT` | compares OUTPUT to the body: returns 1 and shows a diff on a mismatch; or under `TESTMARK_REGEN`, stages OUTPUT as the new body | -| `testmark_patch HANDLE NAME [INFO] <BODY` | stages a new body, and optionally a new info string | -| `testmark_write HANDLE` | writes what's staged into the file, in place | +| `testmarktest_rule_basic GLOB FN` | makes each hunk matching GLOB a case, exercised by FN, with the hunk on stdin, and checked against the case's `output` hunk | +| `testmarktest_provide FN...` | sends more functions into every case, for exercise functions that call them | +| `testmarktest_run DOC...` | runs every case in the docs, reports each, and writes the docs under `TESTMARK_REGEN` | Errors come back as a nonzero status, with a `testmark: ...` line on stderr. The doc comments in [`lib/testmark.bash`](lib/testmark.bash) are the full reference. +Going it alone +-------------- + +The runner is built on primitives you can call yourself: +to read and rewrite testmark docs as plain data, or to check output in a shape the runner doesn't have. +[docs/userguide/going-it-alone.md](docs/userguide/going-it-alone.md) covers them. + + Requirements ------------ diff --git a/docs/hacking/bash-pitfalls.md b/docs/hacking/bash-pitfalls.md index 043cb1c..7b13ca6 100644 --- a/docs/hacking/bash-pitfalls.md +++ b/docs/hacking/bash-pitfalls.md @@ -158,6 +158,56 @@ We've been unable to use `set -e` as a strict mode (unfortunately). That means an awful lot of `cmd || return 1` is required. +The reason is a suppression rule with long reach. +`set -e` is ignored for any command whose failure is being tested: +the condition of an `if` or `while`, a command under `!`, +or any command of an `&&`/`||` list except the last. +So far, so documented. +The reach is that the suppression extends dynamically into every function *called* from such a position, +where `set -e` can neither be re-enabled nor even detected as off. +Library code must expect calls like `testmark_check ... || status=$?`, +so every function here has to assume `set -e` does nothing for it. + +The suppression also reaches into most child processes. +Two kinds of child process matter here, and they behave differently: + +- **a fork of this shell**: a subshell `( )`, an async command `&`, a command substitution `$( )`, or a process substitution `<( )`. + Bash forks without exec, so the child has every variable, function, and piece of internal state of its parent. +- **a new bash process**: `bash -c ...`, or `"$BASH" file`. Bash is exec'd afresh, and gets only the exported environment. + +Measured on bash 4.4, 5.0, 5.2, and 5.3, identically. +`set -e` is on, and `f` runs `false`. +**honored** means the `false` ended the process running `f`; +**IGNORED** means `e` is in `$-`, and the `false` didn't end it; +**off** means `e` is not in `$-`. + +| where `f` runs | called plainly | called from a tested context | +| --- | --- | --- | +| this shell | honored | IGNORED | +| a subshell `( f )` | honored | IGNORED | +| an async command `f &` | honored | IGNORED | +| a command substitution `$(f)` | off | off | +| a command substitution, with `shopt -s inherit_errexit` | honored | IGNORED | +| a process substitution `<(f)` | honored | honored, except as below | +| a new bash process running `set -e; f` | honored | honored | + +Why, in bash's source: being tested is a flag on each parsed command, `CMD_IGNORE_RETURN`. +Bash copies it into compound commands, into a subshell's command, and into a function's body when it's called, +so a subshell or async command, which runs its parent's parsed commands, inherits it. +Both substitutions parse their text afresh; +command substitution puts the flag back on its new commands when its caller was tested (`comsub_ignore_return`), and process substitution doesn't. + +The exception: `eval`, `source`, and `command`, when run in a tested context, also set a global, `builtin_ignoring_errexit`, and while it's set, `set -e` doesn't turn errexit on. +A process substitution keeps that global. +So a process substitution started anywhere inside a `source` or `eval` that is itself in a tested context (`source tests/foo.sh || fail`, say) runs with errexit IGNORED, +and nothing inside it can tell, or undo it. +A new bash process doesn't keep it: exec discards it. + +So the only child where errexit is honored in every case is a new bash process. +This is why the runner gives every case one: +a case is promised a working `set -euo pipefail`, +and the runner can't know whether its test file was sourced or eval'd from a tested context. + ### treat unset variables as fatal We turned `set -u` on in our library code, as a strict mode to catch as many bugs as possible. diff --git a/docs/userguide/going-it-alone.md b/docs/userguide/going-it-alone.md new file mode 100644 index 0000000..24dad95 --- /dev/null +++ b/docs/userguide/going-it-alone.md @@ -0,0 +1,108 @@ +Going it alone: the primitives +============================== + +The runner (`testmarktest_rule_basic` and `testmarktest_run`) is built on a handful of primitives, and you can call them yourself. +Reach for them when the runner doesn't fit: +when you want to read testmark docs as plain data, or rewrite them, or check output in a shape the runner doesn't have. + +They give you more control, but also more to get right. +If the runner fits, use it. + + +The functions +------------- + +Every primitive takes a handle: a name you pick when you parse a doc, and use for that doc in every other call. + +| function | does | +| --- | --- | +| `testmark_parse HANDLE FILE` | parses FILE, and names it HANDLE for the other calls | +| `testmark_names HANDLE` | prints the hunk names, one per line, in document order | +| `testmark_body HANDLE NAME` | prints a hunk's body, exactly | +| `testmark_info HANDLE NAME` | prints a hunk's info string (`text` in ```` ```text ````) | +| `testmark_check HANDLE NAME <OUTPUT` | compares OUTPUT to the body: returns 1 and shows a diff on a mismatch; or under `TESTMARK_REGEN`, stages OUTPUT as the new body | +| `testmark_patch HANDLE NAME [INFO] <BODY` | stages a new body, and optionally a new info string | +| `testmark_write HANDLE` | writes what's staged into the file, in place | + +Errors come back as a nonzero status, with a `testmark: ...` line on stderr. +The doc comments in [`lib/testmark.bash`](../../lib/testmark.bash) are the full reference. + + +Reading and writing docs as data +-------------------------------- + +Read a hunk: + +```bash +testmark_parse doc settings.md +greeting=$(testmark_body doc greeting) +``` + +Replace a hunk's body, then write the file: + +```bash +testmark_patch doc greeting < <(echo "hello again") +testmark_write doc +``` + +Patches are staged until `testmark_write`, which writes them all at once, and then parses the file again. + + +A test loop by hand +------------------- + +[`examples/test-todos-alone.sh`](../../examples/test-todos-alone.sh) does by hand what the runner does for `examples/todos.md`: + +```bash +#!/bin/bash +set -euo pipefail +cd "$(dirname "${BASH_SOURCE[0]}")" +source ../lib/testmark.bash + +testmark_parse doc todos.md +mapfile -t names < <(testmark_names doc) +fails=0 +for name in "${names[@]}"; do + case $name in + */input) + case=${name%/input} + if testmark_check doc "$case/output" < <(testmark_body doc "$name" | ./todos.sh /dev/stdin); then + echo "PASS $case" + else + echo "FAIL $case" + fails=$((fails + 1)) + fi + ;; + esac +done +testmark_write doc +((fails == 0)) +``` + +The `case` statement picks out the hunks this test cares about. +It's the shape these loops tend to grow into: a doc often holds several kinds of hunks, and each kind gets its own branch. + +Everything the runner did for you is now yours to get right: + +- **`testmark_write` at the end.** That's what makes `TESTMARK_REGEN=1` write anything. + Without it, regen stages everything, then throws it away, without a word. + With nothing staged, it does nothing, so call it unconditionally. +- **Counting failures.** Run every case, and exit nonzero if any failed. + A bare check under `set -e` stops at the first mismatch, and you only see one. +- **The exit status.** A check doesn't see it. Put it in the output: `< <(mytool || echo "exit: $?")` adds a line only when the tool fails. +- **Clean shells.** The runner runs each case in a fresh bash; here, everything shares one shell, and one case's leftovers can reach the next. + + +Rules to know +------------- + +### state changes happen in the shell that parsed + +`testmark_check`, `testmark_patch`, and `testmark_write` refuse to run in a subshell, where what they stage would be lost. +So feed a check with `testmark_check doc NAME < <(cmd)`, not `cmd | testmark_check doc NAME`; +and don't call these inside `$(...)`, or inside a test helper that runs its command there. + +### all the other usual rules still apply + +The rules in the README's [writing a test](../../README.md#writing-a-test) about line endings and missing hunks hold here too: +those are properties of the primitives. diff --git a/examples/test-todos-alone.sh b/examples/test-todos-alone.sh new file mode 100755 index 0000000..d544787 --- /dev/null +++ b/examples/test-todos-alone.sh @@ -0,0 +1,25 @@ +#!/bin/bash +## Checks todos.sh against every case in todos.md, as test-todos.sh does, but with the primitives instead of the runner. +## Run with TESTMARK_REGEN=1 to update todos.md to match what todos.sh does now; then review the change with `git diff`. +set -euo pipefail +cd "$(dirname "${BASH_SOURCE[0]}")" +source ../lib/testmark.bash + +testmark_parse doc todos.md +mapfile -t names < <(testmark_names doc) +fails=0 +for name in "${names[@]}"; do + case $name in + */input) + case=${name%/input} + if testmark_check doc "$case/output" < <(testmark_body doc "$name" | ./todos.sh /dev/stdin); then + echo "PASS $case" + else + echo "FAIL $case" + fails=$((fails + 1)) + fi + ;; + esac +done +testmark_write doc +((fails == 0)) diff --git a/examples/test-todos.sh b/examples/test-todos.sh index 61a3ceb..13ef873 100755 --- a/examples/test-todos.sh +++ b/examples/test-todos.sh @@ -1,29 +1,11 @@ #!/bin/bash ## Checks todos.sh against every case in todos.md. ## Run with TESTMARK_REGEN=1 to update todos.md to match what todos.sh does now; then review the change with `git diff`. -set -euo pipefail -cd "$(dirname "$0")" -source ../lib/testmark.bash +source "$(dirname "${BASH_SOURCE[0]}")/../lib/testmark.bash" -work=$(mktemp -d) -trap 'rm -rf "$work"' EXIT +exercise_todos() { + ./todos.sh /dev/stdin +} -testmark_parse doc todos.md -mapfile -t names < <(testmark_names doc) -fails=0 -for name in "${names[@]}"; do - case $name in - */input) - case=${name%/input} - testmark_body doc "$name" >"$work/notes.txt" - if testmark_check doc "$case/output" < <(./todos.sh "$work/notes.txt" | sed "s|$work|WORK|g"); then - echo "PASS $case" - else - echo "FAIL $case" - fails=$((fails + 1)) - fi - ;; - esac -done -testmark_write doc -((fails == 0)) +testmarktest_rule_basic '*/input' exercise_todos +testmarktest_run todos.md diff --git a/examples/todos.md b/examples/todos.md index 2d04c3c..4039dfa 100644 --- a/examples/todos.md +++ b/examples/todos.md @@ -5,7 +5,7 @@ todos.sh, by example Each case below is a pair of hunks: `CASE/input` is a file's content, and `CASE/output` is what `todos.sh` prints for that file. -`test-todos.sh` runs every case, with the file in a temporary directory whose path it replaces with `WORK`. +`test-todos.sh` runs every case, handing `todos.sh` the input as `/dev/stdin`. ### the basics @@ -22,8 +22,8 @@ Lines are numbered from one. [testmark]:# (basic/output) ```text -WORK/notes.txt:2:# TODO: handle spaces in names -WORK/notes.txt:4:# TODO: and newlines +/dev/stdin:2:# TODO: handle spaces in names +/dev/stdin:4:# TODO: and newlines ``` @@ -40,8 +40,8 @@ notes: [testmark]:# (verbatim/output) ```text -WORK/notes.txt:2: TODO: indented with a tab -WORK/notes.txt:3: \TODO: with a backslash +/dev/stdin:2: TODO: indented with a tab +/dev/stdin:3: \TODO: with a backslash ``` diff --git a/lib/testmark.bash b/lib/testmark.bash index 050a507..d4d7590 100644 --- a/lib/testmark.bash +++ b/lib/testmark.bash @@ -4,6 +4,7 @@ ## Needs bash 4.4 or later; on older ones, sourcing fails. ## Every public function is a shim over a private `_testmark_*` implementation, run through `_testmark_call`. ## Data comes out on stdout as exact bytes, and goes in on stdin; errors go to stderr as "testmark: ...", with a nonzero status. +## Data going in is normalized as parsing normalizes hunk bodies: CRLF becomes LF, and a missing final newline is added. ## Documents are addressed by a handle the caller picks at parse time; handles match `[A-Za-z0-9_]+`. # Checked first, in syntax bash 3.2 (macOS's /bin/bash) still runs, so an old bash gets one clear error. @@ -18,6 +19,8 @@ source "${BASH_SOURCE[0]%/*}/testmark/document.bash" source "${BASH_SOURCE[0]%/*}/testmark/patch.bash" # shellcheck source=lib/testmark/check.bash source "${BASH_SOURCE[0]%/*}/testmark/check.bash" +# shellcheck source=lib/testmark/run.bash +source "${BASH_SOURCE[0]%/*}/testmark/run.bash" ## Runs an implementation function in a known environment, whatever the caller's settings. ## @@ -34,6 +37,9 @@ _testmark_call() { return "$_testmark_st" } +# The outermost entry is the script bash is running, unless there's none: then it's this file, alone. +((${#BASH_SOURCE[@]} < 2)) || _testmark_call _testmarktest_init "${BASH_SOURCE[-1]}" + ## Parses FILE and makes it available under HANDLE. ## Parsing again under the same handle replaces the earlier document; if that parse fails, the handle is left unusable. ## @@ -64,7 +70,7 @@ testmark_info() { _testmark_call _testmark_info "$@"; } ## - @param $1 handle ## - @param $2 hunk name ## - @param $3 optional: a new info string; the opening fence line becomes "```" and it. Without it, that line is kept as it is. -## - @stdin the new body, exactly; a final newline is added if it lacks one. No line of it may start with "```". +## - @stdin the new body, normalized as all input is. No line of it may start with "```". testmark_patch() { _testmark_call _testmark_patch "$@"; } ## Writes what's staged into the parsed file, in place, then parses the file again under the same handle. @@ -79,7 +85,46 @@ testmark_write() { _testmark_call _testmark_write "$@"; } ## ## - @param $1 handle ## - @param $2 hunk name -## - @stdin the output: empty, or lines each ending in a newline, without CRLF. +## - @stdin the output, normalized as all input is, then compared exactly. ## - @return 0 if it matches, or if TESTMARK_REGEN staged it; 1 on a mismatch, or any error. ## - @stderr on a mismatch, a line saying so, then a `diff -u` of the expected and actual output. testmark_check() { _testmark_call _testmark_check "$@"; } + +# The high road: a test file makes rules, then runs them over its docs. +# +# exercise_todos() { ./todos.sh /dev/stdin; } +# testmarktest_rule_basic '*/input' exercise_todos +# testmarktest_run todos.md +# +# Rules and provided functions accumulate for the rest of the script. + +## Makes every hunk whose name matches GLOB a case, to be exercised by FN. +## The case is the hunk's name minus its last `/`-segment; the hunk is its trigger. +## testmarktest_run runs FN for each case in a fresh bash, in the script's directory, with the trigger's body on stdin, +## and checks what FN prints, stdout and stderr together, against the case's `output` hunk, using testmark_check. +## If FN exits nonzero, a line `exit status: N` is added to its output before the check. +## +## FN runs under `set -euo pipefail`. It sees exported variables (except BASH_ENV, so its bash starts clean), but no other variables, and no functions but itself and those given to testmarktest_provide. +## +## Bash's own runtime error messages from inside these functions (a command not found, an unbound variable) are relabeled in the output: +## `FN: message`, with no path and no line number, so the text is the same wherever the test runs. +## The relabeling matches lines by their text, so output of the case's own that happens to look like such an error gets relabeled too. +## For that, set TESTMARK_RAW_ERRORS, to any value, and the output is checked exactly as printed. +## +## - @param $1 GLOB: a bash pattern, matched against whole hunk names; `*` matches `/` too. +## - @param $2 FN: the name of a defined function. +testmarktest_rule_basic() { _testmark_call _testmarktest_rule_basic "$@"; } + +## Sends more functions into every case, for case functions that call them. +## +## - @param $@ names of defined functions. +testmarktest_provide() { _testmark_call _testmarktest_provide "$@"; } + +## Runs every rule's cases in each DOC, in document order, then writes each DOC's regen changes (see testmark_check). +## A rule that matches no hunk in any DOC, or a DOC with no match for any rule, is a failure. +## +## - @param $@ DOC paths, relative to the directory of the script being run. +## - @stdout one line per case: PASS or FAIL, the doc, the trigger's name, and the function. +## - @stderr each failure's details. +## - @return 0 if every case passed; 1 otherwise. It always runs every case. +testmarktest_run() { _testmark_call _testmarktest_run "$@"; } diff --git a/lib/testmark/check.bash b/lib/testmark/check.bash index 3d1dcae..741f767 100644 --- a/lib/testmark/check.bash +++ b/lib/testmark/check.bash @@ -10,18 +10,12 @@ _testmark_check() { _testmark_hunk "$1" "$2" || return 1 _testmark_same_shell "$1" || return 1 local input body - _testmark_input "$1: hunk \"$2\"" || return 1 - # A body is "" or lines ending in LF, never CRLF, so other output could never match, and regen would rewrite it on every run. - [[ -z $input || ($input == *$'\n' && $input != *$'\r\n'*) ]] || { - _testmark_err "$1: hunk \"$2\": output must be empty or end in a newline, and contain no CRLF; normalize it before checking" - return 1 - } - + _testmark_input "$1" "$2" || return 1 # Two checks claiming one hunk with different output is a harness bug, whether or not the first one matched. # The same output again is compared like the first time, so a mismatch can't pass by repetition. local -n checked=_testmark_checked_$1 [[ ${checked[$2]-$input} == "$input" ]] || { - _testmark_err "$1: hunk \"$2\": checked twice, with different output" + _testmark_hunk_err "$1" "$2" "checked twice, with different output" return 1 } checked[$2]=$input @@ -33,7 +27,7 @@ _testmark_check() { return fi local -n file=_testmark_file_$1 - _testmark_err "$1: hunk \"$2\": output doesn't match; set TESTMARK_REGEN=1 to accept it" + _testmark_hunk_err "$1" "$2" "output doesn't match" # diff's status isn't checked: the comparison above has decided already, and diff is only here to show it. >&2 diff -u --label "$file: $2 (expected)" --label "$file: $2 (actual)" <(printf '%s' "$body") <(printf '%s' "$input") return 1 diff --git a/lib/testmark/document.bash b/lib/testmark/document.bash index 396bda8..8bdb920 100644 --- a/lib/testmark/document.bash +++ b/lib/testmark/document.bash @@ -20,6 +20,13 @@ _testmark_err() { >&2 printf 'testmark: %s\n' "$1" } +## Prints an error about hunk $2 of document $1, which must both exist: "testmark: FILE:LINE: hunk "NAME": $3". +## LINE is the hunk's annotation line, so a reader can go straight to it. +_testmark_hunk_err() { + local -n file=_testmark_file_$1 start=_testmark_start_$1 + _testmark_err "$file:$((start[$2] - 1)): hunk \"$2\": $3" +} + ## Fails with a message unless $1 is a valid handle. _testmark_handle() { [[ -n $1 && $1 != *[!A-Za-z0-9_]* ]] || { @@ -135,7 +142,8 @@ _testmark_hunk() { _testmark_doc "$1" || return 1 local -n start=_testmark_start_$1 [[ -n ${start[$2]+set} ]] || { - _testmark_err "$1: no hunk named \"$2\"" + local -n file=_testmark_file_$1 + _testmark_err "$file: no hunk named \"$2\"" return 1 } } diff --git a/lib/testmark/patch.bash b/lib/testmark/patch.bash index f3b40fb..a400d85 100644 --- a/lib/testmark/patch.bash +++ b/lib/testmark/patch.bash @@ -12,27 +12,30 @@ _testmark_same_shell() { } } -## Reads all of stdin into the caller's local `input`. +## Reads all of stdin into the caller's local `input`, normalized the way parsing normalizes a hunk's body: +## CRLF becomes LF, and a final newline is added if it's missing. +## So input reads as the body it would become, and a check compares like with like. ## -## - @param $1 what the input is for, as a prefix for the error message +## - @param $1 handle, and $2 hunk name, of the hunk the input is for, for the error message _testmark_input() { # Status 0 means read stopped at a NUL; see "NUL bytes vanish unless you check for them" in docs/hacking/bash-pitfalls.md. if IFS= read -r -d '' input; then - _testmark_err "$1: input contains a NUL byte" + _testmark_hunk_err "$1" "$2" "input contains a NUL byte" return 1 fi + [[ $input != *$'\r\n'* ]] || input=${input//$'\r\n'/$'\n'} + [[ -z $input || $input == *$'\n' ]] || input+=$'\n' } ## Stages a new body for hunk $2 of document $1, which must both exist, replacing anything staged for it before. ## -## - @param $3 the body; a final newline is added if it lacks one +## - @param $3 the body: empty, or ending in a newline ## - @param $4 optional: the new opener line; without it, the hunk's own opener is kept _testmark_stage() { local body=$3 opener=${4-} - [[ -z $body || $body == *$'\n' ]] || body+=$'\n' # Any line starting with ``` closes the code block, so such a body would end its hunk early and corrupt the rest of the doc. [[ $'\n'$body != *$'\n```'* ]] || { - _testmark_err "$1: hunk \"$2\": body has a line starting with \`\`\`, which would end its code block" + _testmark_hunk_err "$1" "$2" "body has a line starting with \`\`\`, which would end its code block" return 1 } local -n lines=_testmark_lines_$1 start=_testmark_start_$1 patch=_testmark_patch_$1 @@ -52,12 +55,12 @@ _testmark_patch() { if (($# == 3)); then # Parse trims info strings, and a newline would end the opener line, so neither would read back as given. [[ $3 != *$'\n'* && $3 != [[:space:]]* && $3 != *[[:space:]] ]] || { - _testmark_err "$1: hunk \"$2\": info string must be one line, without leading or trailing whitespace" + _testmark_hunk_err "$1" "$2" "info string must be one line, without leading or trailing whitespace" return 1 } opener='```'$3$'\n' fi - _testmark_input "$1: hunk \"$2\"" || return 1 + _testmark_input "$1" "$2" || return 1 _testmark_stage "$1" "$2" "$input" "$opener" } diff --git a/lib/testmark/run.bash b/lib/testmark/run.bash new file mode 100644 index 0000000..fb42401 --- /dev/null +++ b/lib/testmark/run.bash @@ -0,0 +1,200 @@ +## run.bash: the high road -- rules that make hunks into cases, and a runner that exercises each case and checks its output. +## Sourced by testmark.bash; the functions here assume they run inside `_testmark_call`. + +# Rules, in the order made, as parallel arrays: rule i is glob i and function i. +_testmarktest_globs=() +_testmarktest_fns=() +# Functions sent into every case, besides the rule's own. +_testmarktest_provided=() +# A case's script sources function i from pinned fd (this base + i): the case's own function, then the provided ones, in order. +# _testmarktest_script assigns the pins; _testmarktest_relabel turns them back into function names. +_testmarktest_fdbase=10 +# The directory of the script being run, absolute; empty if bash isn't running a script file. +# Set when the library is sourced, so a later `cd` can't move it. +_testmarktest_dir= + +## Sets _testmarktest_dir to the directory of script $1, if it's a file. +_testmarktest_init() { + local top=$1 + [[ -f $top ]] || return 0 + [[ $top == */* ]] || top=./$top + _testmarktest_dir=${top%/*} + _testmarktest_dir=${_testmarktest_dir:-/} + [[ $_testmarktest_dir == /* ]] || _testmarktest_dir=$PWD/$_testmarktest_dir +} + +## Fails with a message unless $2 is a defined function; $1 is the public function asking, for the message. +_testmarktest_need_fn() { + declare -F -- "$2" >/dev/null || { + _testmark_err "$1: \"$2\" is not a defined function" + return 1 + } +} + +_testmarktest_rule_basic() { + (($# == 2)) || { + _testmark_err "usage: testmarktest_rule_basic GLOB FN" + return 1 + } + [[ -n $1 ]] || { + _testmark_err "testmarktest_rule_basic: the glob is empty" + return 1 + } + _testmarktest_need_fn testmarktest_rule_basic "$2" || return 1 + _testmarktest_globs+=("$1") + _testmarktest_fns+=("$2") +} + +_testmarktest_provide() { + (($# > 0)) || { + _testmark_err "usage: testmarktest_provide FN..." + return 1 + } + # The pins count up from _testmarktest_fdbase and must stay clear of the fds bash gives + # process substitutions (63, counting down), so a case holds at most 50 functions. + ((${#_testmarktest_provided[@]} + $# + 1 <= 50)) || { + _testmark_err "testmarktest_provide: a case can hold at most 50 functions" + return 1 + } + local fn + for fn; do _testmarktest_need_fn testmarktest_provide "$fn" || return 1; done + _testmarktest_provided+=("$@") +} + +_testmarktest_run() { + (($# > 0)) || { + _testmark_err "usage: testmarktest_run DOC..." + return 1 + } + ((${#_testmarktest_fns[@]} > 0)) || { + _testmark_err "testmarktest_run: no rules; make them with testmarktest_rule_basic first" + return 1 + } + [[ -n $_testmarktest_dir ]] || { + _testmark_err "testmarktest_run: no script file to find the docs beside; run it from a script" + return 1 + } + local -a matched=() + local i doc failed=0 + for i in "${!_testmarktest_fns[@]}"; do matched[i]=0; done + for doc; do _testmarktest_doc "$doc" || failed=1; done + for i in "${!_testmarktest_fns[@]}"; do + ((matched[i] > 0)) || { + _testmark_err "rule \"${_testmarktest_globs[i]}\" ${_testmarktest_fns[i]}: matched no hunk" + failed=1 + } + done + return "$failed" +} + +## Runs every case in doc $1, reports each, and writes the doc's regen patches. +## Counts each rule's triggers in the caller's `matched`. +_testmarktest_doc() { + local doc=$1 path=$1 h=testmarktest_run + [[ $path == /* ]] || path=$_testmarktest_dir/$path + _testmark_parse "$h" "$path" || { + printf 'FAIL %s\n' "$doc" + return 1 + } + local -a names + local name i failed=0 any= + mapfile -t names < <(_testmark_names "$h") + for name in "${names[@]}"; do + for i in "${!_testmarktest_globs[@]}"; do + # shellcheck disable=SC2053 # the glob is a pattern on purpose + [[ $name == ${_testmarktest_globs[i]} ]] || continue + matched[i]=$((matched[i] + 1)) any=1 + if _testmarktest_case "$h" "$name" "${_testmarktest_fns[i]}"; then + printf 'PASS %s: %s (%s)\n' "$doc" "$name" "${_testmarktest_fns[i]}" + else + printf 'FAIL %s: %s (%s)\n' "$doc" "$name" "${_testmarktest_fns[i]}" + failed=1 + fi + done + done + [[ -n $any ]] || { + _testmark_err "$doc: no hunk matches any rule" + failed=1 + } + _testmark_write "$h" || failed=1 + return "$failed" +} + +## Exercises the case of trigger $2 in document $1 with function $3, and checks its output against the case's `output` hunk. +_testmarktest_case() { + [[ $2 == */* ]] || { + _testmark_hunk_err "$1" "$2" "has no \"/\", so it names no case" + return 1 + } + local out=${2%/*}/output stream st + _testmark_hunk "$1" "$out" || return 1 + # The exit status follows the output after a NUL, which bash can't hold; so a second NUL means the output held one. + { + IFS= read -r -d '' stream + if IFS= read -r -d '' st; then + _testmark_hunk_err "$1" "$2" "output contains a NUL byte" + return 1 + fi + } < <( + _testmarktest_exec "$1" "$2" "$3" + printf '\0%s' "$?" + ) + [[ -n ${TESTMARK_RAW_ERRORS+set} ]] || _testmarktest_relabel stream "$3" + if ((st != 0)); then + [[ -z $stream || $stream == *$'\n' ]] || stream+=$'\n' + stream+="exit status: $st"$'\n' + fi + _testmark_check "$1" "$out" < <(printf '%s' "$stream") +} + +## Runs function $3 in a fresh bash, in the script's directory, with the body of hunk $2 of document $1 on stdin. +## stdout and stderr both go to stdout. Runs in a process substitution, so its `cd` and `unset` change nothing outside. +_testmarktest_exec() { + # _testmark_call's `local LC_ALL=C` would reach the case if the caller exported LC_ALL; unsetting it uncovers the caller's own. + # BASH_ENV would be sourced by the case's bash, and a case should start clean. + unset -v LC_ALL BASH_ENV + cd "$_testmarktest_dir" 2>&1 || return + "$BASH" <(_testmarktest_script "$3") < <(_testmark_body "$1" "$2") 2>&1 +} + +## Prints the script a case runs: strict mode, the functions it may call, and a call to function $1. +## This is the one place a case's script is built. +_testmarktest_script() { + local fn fd=$_testmarktest_fdbase + printf 'set -euo pipefail\n' + # Each function is sourced from its own pinned fd, so bash's errors inside one name a source the + # harness chose, whatever fds the environment holds open; _testmarktest_relabel maps it back. + for fn in "$1" "${_testmarktest_provided[@]}"; do + printf 'source /dev/fd/%d %d< <(printf %%s %q)\n' "$fd" "$fd" "$(declare -f -- "$fn")" + fd=$((fd + 1)) + done + printf '%q\n' "$1" +} + +## Relabels bash's own runtime error messages in a case's output, held by the variable named $1: +## a line beginning `/dev/fd/F: line N: `, where F is the pin of case function $2 or of a provided function, +## trades that prefix for the function's name, becoming `FN: ` and the rest. +## The line number is dropped on purpose: it counts lines of `declare -f` output, text that exists in no file. +_testmarktest_relabel() { + local -n _tmr_stream=$1 + # Most output holds none of bash's own errors, and is left as it is, without a walk. + [[ $_tmr_stream == */dev/fd/* ]] || return 0 + local -a fns=("$2" "${_testmarktest_provided[@]}") lines + local i j prefix body num + # Split once, keeping each line's newline: taking each line off the front of the string would copy the rest every time, quadratic in the output's size. + mapfile lines < <(printf '%s' "$_tmr_stream") + for i in "${!lines[@]}"; do + [[ ${lines[i]} == /dev/fd/* ]] || continue + for j in "${!fns[@]}"; do + prefix="/dev/fd/$((_testmarktest_fdbase + j)): line " + [[ ${lines[i]} == "$prefix"* ]] || continue + body=${lines[i]#"$prefix"} + num=${body%%:*} + if [[ -n $num && $num != *[!0-9]* && $body == "$num: "* ]]; then + lines[i]="${fns[j]}: ${body#"$num": }" + fi + break + done + done + printf -v _tmr_stream '%s' "${lines[@]}" +} diff --git a/tests/test-api.sh b/tests/test-api.sh index 53e1389..d0ec45b 100755 --- a/tests/test-api.sh +++ b/tests/test-api.sh @@ -28,10 +28,10 @@ testmark_parse b b.md testmark_parse e empty.md expect_eq "handle a" "$(testmark_body a x)" "one" expect_eq "handle b" "$(testmark_body b x)" "two" -expect_fail "an empty doc doesn't see another doc's hunks" 'testmark: e: no hunk named "x"' testmark_body e x +expect_fail "an empty doc doesn't see another doc's hunks" "testmark: $work/empty.md: no hunk named \"x\"" testmark_body e x expect_ok "re-parse" testmark_parse a b.md expect_eq "re-parse replaces the doc" "$(testmark_body a x)" "two" -expect_fail "re-parse drops the old doc's names" 'testmark: a: no hunk named "old"' testmark_body a old +expect_fail "re-parse drops the old doc's names" "testmark: $work/b.md: no hunk named \"old\"" testmark_body a old expect_eq "re-parse leaves other handles alone" "$(testmark_body b x)" "two" expect_ok "re-parse of the same file" testmark_parse b b.md expect_eq "and it reads" "$(testmark_body b x)" "two" @@ -68,7 +68,7 @@ expect_eq "sourcing inside a function" "$( # --- errors --- testmark_parse r a.md expect_fail "unknown handle" 'testmark: unknown handle "nope"' testmark_body nope x -expect_fail "unknown hunk" 'testmark: r: no hunk named "nope"' testmark_info r nope +expect_fail "unknown hunk" "testmark: $work/a.md: no hunk named \"nope\"" testmark_info r nope expect_fail "invalid handle" 'testmark: invalid handle "a-b": must match [A-Za-z0-9_]+' testmark_parse a-b a.md expect_fail "empty handle" 'testmark: invalid handle "": must match [A-Za-z0-9_]+' testmark_parse '' a.md expect_fail "directory" 'testmark: .: not a readable file' testmark_parse r . @@ -88,7 +88,7 @@ expect_eq "names with shell syntax: list" "$(printf '%s\n' "${got[@]}")" "$(prin for name in "${names[@]}"; do expect_eq "name $name" "$(testmark_body h "$name")" "body of $name" done -expect_fail "an unknown name with shell syntax" 'testmark: h: no hunk named "$(>pwned3)"' testmark_body h '$(>pwned3)' +expect_fail "an unknown name with shell syntax" "testmark: $work/syntax.md: no hunk named \"\$(>pwned3)\"" testmark_body h '$(>pwned3)' expect_fail "a handle with shell syntax" 'testmark: invalid handle "h[$(>pwned4)]": must match [A-Za-z0-9_]+' testmark_body 'h[$(>pwned4)]' x expect_eq "nothing was executed" "$(echo pwned*)" "pwned*" diff --git a/tests/test-check.sh b/tests/test-check.sh index 9c709bf..f8dbcf2 100755 --- a/tests/test-check.sh +++ b/tests/test-check.sh @@ -38,10 +38,17 @@ check_case "one-blank-line hunk, empty output" blank '' 1 check_case "a body of glob characters matches only itself" glob 'x\n' 1 check_case "and itself" glob '*\n' 0 +# --- output is normalized as bodies are, before comparing --- +check_case "a missing final newline is added" a 'alpha' 0 +check_case "CRLF reads as LF" a 'alpha\r\n' 0 +check_case "a lone CR is content" a 'alpha\r' 1 +check_case "CR CR LF leaves one CR" a 'alpha\r\r\n' 1 +check_case "empty output stays empty" empty '' 0 + testmark_parse c c.md expect_fail "a mismatch shows a diff" "$( cat <<-EOF - testmark: c: hunk "a": output doesn't match; set TESTMARK_REGEN=1 to accept it + testmark: $work/c.md:1: hunk "a": output doesn't match --- $work/c.md: a (expected) +++ $work/c.md: a (actual) @@ -1 +1,2 @@ @@ -63,6 +70,12 @@ expect_eq "writes only the mismatch, keeping its info string" "$(bytes c.md)" \ expect_ok "and then the same output checks clean" testmark_check c a <<<'omega' printf -- "$DOC" >c.md +testmark_parse c c.md +expect_ok "regen: output with CRLF and no final newline" env_regen testmark_check c a < <(printf 'omega\r\ntwo') +testmark_write c +expect_eq "is written normalized" "$(bytes c.md)" "$(esc "$(printf -- "${DOC/alpha/omega\\ntwo}")"$'\n')" +printf -- "$DOC" >c.md + testmark_parse c c.md TESTMARK_REGEN='' expect_ok "regen: TESTMARK_REGEN set but empty is regen too" testmark_check c a <<<'omega' testmark_write c @@ -71,15 +84,12 @@ printf -- "$DOC" >c.md # --- both modes --- FENCE='body has a line starting with ```, which would end its code block' -UNHOLDABLE='output must be empty or end in a newline, and contain no CRLF; normalize it before checking' for mode in normal regen; do run=() [[ $mode == normal ]] || run=(env_regen) testmark_parse c c.md - expect_fail "$mode: absent hunk" 'testmark: c: no hunk named "nope"' "${run[@]}" testmark_check c nope <<<'x' - expect_fail "$mode: no final newline" "testmark: c: hunk \"a\": $UNHOLDABLE" "${run[@]}" testmark_check c a < <(printf 'omega') - expect_fail "$mode: CRLF" "testmark: c: hunk \"a\": $UNHOLDABLE" "${run[@]}" testmark_check c a < <(printf 'omega\r\n') - expect_fail "$mode: a NUL byte" 'testmark: c: hunk "a": input contains a NUL byte' "${run[@]}" testmark_check c a < <(printf 'a\0b\n') + expect_fail "$mode: absent hunk" "testmark: $work/c.md: no hunk named \"nope\"" "${run[@]}" testmark_check c nope <<<'x' + expect_fail "$mode: a NUL byte" "testmark: $work/c.md:1: hunk \"a\": input contains a NUL byte" "${run[@]}" testmark_check c a < <(printf 'a\0b\n') expect_fail "$mode: arity" 'testmark: usage: testmark_check HANDLE NAME <ACTUAL' "${run[@]}" testmark_check c <<<'x' expect_ok "$mode: none of which staged anything" testmark_write c expect_eq "$mode: so the file is unchanged" "$(bytes c.md)" "$(esc "$(printf -- "$DOC")"$'\n')" @@ -92,20 +102,20 @@ for mode in normal regen; do "${run[@]}" testmark_check c a <<<"$first" 2>/dev/null || rc=$? [[ $mode == regen || $first == alpha ]] || want=1 expect_eq "$mode: first $first, then the same again" "$rc" "$want" - expect_fail "$mode: first $first, then other output" 'testmark: c: hunk "a": checked twice, with different output' "${run[@]}" testmark_check c a <<<'other' - expect_fail "$mode: first $first, then glob characters" 'testmark: c: hunk "a": checked twice, with different output' "${run[@]}" testmark_check c a <<<'*' + expect_fail "$mode: first $first, then other output" "testmark: $work/c.md:1: hunk \"a\": checked twice, with different output" "${run[@]}" testmark_check c a <<<'other' + expect_fail "$mode: first $first, then glob characters" "testmark: $work/c.md:1: hunk \"a\": checked twice, with different output" "${run[@]}" testmark_check c a <<<'*' done done testmark_parse c c.md -expect_fail "regen: output with a fence line" "testmark: c: hunk \"a\": $FENCE" env_regen testmark_check c a < <(printf 'x\n```\n') +expect_fail "regen: output with a fence line" "testmark: $work/c.md:1: hunk \"a\": $FENCE" env_regen testmark_check c a < <(printf 'x\n```\n') testmark_parse c c.md env_regen testmark_check c a <<<'omega' testmark_write c expect_fail "a write drops what was checked: other output is a mismatch, not a second check" "$( cat <<-EOF - testmark: c: hunk "a": output doesn't match; set TESTMARK_REGEN=1 to accept it + testmark: $work/c.md:1: hunk "a": output doesn't match --- $work/c.md: a (expected) +++ $work/c.md: a (actual) @@ -1 +1 @@ @@ -139,8 +149,10 @@ for name in "${names[@]}"; do done >syntax.md testmark_parse h syntax.md for name in "${names[@]}"; do env_regen testmark_check h "$name" <<<"new $name"; done -for name in "${names[@]}"; do - expect_fail "checked $name twice" "testmark: h: hunk \"$name\": checked twice, with different output" testmark_check h "$name" <<<'other' +# Each name's annotation is on line 5i+1 of syntax.md. +for i in "${!names[@]}"; do + name=${names[i]} + expect_fail "checked $name twice" "testmark: $work/syntax.md:$((5 * i + 1)): hunk \"$name\": checked twice, with different output" testmark_check h "$name" <<<'other' done testmark_write h for name in "${names[@]}"; do diff --git a/tests/test-patch.sh b/tests/test-patch.sh index b0938d7..e5a512c 100755 --- a/tests/test-patch.sh +++ b/tests/test-patch.sh @@ -36,6 +36,7 @@ patch_case empty "$DOC" 'prose\n\n[testmark]:# (x)\n```text\n```\n\nafter\n' '' patch_case one-blank-line-diverges "$DOC" 'prose\n\n[testmark]:# (x)\n```text\n\n```\n\nafter\n' '\n' x patch_case fill-empty '[testmark]:# (x)\n```\n```\n' '[testmark]:# (x)\n```\nnew\n```\n' 'new\n' x patch_case lone-cr "$DOC" 'prose\n\n[testmark]:# (x)\n```text\na\rb\n```\n\nafter\n' 'a\rb\n' x +patch_case crlf-body "$DOC" 'prose\n\n[testmark]:# (x)\n```text\na\nb\n```\n\nafter\n' 'a\r\nb' x patch_case backticks-not-at-line-start "$DOC" 'prose\n\n[testmark]:# (x)\n```text\na ```\n ```\n```\n\nafter\n' 'a ```\n ```\n' x patch_case one-of-three '[testmark]:# (a)\n```\none\n```\n[testmark]:# (b)\n```\ntwo\n```\n[testmark]:# (c)\n```\nthree\n```\n' \ '[testmark]:# (a)\n```\none\n```\n[testmark]:# (b)\n```\nnew\n```\n[testmark]:# (c)\n```\nthree\n```\n' 'new\n' b @@ -86,7 +87,7 @@ expect_eq "staging again replaces all of what was staged, info string included" cp orig.md two.md testmark_parse t two.md testmark_patch t a <<<'staged' -expect_fail "a refused patch" 'testmark: t: hunk "a": body has a line starting with ```, which would end its code block' testmark_patch t a <<<'```' +expect_fail "a refused patch" "testmark: $work/two.md:1: hunk \"a\": body has a line starting with \`\`\`, which would end its code block" testmark_patch t a <<<'```' testmark_write t mapfile -t got < <(two_md '[testmark]:# (a)\n```\nstaged\n```\n\n[testmark]:# (b)\n```text\ntwo\n```\n') expect_eq "leaves what was staged before" "${got[0]}" "${got[1]}" @@ -141,18 +142,18 @@ printf '[testmark]:# (x)\n```\nold\n```\n' >r.md testmark_parse r r.md FENCE='body has a line starting with ```, which would end its code block' INFO='info string must be one line, without leading or trailing whitespace' -expect_fail "unknown hunk" 'testmark: r: no hunk named "nope"' testmark_patch r nope <<<'x' +expect_fail "unknown hunk" "testmark: $work/r.md: no hunk named \"nope\"" testmark_patch r nope <<<'x' expect_fail "unknown handle" 'testmark: unknown handle "nope"' testmark_patch nope x <<<'x' expect_fail "write: unknown handle" 'testmark: unknown handle "nope"' testmark_write nope -expect_fail "a fence line first" "testmark: r: hunk \"x\": $FENCE" testmark_patch r x < <(printf '```\nx\n') -expect_fail "a fence line later" "testmark: r: hunk \"x\": $FENCE" testmark_patch r x < <(printf 'x\n```json\n') -expect_fail "a fence line without its newline" "testmark: r: hunk \"x\": $FENCE" testmark_patch r x < <(printf 'x\n```') -expect_fail "a four-backtick fence line" "testmark: r: hunk \"x\": $FENCE" testmark_patch r x < <(printf '````\n') -expect_fail "info: a newline" "testmark: r: hunk \"x\": $INFO" testmark_patch r x $'a\nb' <<<'x' -expect_fail "info: a final newline" "testmark: r: hunk \"x\": $INFO" testmark_patch r x $'a\n' <<<'x' -expect_fail "info: leading space" "testmark: r: hunk \"x\": $INFO" testmark_patch r x ' a' <<<'x' -expect_fail "info: trailing tab" "testmark: r: hunk \"x\": $INFO" testmark_patch r x $'a\t' <<<'x' -expect_fail "a NUL byte" 'testmark: r: hunk "x": input contains a NUL byte' testmark_patch r x < <(printf 'a\0b\n') +expect_fail "a fence line first" "testmark: $work/r.md:1: hunk \"x\": $FENCE" testmark_patch r x < <(printf '```\nx\n') +expect_fail "a fence line later" "testmark: $work/r.md:1: hunk \"x\": $FENCE" testmark_patch r x < <(printf 'x\n```json\n') +expect_fail "a fence line without its newline" "testmark: $work/r.md:1: hunk \"x\": $FENCE" testmark_patch r x < <(printf 'x\n```') +expect_fail "a four-backtick fence line" "testmark: $work/r.md:1: hunk \"x\": $FENCE" testmark_patch r x < <(printf '````\n') +expect_fail "info: a newline" "testmark: $work/r.md:1: hunk \"x\": $INFO" testmark_patch r x $'a\nb' <<<'x' +expect_fail "info: a final newline" "testmark: $work/r.md:1: hunk \"x\": $INFO" testmark_patch r x $'a\n' <<<'x' +expect_fail "info: leading space" "testmark: $work/r.md:1: hunk \"x\": $INFO" testmark_patch r x ' a' <<<'x' +expect_fail "info: trailing tab" "testmark: $work/r.md:1: hunk \"x\": $INFO" testmark_patch r x $'a\t' <<<'x' +expect_fail "a NUL byte" "testmark: $work/r.md:1: hunk \"x\": input contains a NUL byte" testmark_patch r x < <(printf 'a\0b\n') expect_fail "patch arity, too few" 'testmark: usage: testmark_patch HANDLE NAME [INFO] <BODY' testmark_patch r <<<'x' expect_fail "patch arity, too many" 'testmark: usage: testmark_patch HANDLE NAME [INFO] <BODY' testmark_patch r x json more <<<'x' expect_fail "write arity" 'testmark: usage: testmark_write HANDLE' testmark_write r r diff --git a/tests/test-run.sh b/tests/test-run.sh new file mode 100755 index 0000000..3559860 --- /dev/null +++ b/tests/test-run.sh @@ -0,0 +1,474 @@ +#!/bin/bash +## The high road: testmarktest_rule_basic, testmarktest_provide, and testmarktest_run. +## Each case writes a doc and a test script, then runs the script in a fresh bash, from another directory. +## These assert in plain bash, not through the runner itself, so a broken runner can't pass its own tests. +set -euo pipefail +cd "$(dirname "$0")" +# shellcheck source=tests/lib.sh +source ./lib.sh + +unset TESTMARK_REGEN TESTMARK_RAW_ERRORS +mkdir t + +## Writes t/doc.md and t/test.sh, runs the script from $work, and checks what it printed and its status. +## The script gets a line sourcing the library first. $work in the output shows as WORK. +## +## - @param $1 description +## - @param $2 t/doc.md's content +## - @param $3 the rest of t/test.sh +## - @param $4 the expected stdout, then a line `--- stderr`, then the expected stderr, then `--- status N` +run_case() { + printf '%s' "$2" >t/doc.md + printf 'source %q\n%s\n' "$ROOT/lib/testmark.bash" "$3" >t/test.sh + local rc=0 got + "$BASH" t/test.sh >"$work/out" 2>"$work/err" </dev/null || rc=$? + got="$(<"$work/out")"$'\n'"--- stderr"$'\n'"$(<"$work/err")"$'\n'"--- status $rc" + got=${got//$work/WORK} + expect_eq "$1" "$got" "$4" +} + +## Checks t/doc.md's bytes, as `bytes` quotes them. +## +## - @param $1 description +## - @param $2 the expected document, as a printf format +doc_is() { + expect_eq "$1" "$(bytes t/doc.md)" "$(esc "$(printf -- "$2" && printf x)" | sed 's/x"$/"/')" +} + +## Prints a doc with one case, `a`: `a/in` holds "hello", and `a/output` holds $1, which must be empty or end in a newline. +one() { + printf '[testmark]:# (a/in)\n```\nhello\n```\n\n[testmark]:# (a/output)\n```\n%s```\n' "$1" +} + +# --- passing, failing, and what the output stream holds --- + +run_case "a match passes" "$(one $'hello\n')" ' +exercise_cat() { cat; } +testmarktest_rule_basic "*/in" exercise_cat +testmarktest_run doc.md' "PASS doc.md: a/in (exercise_cat) +--- stderr + +--- status 0" + +run_case "a mismatch fails, with a diff" "$(one $'bye\n')" ' +exercise_cat() { cat; } +testmarktest_rule_basic "*/in" exercise_cat +testmarktest_run doc.md' "FAIL doc.md: a/in (exercise_cat) +--- stderr +testmark: WORK/t/doc.md:6: hunk \"a/output\": output doesn't match +--- WORK/t/doc.md: a/output (expected) ++++ WORK/t/doc.md: a/output (actual) +@@ -1 +1 @@ +-bye ++hello +--- status 1" + +run_case "stderr is in the stream, in order" "$(one $'out\nerr\nout\n')" ' +exercise_both() { echo out; echo err >&2; echo out; } +testmarktest_rule_basic "*/in" exercise_both +testmarktest_run doc.md' "PASS doc.md: a/in (exercise_both) +--- stderr + +--- status 0" + +run_case "a nonzero exit is a line of output" "$(one $'hello\nexit status: 3\n')" ' +exercise_fail() { cat; return 3; } +testmarktest_rule_basic "*/in" exercise_fail +testmarktest_run doc.md' "PASS doc.md: a/in (exercise_fail) +--- stderr + +--- status 0" + +run_case "a missing final newline is added" "$(one $'x\n')" ' +exercise_x() { printf x; } +testmarktest_rule_basic "*/in" exercise_x +testmarktest_run doc.md' "PASS doc.md: a/in (exercise_x) +--- stderr + +--- status 0" + +run_case "the exit status goes on its own line" "$(one $'x\nexit status: 1\n')" ' +exercise_x() { printf x; false; } +testmarktest_rule_basic "*/in" exercise_x +testmarktest_run doc.md' "PASS doc.md: a/in (exercise_x) +--- stderr + +--- status 0" + +run_case "empty output matches an empty hunk" "$(one '')" ' +exercise_quiet() { :; } +testmarktest_rule_basic "*/in" exercise_quiet +testmarktest_run doc.md' "PASS doc.md: a/in (exercise_quiet) +--- stderr + +--- status 0" + +run_case "CRLF reads as LF" "$(one $'a\n')" ' +exercise_crlf() { printf "a\r\n"; } +testmarktest_rule_basic "*/in" exercise_crlf +testmarktest_run doc.md' "PASS doc.md: a/in (exercise_crlf) +--- stderr + +--- status 0" + +run_case "a NUL byte fails the case" "$(one '')" ' +exercise_nul() { printf "a\0b\n"; } +testmarktest_rule_basic "*/in" exercise_nul +testmarktest_run doc.md' "FAIL doc.md: a/in (exercise_nul) +--- stderr +testmark: WORK/t/doc.md:1: hunk \"a/in\": output contains a NUL byte +--- status 1" + +# --- the case's process --- + +run_case "stdin is the trigger's body, exactly" $'[testmark]:# (a/in)\n```\n two\\\n\n```\n\n[testmark]:# (a/output)\n```\n two\\\n\n|\n```\n\n[testmark]:# (b/in)\n```\n```\n\n[testmark]:# (b/output)\n```\n|\n```\n' ' +exercise_bar() { cat; echo "|"; } +testmarktest_rule_basic "*/in" exercise_bar +testmarktest_run doc.md' "PASS doc.md: a/in (exercise_bar) +PASS doc.md: b/in (exercise_bar) +--- stderr + +--- status 0" + +run_case "set -e works in a case, even with the run under if and ||" "$(one $'exit status: 1\n')" ' +exercise_e() { false; echo unreached; } +testmarktest_rule_basic "*/in" exercise_e +if testmarktest_run doc.md; then echo ran; fi +testmarktest_run doc.md || echo failed' "PASS doc.md: a/in (exercise_e) +ran +PASS doc.md: a/in (exercise_e) +--- stderr + +--- status 0" + +run_case "set -e works in a case even when the run is inside a source that's tested" "$(one $'exit status: 1\n')" ' +printf "%s\n" "exercise_e() { false; echo unreached; }" "testmarktest_rule_basic \"*/in\" exercise_e" "testmarktest_run doc.md" >"$PWD/inner.bash" +source "$PWD/inner.bash" || :' "PASS doc.md: a/in (exercise_e) +--- stderr + +--- status 0" + +run_case "set -e works in a case even when the run is inside an eval that's tested" "$(one $'exit status: 1\n')" ' +eval "exercise_e() { false; echo unreached; }; testmarktest_rule_basic \"*/in\" exercise_e; testmarktest_run doc.md" || :' "PASS doc.md: a/in (exercise_e) +--- stderr + +--- status 0" + +run_case "pipefail is on in a case" "$(one $'exit status: 1\n')" ' +exercise_p() { false | true; echo unreached; } +testmarktest_rule_basic "*/in" exercise_p +testmarktest_run doc.md' "PASS doc.md: a/in (exercise_p) +--- stderr + +--- status 0" + +run_case "nounset is on in a case; bash's error is relabeled with the function's name" "$(one $'exercise_u: nope: unbound variable\nexit status: 1\n')" ' +exercise_u() { echo "$nope"; } +testmarktest_rule_basic "*/in" exercise_u +testmarktest_run doc.md' "PASS doc.md: a/in (exercise_u) +--- stderr + +--- status 0" + +run_case "relabeling touches only bash's error lines, amid other output" "$(one $'before\n/dev/fd/99: line 3: not ours\n\nexercise_m: nope_cmd: command not found\nafter, no newline\nexit status: 1\n')" ' +exercise_m() { echo before; echo "/dev/fd/99: line 3: not ours"; echo; nope_cmd || :; printf "after, no newline"; false; } +testmarktest_rule_basic "*/in" exercise_m +testmarktest_run doc.md' "PASS doc.md: a/in (exercise_m) +--- stderr + +--- status 0" + +run_case "held-open fds don't change a case's error text" "$(one $'exercise_u: nope: unbound variable\nexit status: 1\n')" ' +exec 10>/dev/null 61>/dev/null 62>/dev/null 63>/dev/null +exercise_u() { echo "$nope"; } +testmarktest_rule_basic "*/in" exercise_u +testmarktest_run doc.md' "PASS doc.md: a/in (exercise_u) +--- stderr + +--- status 0" + +run_case "TESTMARK_RAW_ERRORS keeps bash's own text" "$(one $'/dev/fd/10: line 3: nope: unbound variable\nexit status: 1\n')" ' +exercise_u() { echo "$nope"; } +testmarktest_rule_basic "*/in" exercise_u +TESTMARK_RAW_ERRORS=1 testmarktest_run doc.md' "PASS doc.md: a/in (exercise_u) +--- stderr + +--- status 0" + +run_case "the caller's set -e doesn't stop a run partway" '[testmark]:# (a/in) +``` +``` + +[testmark]:# (a/output) +``` +wrong +``` + +[testmark]:# (b/in) +``` +``` + +[testmark]:# (b/output) +``` +```' ' +set -euo pipefail +exercise_cat() { cat; } +testmarktest_rule_basic "*/in" exercise_cat +testmarktest_run doc.md +echo unreached' "FAIL doc.md: a/in (exercise_cat) +PASS doc.md: b/in (exercise_cat) +--- stderr +testmark: WORK/t/doc.md:5: hunk \"a/output\": output doesn't match +--- WORK/t/doc.md: a/output (expected) ++++ WORK/t/doc.md: a/output (actual) +@@ -1 +0,0 @@ +-wrong +--- status 1" + +run_case "the case runs in the script's dir; docs are found there" "$(one "$work/t"$'\n')" ' +exercise_pwd() { pwd; } +testmarktest_rule_basic "*/in" exercise_pwd +cd / +testmarktest_run doc.md' "PASS doc.md: a/in (exercise_pwd) +--- stderr + +--- status 0" + +run_case "exported variables cross; others don't" "$(one $'yes\nunset\n')" ' +export EXPORTED=yes +NOT_EXPORTED=no +exercise_env() { echo "$EXPORTED"; echo "${NOT_EXPORTED-unset}"; } +testmarktest_rule_basic "*/in" exercise_env +testmarktest_run doc.md' "PASS doc.md: a/in (exercise_env) +--- stderr + +--- status 0" + +run_case "the caller's exported LC_ALL crosses, not the library's" "$(one $'C.UTF-8\n')" ' +export LC_ALL=C.UTF-8 +exercise_lc() { echo "$LC_ALL"; } +testmarktest_rule_basic "*/in" exercise_lc +testmarktest_run doc.md' "PASS doc.md: a/in (exercise_lc) +--- stderr + +--- status 0" + +run_case "an unset LC_ALL stays unset in the case" "$(one $'unset\n')" ' +unset LC_ALL +exercise_lc() { echo "${LC_ALL-unset}"; } +testmarktest_rule_basic "*/in" exercise_lc +testmarktest_run doc.md' "PASS doc.md: a/in (exercise_lc) +--- stderr + +--- status 0" + +run_case "other functions don't cross, the library included" "$(one $'exercise_h: helper: command not found\nexit status: 127\n')" ' +helper() { echo helped; } +exercise_h() { declare -F | grep testmark || :; helper; } +testmarktest_rule_basic "*/in" exercise_h +testmarktest_run doc.md' "PASS doc.md: a/in (exercise_h) +--- stderr + +--- status 0" + +run_case "an error in a provided function is labeled with that function's name" "$(one $'helper: nope_cmd: command not found\nexit status: 127\n')" ' +helper() { + nope_cmd +} +exercise_h() { + : one + : two + helper +} +testmarktest_provide helper +testmarktest_rule_basic "*/in" exercise_h +testmarktest_run doc.md' "PASS doc.md: a/in (exercise_h) +--- stderr + +--- status 0" + +run_case "BASH_ENV doesn't reach the case" "$(one $'quiet\n')" ' +echo "echo BASH_ENV ran" >"$PWD/env.bash" +export BASH_ENV=$PWD/env.bash +exercise_quiet() { echo quiet; } +testmarktest_rule_basic "*/in" exercise_quiet +testmarktest_run doc.md' "PASS doc.md: a/in (exercise_quiet) +--- stderr + +--- status 0" + +run_case "provided functions cross" "$(one $'helped\n')" ' +helper() { echo helped; } +exercise_h() { helper; } +testmarktest_provide helper +testmarktest_rule_basic "*/in" exercise_h +testmarktest_run doc.md' "PASS doc.md: a/in (exercise_h) +--- stderr + +--- status 0" + +# --- regen --- + +run_case "regen writes the output" "$(one $'bye\n')" ' +exercise_cat() { cat; } +testmarktest_rule_basic "*/in" exercise_cat +TESTMARK_REGEN=1 testmarktest_run doc.md && testmarktest_run doc.md' "PASS doc.md: a/in (exercise_cat) +PASS doc.md: a/in (exercise_cat) +--- stderr + +--- status 0" +expect_eq "regen wrote the doc" "$(<t/doc.md)" "$(one $'hello\n')" + +run_case "regen writes every doc, and only what changed" "$(one $'hello\n')" ' +printf "%s\n" "[testmark]:# (b/in)" "\`\`\`" "x" "\`\`\`" "" "[testmark]:# (b/output)" "\`\`\`text" "\`\`\`" >"${BASH_SOURCE%/*}/two.md" +exercise_cat() { cat; } +testmarktest_rule_basic "*/in" exercise_cat +TESTMARK_REGEN=1 testmarktest_run doc.md two.md +cat "${BASH_SOURCE%/*}/two.md"' "PASS doc.md: a/in (exercise_cat) +PASS two.md: b/in (exercise_cat) +[testmark]:# (b/in) +\`\`\` +x +\`\`\` + +[testmark]:# (b/output) +\`\`\`text +x +\`\`\` +--- stderr + +--- status 0" +expect_eq "regen left an unchanged doc as it was" "$(<t/doc.md)" "$(one $'hello\n')" + +# --- what makes a run fail, besides a case --- + +run_case "a missing output hunk fails the case" '[testmark]:# (a/in) +``` +```' ' +exercise_cat() { cat; } +testmarktest_rule_basic "*/in" exercise_cat +testmarktest_run doc.md +TESTMARK_REGEN=1 testmarktest_run doc.md' "FAIL doc.md: a/in (exercise_cat) +FAIL doc.md: a/in (exercise_cat) +--- stderr +testmark: WORK/t/doc.md: no hunk named \"a/output\" +testmark: WORK/t/doc.md: no hunk named \"a/output\" +--- status 1" + +run_case "a trigger without a / fails" '[testmark]:# (in) +``` +```' ' +exercise_cat() { cat; } +testmarktest_rule_basic "*in" exercise_cat +testmarktest_run doc.md' "FAIL doc.md: in (exercise_cat) +--- stderr +testmark: WORK/t/doc.md:1: hunk \"in\": has no \"/\", so it names no case +--- status 1" + +run_case "a rule matching nothing fails the run, after every case runs" "$(one $'hello\n')" ' +exercise_cat() { cat; } +testmarktest_rule_basic "*/inn" exercise_cat +testmarktest_rule_basic "*/in" exercise_cat +testmarktest_run doc.md' "PASS doc.md: a/in (exercise_cat) +--- stderr +testmark: rule \"*/inn\" exercise_cat: matched no hunk +--- status 1" + +run_case "a doc with no match fails the run" "$(one $'hello\n')" ' +printf "%s\n" "[testmark]:# (b/script)" "\`\`\`" "\`\`\`" >"${BASH_SOURCE%/*}/two.md" +exercise_cat() { cat; } +testmarktest_rule_basic "*/in" exercise_cat +testmarktest_run two.md doc.md' "PASS doc.md: a/in (exercise_cat) +--- stderr +testmark: two.md: no hunk matches any rule +--- status 1" + +run_case "a doc that doesn't parse fails the run; the others still run" "$(one $'hello\n')" ' +exercise_cat() { cat; } +testmarktest_rule_basic "*/in" exercise_cat +testmarktest_run nope.md doc.md' "FAIL nope.md +PASS doc.md: a/in (exercise_cat) +--- stderr +testmark: WORK/t/nope.md: not a readable file +--- status 1" + +run_case "a run with no rules is an error" "$(one '')" ' +testmarktest_run doc.md' " +--- stderr +testmark: testmarktest_run: no rules; make them with testmarktest_rule_basic first +--- status 1" + +# --- several rules --- + +run_case "rules pick out their own hunks, in document order" '[testmark]:# (one/script) +``` +echo one +``` + +[testmark]:# (one/output) +``` +one +``` + +[testmark]:# (two/in) +``` +two +``` + +[testmark]:# (two/output) +``` +two +``` + +[testmark]:# (three/script) +``` +echo three +``` + +[testmark]:# (three/output) +``` +three +```' ' +exercise_script() { bash; } +exercise_cat() { cat; } +testmarktest_rule_basic "*/script" exercise_script +testmarktest_rule_basic "*/in" exercise_cat +testmarktest_run doc.md' "PASS doc.md: one/script (exercise_script) +PASS doc.md: two/in (exercise_cat) +PASS doc.md: three/script (exercise_script) +--- stderr + +--- status 0" + +run_case "a glob's * matches /; the case is all but the last segment" '[testmark]:# (a/b/c/in) +``` +deep +``` + +[testmark]:# (a/b/c/output) +``` +deep +```' ' +exercise_cat() { cat; } +testmarktest_rule_basic "*/in" exercise_cat +testmarktest_run doc.md' "PASS doc.md: a/b/c/in (exercise_cat) +--- stderr + +--- status 0" + +# --- arguments --- + +defined() { :; } +expect_fail "rule: no arguments" 'testmark: usage: testmarktest_rule_basic GLOB FN' testmarktest_rule_basic +expect_fail "rule: one argument" 'testmark: usage: testmarktest_rule_basic GLOB FN' testmarktest_rule_basic '*/in' +expect_fail "rule: three arguments" 'testmark: usage: testmarktest_rule_basic GLOB FN' testmarktest_rule_basic '*/in' defined output +expect_fail "rule: an empty glob" 'testmark: testmarktest_rule_basic: the glob is empty' testmarktest_rule_basic '' defined +expect_fail "rule: not a function" 'testmark: testmarktest_rule_basic: "cat" is not a defined function' testmarktest_rule_basic '*/in' cat +expect_fail "provide: no arguments" 'testmark: usage: testmarktest_provide FN...' testmarktest_provide +expect_fail "provide: not a function" 'testmark: testmarktest_provide: "nope" is not a defined function' testmarktest_provide defined nope +expect_fail "run: no arguments" 'testmark: usage: testmarktest_run DOC...' testmarktest_run +expect_eq "refused calls leave no rules" "${#_testmarktest_fns[@]} ${#_testmarktest_provided[@]}" "0 0" +got=$("$BASH" -c 'source "$1"; f() { :; }; testmarktest_rule_basic "*" f; testmarktest_run doc.md' _ "$ROOT/lib/testmark.bash" 2>&1) || got+=" (status $?)" +expect_eq "run: not from a script file" "$got" 'testmark: testmarktest_run: no script file to find the docs beside; run it from a script (status 1)' + +exit "$RESULT"