testmark-bash ============= testmark-bash reads, checks, and patches [testmark](https://testmark.dev/) documents, from bash. Testmark is data in markdown: a fenced code block, named by a comment line just above it. That makes a markdown file a good home for test fixtures: the prose around them explains them, they render nicely anywhere markdown does, and a program can rewrite them. So with this library, a bash test can check a tool's output against a block in a markdown file; and when the output changes on purpose, rewrite the block, and leave you a `git diff` to review. ```bash source path/to/lib/testmark.bash testmark_parse doc fixtures.md testmark_check doc greeting/output < <(mytool greet world) testmark_write doc ``` It needs bash 4.4 or later. The workflow ------------ Write the input, and an empty block for the output to go in: ``` [testmark]:# (basic/input) ```text # TODO: handle spaces in names ``` [testmark]:# (basic/output) ```text ``` ``` Run the test. It fails, and shows a diff of what your tool printed against the empty block. Run it again with `TESTMARK_REGEN=1` set. The output gets written into the block. Read it with `git diff`. If it's right, commit it. Later, when the tool's output changes on purpose, it's the same: regen, read the diff, commit. The diff is the review. A whole example --------------- [`examples/`](examples/) has a small tool, `todos.sh`, a fixtures doc for it, `todos.md`, and this test, which `make test` runs: ```bash #!/bin/bash ## Checks todos.sh against every case in todos.md. ## Run with TESTMARK_REGEN=1 to update todos.md to match what todos.sh does now; then review the change with `git diff`. set -euo pipefail cd "$(dirname "$0")" source ../lib/testmark.bash work=$(mktemp -d) trap 'rm -rf "$work"' EXIT testmark_parse doc todos.md mapfile -t names < <(testmark_names doc) fails=0 for name in "${names[@]}"; do [[ $name == */input ]] || continue case=${name%/input} testmark_body doc "$case/input" >"$work/notes.txt" if testmark_check doc "$case/output" < <(./todos.sh "$work/notes.txt" | sed "s|$work|WORK|g"); then echo "PASS $case" else echo "FAIL $case" fails=$((fails + 1)) fi done testmark_write doc ((fails == 0)) ``` What to copy from it: - **One case per name.** Hunks named `CASE/input` and `CASE/output` make a table; adding a case is adding two blocks to the doc, with no change to the test. - **Normalize, then check exactly.** Anything that changes from run to run, like a temp dir, gets replaced before the check (here, with `sed`), so the check can stay exact. Prefer this over grepping for a fragment of the output: an exact check can't pass by accident, it shows the reader what the output really looks like, and regen can write it. - **`< <(...)` into the check.** Not a pipe; see the rules below. - **`testmark_write` at the end.** That's what makes `TESTMARK_REGEN=1` write anything. With nothing to write, it does nothing, so call it unconditionally. A check doesn't see the tool's exit status. When it matters, put it in the output: `< <(mytool || echo "exit: $?")` adds a line only when the tool fails. Rules to know ------------- **State changes happen in the shell that parsed.** `testmark_check`, `testmark_patch`, and `testmark_write` refuse to run in a subshell, where what they stage would be lost. So feed a check with `testmark_check doc NAME < <(cmd)`, not `cmd | testmark_check doc NAME`; and don't call these inside `$(...)`, or inside a test helper that runs its command there. **Output must be empty, or end in a newline, with no CRLF.** A hunk's body can't hold anything else, so `testmark_check` rejects it outright, rather than failing forever. Normalize it first. **A check never creates a hunk.** Regen rewrites the body of a hunk that's there; a missing one is an error in both modes. That's why the workflow starts with an empty block. Functions --------- | function | does | | --- | --- | | `testmark_parse HANDLE FILE` | parses FILE, and names it HANDLE for the other calls | | `testmark_names HANDLE` | prints the hunk names, one per line, in document order | | `testmark_body HANDLE NAME` | prints a hunk's body, exactly | | `testmark_info HANDLE NAME` | prints a hunk's info string (`text` in ```` ```text ````) | | `testmark_check HANDLE NAME