testmark-bash ============= testmark-bash reads, checks, and patches [testmark](https://testmark.dev/) documents, from bash. Testmark is data in markdown: a fenced code block, named by a comment line just above it. It makes a markdown file a good home for test fixtures: the prose around them explains them, they render nicely anywhere markdown does, and yet a program can easily read the data hunks (and write and rewrite them, on demand). With this library, a simple bash script can either use markdown files for plain data storage... Or... go all the way into literate test-driven development with fixtures: check any command or script's output against a block of data in a markdown file; AND have an easy way to regenerate the block's content from that selfsame script on demand, which makes it easy to update fixtures when output changes on purpose. ```bash source path/to/lib/testmark.bash exercise_mytool() { mytool --read-stdin } testmarktest_rule_basic '*/input' exercise_mytool testmarktest_run fixtures.md ``` That's a whole test file. Every hunk in `fixtures.md` named `SOMETHING/input` is a case: it's fed to `exercise_mytool`, and what that prints is checked against the hunk named `SOMETHING/output`. Adding a case is adding two blocks to the doc. It needs bash 4.4 or later. The workflow ------------ Write the input, and an empty block for the output to go in: ``` [testmark]:# (basic/input) ```text # TODO: handle spaces in names ``` [testmark]:# (basic/output) ```text ``` ``` Run the test. It fails, and shows a diff of what your tool printed against the empty block. Run it again with `TESTMARK_REGEN=1` set. The output gets written into the block. Read it with `git diff`. If it's right, commit it. Later, when the tool's output changes on purpose, it's the same: regen, read the diff, commit. The diff is the review. Writing a test -------------- A test file makes rules, then runs them over its docs. **A rule** is a glob and a function: `testmarktest_rule_basic GLOB FN`. Every hunk whose name matches GLOB triggers a case, and FN exercises it. The case is the hunk's name minus its last segment, so `greeting/input` is the case `greeting`, and its output goes in `greeting/output`. In the glob, `*` matches `/` too, so `'*/input'` finds `a/b/input` as well. **The function** is where your tool gets run. We call ours `exercise_SOMETHING`, by convention. Each case runs it in a fresh bash, so it's a clean shell: - stdin is the trigger hunk's body, exactly. - `set -euo pipefail` is on. A failing command stops the case, and the failure shows up in its output. Bash's own error messages there are labeled with the function's name, nothing more, so they read the same wherever the test runs. - The working directory is the test script's, so a relative path like `./mytool` works, wherever the test is run from. `$TESTCASE_DIR` holds that directory too, for after a `cd`. - It sees exported variables (like `PATH`), but no other variables, and no other functions. If it calls helpers, pass them to `testmarktest_provide helper1 helper2` first. **What gets checked** is everything the function prints, stdout and stderr together, in order. If the function exits nonzero, a last line `exit status: N` is added, so a case can't pass while its tool fails, unless the fixture says it should. On success nothing is added: the fixture for a tool that works is just its output. Line endings don't count. CRLF reads as LF, and a missing final newline is added, in hunks and in output alike. **`testmarktest_run DOC...`** runs every case, prints a `PASS` or `FAIL` line for each, and writes the docs if `TESTMARK_REGEN` is set. Doc paths are relative to the test script. It fails if any case failed, and also if a rule matched no hunk at all, or a doc matched no rule: a typo in a glob can't pass by running nothing. A doc can hold several kinds of cases. Give each kind its own rule: ```bash testmarktest_rule_basic '*/script' exercise_script testmarktest_rule_basic '*/input' exercise_mytool ``` Nothing adds hunks. The output hunk must exist already, even if empty: regen fills in a block that's there, and a missing one fails the case. That's why the workflow starts with an empty block. A whole example --------------- [`examples/`](examples/) has a small tool, `todos.sh`, a fixtures doc for it, `todos.md`, and this test, which `make test` runs: ```bash #!/bin/bash ## Checks todos.sh against every case in todos.md. ## Run with TESTMARK_REGEN=1 to update todos.md to match what todos.sh does now; then review the change with `git diff`. source "$(dirname "${BASH_SOURCE[0]}")/../lib/testmark.bash" exercise_todos() { ./todos.sh /dev/stdin } testmarktest_rule_basic '*/input' exercise_todos testmarktest_run todos.md ``` **Normalize, then check exactly.** If your tool's output has anything that changes from run to run, like a temp dir or a timing, replace it in the function: ```bash exercise_build() { mytool build | sed 's/in [0-9.]*s$/in Ns/' } ``` Then the check can stay exact. Prefer this over grepping for a fragment of the output: an exact check can't pass by accident, it shows the reader what the output really looks like, and regen can write it. More than one hunk in or out ---------------------------- When a case needs several hunks, or gives several outputs, make its rule with `testmarktest_rule` instead. Its function gets nothing on stdin; it reads and writes hunks itself, with `testcase_*` functions. [`examples/test-todos-files.sh`](examples/test-todos-files.sh) runs `todos.sh` on two files at once: ```bash exercise_todos_files() { testcase_withtmpdir testcase_read_myhunk mine.txt >mine.txt testcase_read_anyhunk shared/theirs.txt >theirs.txt "$TESTCASE_DIR/todos.sh" mine.txt theirs.txt | testcase_output output } testmarktest_rule '*/mine.txt' exercise_todos_files testmarktest_run todos-files.md ``` - `testcase_read_myhunk NAME` prints the case's hunk `CASE/NAME`. - `testcase_read_anyhunk NAME` prints any hunk in the doc, by its whole name: for hunks many cases share. - `testcase_output NAME` checks what it reads against `CASE/NAME`. Call it once per output. - `testcase_withtmpdir` moves the case into a new temp dir, removed when the case ends. It's the only one that makes files. `$TESTCASE_DIR` still holds the test script's directory, for reaching the tool, like above. Anything else the function prints, or a nonzero exit, has no hunk to go in, so it fails the case. The fixture for a case that works is just its outputs. Functions --------- | function | does | | --- | --- | | `testmarktest_rule_basic GLOB FN` | makes each hunk matching GLOB a case, exercised by FN, with the hunk on stdin, and checked against the case's `output` hunk | | `testmarktest_rule GLOB FN` | makes each hunk matching GLOB a case, exercised by FN, which reads hunks and gives outputs with the `testcase_*` functions | | `testmarktest_provide FN...` | sends more functions into every case, for exercise functions that call them | | `testmarktest_run DOC...` | runs every case in the docs, reports each, and writes the docs under `TESTMARK_REGEN` | Errors come back as a nonzero status, with a `testmark: ...` line on stderr. The doc comments in [`lib/testmark.bash`](lib/testmark.bash) are the full reference. Going it alone -------------- The runner is built on primitives you can call yourself: to read and rewrite testmark docs as plain data, or to check output in a shape the runner doesn't have. [docs/userguide/going-it-alone.md](docs/userguide/going-it-alone.md) covers them. Requirements ------------ Bash 4.4 or later. Older bashes lack things the library is built on: `local -`, for one, which lets it set shell options for itself and put yours back, arrived in 4.4. [docs/hacking/bash-versions.md](docs/hacking/bash-versions.md) has more. On an older bash, sourcing prints `testmark: needs bash 4.4 or later; this is 3.2.57(1)-release` and returns 1. (That's macOS's `/bin/bash`. Use a newer bash from a package manager.) Nothing else, except `diff`, to show a mismatch. The library doesn't depend on your shell options or locale; it sets its own on every call, and puts yours back. Contributing ------------ Run `make setup` once after cloning: it activates the git hooks. Run `make` to see all commands. Before committing anything, run `make all`. There should be no warning messages. How the library works, and the bash pitfalls it steers around, are in [docs/hacking/](docs/hacking/). License ------- SPDX-License-Identifier: Apache-2.0 OR MIT