testmark-bash🔗

testmark-bash reads, checks, and patches testmark documents, from bash.

Testmark is data in markdown: a fenced code block, named by a comment line just above it. That makes a markdown file a good home for test fixtures: the prose around them explains them, they render nicely anywhere markdown does, and a program can rewrite them.

So with this library, a bash test can check a tool's output against a block in a markdown file; and when the output changes on purpose, rewrite the block, and leave you a git diff to review.

source path/to/lib/testmark.bash

testmark_parse doc fixtures.md
testmark_check doc greeting/output < <(mytool greet world)
testmark_write doc

It needs bash 4.4 or later.

The workflow🔗

Write the input, and an empty block for the output to go in:

	[testmark]:# (basic/input)
	```text
	# TODO: handle spaces in names
	```

	[testmark]:# (basic/output)
	```text
	```

Run the test. It fails, and shows a diff of what your tool printed against the empty block.

Run it again with TESTMARK_REGEN=1 set. The output gets written into the block.

Read it with git diff. If it's right, commit it.

Later, when the tool's output changes on purpose, it's the same: regen, read the diff, commit. The diff is the review.

A whole example🔗

examples/ has a small tool, todos.sh, a fixtures doc for it, todos.md, and this test, which make test runs:

#!/bin/bash
## Checks todos.sh against every case in todos.md.
## Run with TESTMARK_REGEN=1 to update todos.md to match what todos.sh does now; then review the change with `git diff`.
set -euo pipefail
cd "$(dirname "$0")"
source ../lib/testmark.bash

work=$(mktemp -d)
trap 'rm -rf "$work"' EXIT

testmark_parse doc todos.md
mapfile -t names < <(testmark_names doc)
fails=0
for name in "${names[@]}"; do
	[[ $name == */input ]] || continue
	case=${name%/input}
	testmark_body doc "$case/input" >"$work/notes.txt"
	if testmark_check doc "$case/output" < <(./todos.sh "$work/notes.txt" | sed "s|$work|WORK|g"); then
		echo "PASS  $case"
	else
		echo "FAIL  $case"
		fails=$((fails + 1))
	fi
done
testmark_write doc
((fails == 0))

What to copy from it:

• One case per name. Hunks named CASE/input and CASE/output make a table; adding a case is adding two blocks to the doc, with no change to the test.
• Normalize, then check exactly. Anything that changes from run to run, like a temp dir, gets replaced before the check (here, with sed), so the check can stay exact. Prefer this over grepping for a fragment of the output: an exact check can't pass by accident, it shows the reader what the output really looks like, and regen can write it.
• < <(...) into the check. Not a pipe; see the rules below.
• testmark_write at the end. That's what makes TESTMARK_REGEN=1 write anything. With nothing to write, it does nothing, so call it unconditionally.

A check doesn't see the tool's exit status. When it matters, put it in the output: < <(mytool || echo "exit: $?") adds a line only when the tool fails.

Rules to know🔗

State changes happen in the shell that parsed. testmark_check, testmark_patch, and testmark_write refuse to run in a subshell, where what they stage would be lost. So feed a check with testmark_check doc NAME < <(cmd), not cmd | testmark_check doc NAME; and don't call these inside $(...), or inside a test helper that runs its command there.

Output must be empty, or end in a newline, with no CRLF. A hunk's body can't hold anything else, so testmark_check rejects it outright, rather than failing forever. Normalize it first.

A check never creates a hunk. Regen rewrites the body of a hunk that's there; a missing one is an error in both modes. That's why the workflow starts with an empty block.

Functions🔗

function	does
testmark_parse HANDLE FILE	parses FILE, and names it HANDLE for the other calls
testmark_names HANDLE	prints the hunk names, one per line, in document order
testmark_body HANDLE NAME	prints a hunk's body, exactly
testmark_info HANDLE NAME	prints a hunk's info string (text in ```text)
testmark_check HANDLE NAME <OUTPUT	compares OUTPUT to the body: returns 1 and shows a diff on a mismatch; or under TESTMARK_REGEN, stages OUTPUT as the new body
testmark_patch HANDLE NAME [INFO] <BODY	stages a new body, and optionally a new info string
testmark_write HANDLE	writes what's staged into the file, in place

Errors come back as a nonzero status, with a testmark: ... line on stderr. The doc comments in lib/testmark.bash are the full reference.

Requirements🔗

Bash 4.4 or later. Older bashes lack things the library is built on: local -, for one, which lets it set shell options for itself and put yours back, arrived in 4.4. docs/hacking/bash-versions.md has more. On an older bash, sourcing prints testmark: needs bash 4.4 or later; this is 3.2.57(1)-release and returns 1. (That's macOS's /bin/bash. Use a newer bash from a package manager.)

Nothing else, except diff, to show a mismatch. The library doesn't depend on your shell options or locale; it sets its own on every call, and puts yours back.

Contributing🔗

Run make setup once after cloning: it activates the git hooks. Run make to see all commands. Before committing anything, run make all. There should be no warning messages.

How the library works, and the bash pitfalls it steers around, are in docs/hacking/.

License🔗

SPDX-License-Identifier: Apache-2.0 OR MIT