testmark-bash🔗

testmark-bash reads, checks, and patches testmark documents, from bash.

Testmark is data in markdown: a fenced code block, named by a comment line just above it. It makes a markdown file a good home for test fixtures: the prose around them explains them, they render nicely anywhere markdown does, and yet a program can easily read the data hunks (and write and rewrite them, on demand).

With this library, a simple bash script can either use markdown files for plain data storage...

Or... go all the way into literate test-driven development with fixtures: check any command or script's output against a block of data in a markdown file; AND have an easy way to regenerate the block's content from that selfsame script on demand, which makes it easy to update fixtures when output changes on purpose.

source path/to/lib/testmark.bash

exercise_mytool() {
	mytool --read-stdin
}

testmarktest_rule_basic '*/input' exercise_mytool
testmarktest_run fixtures.md

That's a whole test file. Every hunk in fixtures.md named SOMETHING/input is a case: it's fed to exercise_mytool, and what that prints is checked against the hunk named SOMETHING/output. Adding a case is adding two blocks to the doc.

It needs bash 4.4 or later.

The workflow🔗

Write the input, and an empty block for the output to go in:

	[testmark]:# (basic/input)
	```text
	# TODO: handle spaces in names
	```

	[testmark]:# (basic/output)
	```text
	```

Run the test. It fails, and shows a diff of what your tool printed against the empty block.

Run it again with TESTMARK_REGEN=1 set. The output gets written into the block.

Read it with git diff. If it's right, commit it.

Later, when the tool's output changes on purpose, it's the same: regen, read the diff, commit. The diff is the review.

Writing a test🔗

A test file makes rules, then runs them over its docs.

A rule is a glob and a function: testmarktest_rule_basic GLOB FN. Every hunk whose name matches GLOB triggers a case, and FN exercises it. The case is the hunk's name minus its last segment, so greeting/input is the case greeting, and its output goes in greeting/output. In the glob, * matches / too, so '*/input' finds a/b/input as well.

The function is where your tool gets run. We call ours exercise_SOMETHING, by convention. Each case runs it in a fresh bash, so it's a clean shell:

• stdin is the trigger hunk's body, exactly.
• set -euo pipefail is on. A failing command stops the case, and the failure shows up in its output.
• The working directory is the test script's, so a relative path like ./mytool works, wherever the test is run from.
• It sees exported variables (like PATH), but no other variables, and no other functions. If it calls helpers, pass them to testmarktest_provide helper1 helper2 first.

What gets checked is everything the function prints, stdout and stderr together, in order. If the function exits nonzero, a last line exit status: N is added, so a case can't pass while its tool fails, unless the fixture says it should. On success nothing is added: the fixture for a tool that works is just its output.

Line endings don't count. CRLF reads as LF, and a missing final newline is added, in hunks and in output alike.

testmarktest_run DOC... runs every case, prints a PASS or FAIL line for each, and writes the docs if TESTMARK_REGEN is set. Doc paths are relative to the test script. It fails if any case failed, and also if a rule matched no hunk at all, or a doc matched no rule: a typo in a glob can't pass by running nothing.

A doc can hold several kinds of cases. Give each kind its own rule:

testmarktest_rule_basic '*/script' exercise_script
testmarktest_rule_basic '*/input' exercise_mytool

Nothing adds hunks. The output hunk must exist already, even if empty: regen fills in a block that's there, and a missing one fails the case. That's why the workflow starts with an empty block.

A whole example🔗

examples/ has a small tool, todos.sh, a fixtures doc for it, todos.md, and this test, which make test runs:

#!/bin/bash
## Checks todos.sh against every case in todos.md.
## Run with TESTMARK_REGEN=1 to update todos.md to match what todos.sh does now; then review the change with `git diff`.
source "$(dirname "${BASH_SOURCE[0]}")/../lib/testmark.bash"

exercise_todos() {
	./todos.sh /dev/stdin
}

testmarktest_rule_basic '*/input' exercise_todos
testmarktest_run todos.md

Normalize, then check exactly. If your tool's output has anything that changes from run to run, like a temp dir or a timing, replace it in the function:

exercise_build() {
	mytool build | sed 's/in [0-9.]*s$/in Ns/'
}

Then the check can stay exact. Prefer this over grepping for a fragment of the output: an exact check can't pass by accident, it shows the reader what the output really looks like, and regen can write it.

Functions🔗

function	does
testmarktest_rule_basic GLOB FN	makes each hunk matching GLOB a case, exercised by FN, with the hunk on stdin, and checked against the case's output hunk
testmarktest_provide FN...	sends more functions into every case, for exercise functions that call them
testmarktest_run DOC...	runs every case in the docs, reports each, and writes the docs under TESTMARK_REGEN

Errors come back as a nonzero status, with a testmark: ... line on stderr. The doc comments in lib/testmark.bash are the full reference.

Going it alone🔗

The runner is built on primitives you can call yourself: to read and rewrite testmark docs as plain data, or to check output in a shape the runner doesn't have. docs/userguide/going-it-alone.md covers them.

Requirements🔗

Bash 4.4 or later. Older bashes lack things the library is built on: local -, for one, which lets it set shell options for itself and put yours back, arrived in 4.4. docs/hacking/bash-versions.md has more. On an older bash, sourcing prints testmark: needs bash 4.4 or later; this is 3.2.57(1)-release and returns 1. (That's macOS's /bin/bash. Use a newer bash from a package manager.)

Nothing else, except diff, to show a mismatch. The library doesn't depend on your shell options or locale; it sets its own on every call, and puts yours back.

Contributing🔗

Run make setup once after cloning: it activates the git hooks. Run make to see all commands. Before committing anything, run make all. There should be no warning messages.

How the library works, and the bash pitfalls it steers around, are in docs/hacking/.

License🔗

SPDX-License-Identifier: Apache-2.0 OR MIT