testmark-bash ============= testmark-bash reads, checks, and patches [testmark](https://testmark.dev/) documents, from bash. Testmark is data in markdown: a fenced code block, named by a comment line just above it. It makes a markdown file a good home for test fixtures: the prose around them explains them, they render nicely anywhere markdown does, and yet a program can easily read the data hunks (and write and rewrite them, on demand). With this library, a simple bash script can either use markdown files for plain data storage... Or... go all the way into literate test-driven development with fixtures: check any command or script's output against a block of data in a markdown file; AND have an easy way to regenerate the block's content from that selfsame script on demand, which makes it easy to update fixtures when output changes on purpose. ```bash source path/to/lib/testmark.bash exercise_mytool() { mytool --read-stdin } testmarktest_rule_basic '*/input' exercise_mytool testmarktest_run fixtures.md ``` That's a whole test file. Every hunk in `fixtures.md` named `SOMETHING/input` is a case: it's fed to `exercise_mytool`, and what that prints is checked against the hunk named `SOMETHING/output`. Adding a case is adding two blocks to the doc. It needs bash 4.4 or later. The workflow ------------ Write the input, and an empty block for the output to go in: ``` [testmark]:# (basic/input) ```text # TODO: handle spaces in names ``` [testmark]:# (basic/output) ```text ``` ``` Run the test. It fails, and shows a diff of what your tool printed against the empty block. Run it again with `TESTMARK_REGEN=1` set. The output gets written into the block. Read it with `git diff`. If it's right, commit it. Later, when the tool's output changes on purpose, it's the same: regen, read the diff, commit. The diff is the review. Writing a test -------------- A test file makes rules, then runs them over its docs. **A rule** is a glob and a function: `testmarktest_rule_basic GLOB FN`. Every hunk whose name matches GLOB triggers a case, and FN exercises it. The case is the hunk's name minus its last segment, so `greeting/input` is the case `greeting`, and its output goes in `greeting/output`. In the glob, `*` matches `/` too, so `'*/input'` finds `a/b/input` as well. **The function** is where your tool gets run. We call ours `exercise_SOMETHING`, by convention. Each case runs it in a fresh bash, so it's a clean shell: - stdin is the trigger hunk's body, exactly. - `set -euo pipefail` is on. A failing command stops the case, and the failure shows up in its output. - The working directory is the test script's, so a relative path like `./mytool` works, wherever the test is run from. - It sees exported variables (like `PATH`), but no other variables, and no other functions. If it calls helpers, pass them to `testmarktest_provide helper1 helper2` first. **What gets checked** is everything the function prints, stdout and stderr together, in order. Two things get added, only when they're needed, so the fixture for a tool that works is just its output: - If the function exits nonzero, a last line `exit status: N`. A case can't pass while its tool fails, unless the fixture says it should. - If the output doesn't end in a newline, it gets one, then a line `\ No newline at end of output`. Output with CRLF line endings fails the case; normalize it in the function, with `tr -d '\r'`. **`testmarktest_run DOC...`** runs every case, prints a `PASS` or `FAIL` line for each, and writes the docs if `TESTMARK_REGEN` is set. Doc paths are relative to the test script. It fails if any case failed, and also if a rule matched no hunk at all, or a doc matched no rule: a typo in a glob can't pass by running nothing. A doc can hold several kinds of cases. Give each kind its own rule: ```bash testmarktest_rule_basic '*/script' exercise_script testmarktest_rule_basic '*/input' exercise_mytool ``` The output hunk must exist already, even if empty: regen fills in a block that's there, and a missing one fails the case. That's why the workflow starts with an empty block. A whole example --------------- [`examples/`](examples/) has a small tool, `todos.sh`, a fixtures doc for it, `todos.md`, and this test, which `make test` runs: ```bash #!/bin/bash ## Checks todos.sh against every case in todos.md. ## Run with TESTMARK_REGEN=1 to update todos.md to match what todos.sh does now; then review the change with `git diff`. source "$(dirname "$0")/../lib/testmark.bash" exercise_todos() { ./todos.sh /dev/stdin } testmarktest_rule_basic '*/input' exercise_todos testmarktest_run todos.md ``` **Normalize, then check exactly.** If your tool's output has anything that changes from run to run, like a temp dir or a timing, replace it in the function: ```bash exercise_build() { mytool build | sed 's/in [0-9.]*s$/in Ns/' } ``` Then the check can stay exact. Prefer this over grepping for a fragment of the output: an exact check can't pass by accident, it shows the reader what the output really looks like, and regen can write it. Functions --------- | function | does | | --- | --- | | `testmarktest_rule_basic GLOB FN` | makes each hunk matching GLOB a case, exercised by FN, with the hunk on stdin, and checked against the case's `output` hunk | | `testmarktest_provide FN...` | sends more functions into every case, for exercise functions that call them | | `testmarktest_run DOC...` | runs every case in the docs, reports each, and writes the docs under `TESTMARK_REGEN` | Errors come back as a nonzero status, with a `testmark: ...` line on stderr. The doc comments in [`lib/testmark.bash`](lib/testmark.bash) are the full reference. Going it alone -------------- The runner is built on primitives you can call yourself, when you want something it doesn't do: | function | does | | --- | --- | | `testmark_parse HANDLE FILE` | parses FILE, and names it HANDLE for the other calls | | `testmark_names HANDLE` | prints the hunk names, one per line, in document order | | `testmark_body HANDLE NAME` | prints a hunk's body, exactly | | `testmark_info HANDLE NAME` | prints a hunk's info string (`text` in ```` ```text ````) | | `testmark_check HANDLE NAME