Expanding greatly in response to code review.
commit 3c24d09dbeec7051902b64d8e6836032576ae2f2 · parent 7978cc95fe
· browse this tree
Eric Myhre, 2022-11-27 01:08:04 UTC
Some typo fixes. Much more linking. A few more configuration hooks get mentions. Features have a pattern description before the literal example. Examples are more completely narrated. Added a whole section about script-vs-sequence (because the one tiny paranthetical about that wasn't even close to enough).
diff --git a/README.md b/README.md index c9a6101..5cb9187 100644 --- a/README.md +++ b/README.md @@ -308,7 +308,7 @@ There's a bunch more features too, like ways to check exit codes, ways to check stdout and stderr streams separately, ways to populate fixture filesystems (using hunks like `mytest/fs/dirname/filename.ext`), etc. -See that package, and its additional README, for more details! +See that package, and its additional [README](./testexec/README.md), for more details! Changelog diff --git a/testexec/README.md b/testexec/README.md index b771ace..5a96961 100644 --- a/testexec/README.md +++ b/testexec/README.md @@ -1,12 +1,12 @@ the testexec extension to testmark ================================== -See the README in the parent directory for a broader introduction to testmark as a format. +See the [README](../README.md) in the parent directory for a broader introduction to testmark as a format. "testexec" is a convention used within the testmark format to describe tests for executing processes. -Use the [well-known hunk names](well-known-hunk-names) to write your fixtures, +Use the [well-known hunk names](#well-known-hunk-names) to write your fixtures, then connect it to `go test` with the connecting the code you can see in the [example code](#examples). It supports [autopatching and fixture generation](#autopatching), and the [exec hooks are configurable](#configurable-exec-hooks). @@ -22,9 +22,8 @@ Firstly, either of these two forms can be used to specify the commands to test: - "`script`" -- will feed the entire hunk of text as a script to a shell (by default, bash). - "`sequence`" -- will run each line as a command as its own executable (parsing args by simply whitespace split). -("sequence" mode is meant to be a little less complicated to implement, and more portable, but is also a bit less flexible. -"script" mode can do whatever the shell does -- so, handle quoting, redirections, etc -- the only challenge is then you have to know what your shell does!) -i +(See [Script vs Sequence](#script-vs-sequence) for more on why there's two options.) + Second, there's the hunks that contain output assertions: - "`output`" -- if present, will cause the commands to be given a unified stdout and stderr buffer, and it will be checked against this data when done. @@ -47,10 +46,14 @@ Input streams (a.k.a. "stdin") can also be specified: Beyond this, you can also specify fixture filesystems: -- "`fs/somedir/thefile.ext`" -- (and names of that general pattern: anything starting with "fs/") +- "`fs/*`" -- everything under here will be placed in a (temporary!) working directory during the run. +- "`fs/somedir/thefile.ext`" -- for example, causes "somedir" to be created, and places "thefile.ext" inside it. + +And last of all, sequences of causally related tests can be created. +Any time a testexec script or sequence has siblings named "`then-*`", +that creates sub-tests. -And last of all, sequences of causally related tests can be created: -given a series of hunks with names like the following: +For example, given a series of hunks with names like the following: ``` test-one/script @@ -59,20 +62,32 @@ test-one/then-test-two/then-test-three/script test-one/then-test-two-b/script ``` -In a scenario like the above, the nested "then-*" tests will inherit the working directory from their parent -- +In the scenario above, a total of four tests will occur. +Each of the tests that are related by the nested "`then-*`" hunk naming will inherit the working directory from their parent -- or more specifically, a *copy* of it. -So: the script for `then-test-two` receives whatever the filesystem state left behind by `test-one` was; -the `then-test-three` script receives whatever filesystem state left behind by `then-test-two` was; -and `then-test-two-b` receives the filesystem as it was *after `test-one`* (! _not_ what was left by `then-test-three`). -(This lets you do one-time setup work, or build narratives -- while also still keeping tests isolated.) +Tests that *aren't* related don't receive copies of the working directories! +(This means you can make diverging tests of various filesystem state evolution stories easily.) + +Walking through the scenario above step by step: + +- the script for `test-one` executes first; +- the script for `then-test-two` receives whatever filesystem state was left behind by the evalution of `test-one`; +- the script for `then-test-three` receives whatever filesystem state was left behind by `then-test-two`; +- and the script for `then-test-two-b` receives the filesystem as it was *after `test-one`*. + - Note! _not_ what was left by `then-test-three`! The relationships are only explicit :) and the order of the testmark hunks doesn't matter. + +The "`then-*`" system (and its helpful support for filesystem forking), +lets you do one-time setup work, or build narratives -- while still keeping tests cleanly isolated and very explicit. Examples -------- -See the `selfexercise.md` file in this directory for fully worked examples. +See the [`selfexercise.md`](./selfexercise.md) file in this directory for fully worked examples. +(Note: if you're reading this on Github: You may also want to look at the raw mode of it! +You won't see the testmark annotations in the rendered markdown on Github!) -See the `testexec_test.go` file for the complete wiring to execute the examples -- +See the [`testexec_test.go`](./testexec_test.go) file for an example of the complete wiring to execute the examples -- including the autopatcher support. @@ -98,14 +113,63 @@ Note that regen mode will only update hunks that already exist; it won't add the (E.g., if you only have a `stdout` hunk, regen won't _add_ a `stderr` hunk.) -Configurable Exec Hooks ------------------------ +Script vs Sequence +------------------ + +The testexec convention covers two different kinds of instruction specification hunk: +"script" and "sequence". + +Each mode is similar, and the content of each looks roughly like a shell script, but they differ slightly. + +"sequence" mode is defined as just splitting words based on whitespace, and processing args that way. +It's meant to be a little less complicated to implement, and is a bit more portable (it's not invoking a shell!), +but of course it's also a bit less flexible. + +"script" mode is defined as doing whatever a shell does -- so, it should handle quoting, support redirections, etc. +In exchange, the challenge is then you have to know what your shell does! +(And internally, the default implementation is literally executing the host shell -- so there's some portability considerations, there.) + +In this library, each of these modes can be set up to do something different, by use of different callbacks when you're setting up the code -- +see the next section about [configurable exec hooks](#configurable-exec-hooks). + +In practice: we often find that the "sequence" mode's callback is easy to implement +and connect to the program's "main" method... +which makes it very easy to write tests that exercise a CLI-style application! +(The alternative would likely be doing a separate "go install" phase and invoking the application again as a subprocess, +but this is considerably more complicated to implement and maintain.) +But we still also find people usually reaching for "script" mode when testing how one program interacts with others, +because "sequence" mode is incapable of supporting piping programs together, etc. +Ultimately, both approaches have some attractions :) + + +Configuration Hooks +------------------- + +### Configurable Exec Hooks The default behaviors for this testexec implementation are: -- the exec mode does literally the system exec calls (e.g. for a single process). +- the sequence mode does literally the system exec calls (e.g. for a single process). - the script mode will invoke bash and feed the script to it. This is customizable in the Go code. See the `testexec.Tester` struct, and its fields `ExecFn` and `ScriptFn` for the two callbacks which can replace the default execution behaviors. + +### Bring your own assertion library + +... if you want. + +You can configure your choice of test equality checking function in the `testexec.Tester` struct by setting the `AssertFn` field. + +(The authors of this package happen to like `frankban/quicktest`, for example -- but we didn't want to force that choice on you!) + +The default function does some very simple string comparisons, +and has zero dependencies, +but does not provide rich diff output or any other nice-to-have features. + +### Filtrations + +Sometimes you want to test an application that is mostly predictable, but perhaps includes some unpredictable outputs, like timestamps for example. + +You can set the `FilterFn` field in the `testexec.Tester` struct to apply some normalizing transforms to the output streams before doing the comparisons.