Exploratory testing: core journeys (2026-09-26)¶
This report gives the results of one exploratory testing pass. The pass used
the mutarust command, which is the supported interface, on small scratch
Cargo packages. The pass found two confirmed bugs:
- #198: Users cannot get a blacklist checksum from any Mutarust output.
- #199: CI report paths are relative to the Cargo workspace, not the Git repository.
Set-up¶
| Item | Value |
|---|---|
| Build | mutarust 0.1.9, commit 406beab. Linux release binary (cargo build --release --locked) in rust:1.85-slim. See et.Dockerfile. |
| Coverage build | Same image, with llvm-tools-preview and cargo-llvm-cov 0.6.16. See cov.Dockerfile. |
| Re-check | The two bugs were also replayed on main at e599200 (after #197). See replay-e599200.txt. |
| Limits | --cpus=1 --memory=512m --network=none for each run. The coverage runs used --memory=1g. See run.sh. |
| Starting state | A new Git repository with the pricing package: four functions and three weak unit tests. A two-member Cargo workspace (core, app) with tests in tests/. A spin package with a loop that does not stop when mutated. |
| Configuration | No configuration file, except where a step names --config. |
| Evidence | docs/exploratory-testing/2026-09-26-core-journeys/ |
Readiness check: mutarust --version printed mutarust 0.1.9 in the capped
container.
Journeys¶
J1: Find weak tests and apply a score gate¶
Goal: the user runs mutarust ., sees the escaped mutants, and sets a score
gate. The tool must not change the user source tree.
| Step | Result | Evidence |
|---|---|---|
--dry-run, --list-files, then mutarust . |
38 mutants: 14 killed, 20 escaped, 4 skipped. Score 47.37% = (14 + 4) / 38, as docs/cli.md specifies. Each escaped mutant is a real gap in the tests. git status is clean. No target/ or Cargo.lock in the user package. |
j1-first-run.txt |
--min-msi 48, then --min-msi 47 |
Exit value 4 with a clear message, then exit value 0. The limit is correct. | j1-gates-reports.txt |
| All report outputs | report.json, mutarust-report.html, mutarust-summary.json, mutarust-agentic.json, and mutarust-gitlab.json were written. Their counts agree with the terminal. |
j1-gates-reports.txt |
--coverage --min-covered-msi 60 |
10 mutants are not covered. They are exactly the lines that no test calls (sum, return 99). Covered-code score 57.14% = 16 / 28. Exit value 4. --per-test gives the same results as the run without coverage. |
j1-coverage.txt |
Two-member workspace, mutarust ... |
Correct source names (core/src/lib.rs). The tests in tests/ killed the mutants. The user tree is not changed. |
j1-workspace.txt |
Mutant that does not stop, --timeout 5 |
5 errored mutants with "timed out after 5 seconds". No child process remains. /tmp is empty after the run. |
j1-timeout.txt |
| Incorrect input | An unknown configuration field, --workers 0, an incorrect --output-statuses, an unknown package, and --min-msi 101 each give exit value 3 with a clear cause. |
j1-config-errors.txt |
J2: Accept the current escaped mutants on a legacy crate¶
Goal: record a baseline. Then fail only on new escaped mutants. Accept one
false positive with --blacklist.
| Step | Result | Evidence |
|---|---|---|
--update-baseline, then --baseline ... --fail-on-escaped |
20 IDs were written. The next run gave exit value 0. | j2-baseline.txt |
| Add a new function at the top of the file (all lines move) | The old IDs stay the same. 3 new escaped mutants cause exit value 4. | j2-baseline-new-code.txt |
--run-mutant-id with a real ID and with an unknown ID |
The real ID shows only that mutant. The unknown ID gives exit value 3 with a clear message. | j2-baseline.txt |
--blacklist with the printed ID |
Failure (B1). No effect and no warning. No output contains a checksum. | j2-blacklist.txt |
J3: Test only changed lines in CI¶
Goal: in a pull request, mutate only the changed lines. Show the escaped mutants as CI annotations, and apply a score gate.
| Step | Result | Evidence |
|---|---|---|
Feature branch, --git-diff-lines --git-diff-base main |
Only lines 10 and 24 (the changed lines) were mutated. A comment-only change gave no mutants. | j3-git-diff.txt, j3-git-diff-run.txt |
Clone with origin/HEAD, default base, --logger-github --min-msi 50 |
The default base came from origin/HEAD. The run gave warnings and exit value 4. On main with no changes, the run gave exit value 4 without --ignore-msi-with-no-mutations and exit value 0 with it, as documented. |
j3-ci-clone.txt |
Package in a repository subdirectory (crates/pricing) |
Failure (B2). The line selection is correct, but the warnings and report paths are src/lib.rs, not crates/pricing/src/lib.rs. |
j3-subdir-crate.txt, j3-subdir-git-diff.txt |
Confirmed bugs¶
B1: Users cannot get a blacklist checksum (#198)¶
- User impact: the documented
--blacklistjourney fails. If a user puts the printedID:value in the file, it has no effect and Mutarust gives no warning. - Starting conditions: a new Cargo package with one weak test. No configuration.
- Replay: replay-blacklist.sh.
- Expected: an output gives the checksum for each mutant, or the documents
tell how to get it.
README.mdsays "Blacklist a known false positive by checksum". - Actual: the terminal output,
--debug, and all six report outputs do not contain the checksum. The blacklist works only with the MD5 value that the user calculates from the format insrc/evidence.rs. - Repeat observations: two replays from a clean state on
406beab, one one599200, and one in the scratch package. See replay-blacklist.txt.
B2: Report paths are relative to the Cargo workspace (#199)¶
- User impact: the documented CI command, used in a repository that has the Rust package in a subdirectory, gives GitHub warnings and GitLab Code Quality findings for a path that does not exist. The pull request cannot show them on the file.
- Starting conditions: a Git repository with one Cargo package at
rust/calc/. - Replay: replay-paths.sh.
- Expected:
file=rust/calc/src/lib.rs.docs/json-outputs.mdsays "Source names are repository-relative". - Actual:
file=src/lib.rsin the GitHub warnings,mutarust-gitlab.json, andmutarust-agentic.json. The cause issource_rootinsrc/execution.rs, which is the Cargo workspace root. - Repeat observations: two replays from a clean state on
406beab, one one599200, and three runs in the scratch repository. See replay-paths.txt.
Unresolved candidates¶
None.
Rejected candidates¶
test_filesis empty inmutarust-agentic.jsonwhen the tests are in#[cfg(test)]in the same file.docs/json-outputs.mddefines the field as "Nearby test file paths". The result agrees with the documents.coveredCodeMsiis0.0without--coverage. The documented example shows the same value.- In the terminal per-mutator table, the
Killedcolumn includes errored mutants (7 in the table, 2 in the summary, inj1-timeout.txt).docs/json-outputs.mddefines this formutatorStats, "as in Mutago". - A
--git-diff-linesrun stopped with "clean cargo test failed". The cause was the test scenario: the change broke a test. Mutarust must stop before mutation in this condition, asdocs/cli.mdsays. That output was replaced by the next run and is not kept as a file.
Usability observations¶
These are observations from the journeys. The suggestions are not bugs.
- The
--fail-on-escapedfailure says "3 new mutant(s) escaped" but does not identify them. The result lines do not mark new and accepted mutants differently. Suggestion: print the new IDs with the failure message. - A skipped mutant prints the full compiler error, which can be 20 lines for each mutant. In the first run, the compiler errors were 56 of 259 lines.
- In a repository with no
origin/HEAD,--git-diff-lineson a feature branch uses the current branch as the base. It finds only uncommitted changes and reports 0 mutants with exit value 0. This agrees with the documents, but a user can think that the run passed. docs/cli.mddoes not say that the terminal per-mutatorKilledcolumn includes errored mutants.
Areas not explored¶
--execcustom test commands,--no-exec,--do-not-remove-tmp-folder.--test-flags,--test-recursive,--timeout-coefficient,--workersmore than 1 (all runs used one CPU).- Custom mutators,
--print-ast, Bash completion, source annotations. - The HTML report was checked for its presence only. Its content was not examined in a browser.
- macOS host binaries. The pass used only Linux containers.
Limits of this pass¶
- Each run used one CPU and 512 MB of memory. Parallel-worker behaviour was not examined.
- Git operations in the container used
safe.directory "*"for the mounted scratch repository. This setting does not change Mutarust behaviour. - The coverage image build needed network access. The coverage runs did not.