Skip to content

Exploratory testing: gates, baselines, and changed lines

Date: 2026-11-02.

Result

No product bug was confirmed in these journeys. No issue was filed. The pass used commit 1879275, after the merge of pull request #231. It did not test the changes in the open pull requests. The existing worker lock issue #216 and report path issue #199 were not reproduced or refiled in this pass.

Setup

The pass built the Linux debug binary with cargo build --locked -j 2 in a container based on rust:1.93-slim. The container had two CPUs and 2 GiB of memory. This differs from the repository recommendation of one CPU and 512 MiB. The pass makes no timing comparison. The build used the repository Cargo.lock. The runs used one Mutarust worker and had no network access. Cargo used offline mode for the consumer package. Cargo cache, build files, temporary mutation areas, and the consumer package were inside the worktree. The consumer package had no external dependencies.

The replay script creates the consumer package and a local Git repository. It selects only conditional/bool-literal. The source has two functions, and a unit test checks one of them. The script creates a clean package at the start and resets the source before the changed-line journey. The script records command output and report content in a new directory under .scratch. The files linked below record the second run. It matched the first run except for the agent-ready report timestamp. The comparison hashes exclude that timestamp. After the replay, the script was changed to write to a new scratch directory instead of replacing the evidence. A third run of the changed script matched the recorded output, apart from the timestamp. Use a disposable worktree. To replay, build the image and binary, then run the script without network access:

docker build -t mutarust-explore docs/exploratory-testing/2026-11-02-gates-baseline-scope
mkdir -p .scratch/cargo .scratch/tmp .scratch/home
docker run --rm --cpus=2 --memory=2g --user "$(id -u):$(id -g)" -v "$PWD:/work" -w /work \
  -e HOME=/work/.scratch/home -e CARGO_HOME=/work/.scratch/cargo \
  -e CARGO_TARGET_DIR=/work/.scratch/target -e TMPDIR=/work/.scratch/tmp \
  mutarust-explore cargo build --locked -j 2
docker run --rm --cpus=2 --memory=2g --network=none --user "$(id -u):$(id -g)" \
  -v "$PWD:/work" -w /work -e HOME=/work/.scratch/home \
  -e CARGO_HOME=/work/.scratch/cargo \
  mutarust-explore sh docs/exploratory-testing/2026-11-02-gates-baseline-scope/replay.sh

The first attempt used an image without Git and stopped before the journeys. After Git was installed, the first source used pub fn checked() -> bool { true } and pub fn unchecked() -> bool { true }. That produced zero mutants. The initial output records the result. The mutator guide limits this mutator to specific expression positions, so this was not classified as a product bug. The replay script uses local initializers instead.

Journey 1: reports and the mutation score gate

The goal was to find a weak test and use the mutation score as a gate. With one tested function and one untested function, the normal run reported one killed and one escaped mutant, a 50% mutation score, and exit value 0. The full, compact, agent-ready, and GitLab JSON outputs showed one escaped mutant. The three reports that contain stable mutant IDs showed the same ID. The HTML report contained the untested function. A second run with --min-msi 100 returned 4. The source SHA-256 before and after the runs was the same. See journey 1 output.

Journey 2: accept an existing escape

The goal was to accept one known escape without accepting a new one. The baseline update wrote the stable mutant ID of the escape. A run with that baseline and --fail-on-escaped returned 0. After the script added another untested function, the same command returned 4 and reported one new escaped mutant. The baseline file still contained only the original ID. See journey 2 output.

Journey 3: select changed lines

The goal was to test only the changed source. A dry run against HEAD with no source changes selected zero mutants. After the script added an untested function, a normal changed-line run selected one escaped mutant. The recorded Git diff before and after the run was the same. See journey 3 output.

Limits and observations

The pass used one mutator and one small crate with Rust 1.93. It did not exercise LLVM coverage, per-test coverage, parallel workers, custom test commands, or large workspaces. The changed-line journey used an unstaged change against HEAD; it did not test staged changes, commits, or renames. The baseline journey did not move the original escaped line. It did not verify HTML rendering in a browser or validate every field against the published schemas. The report checks read the files and compared their visible counts and IDs. The empty first run illustrates that a selected mutator does not necessarily change every literal. The mutator guide documents the supported positions.