Results and logs¶
Every run writes into its run directory, which
[project] output_dir chooses, or
--output-dir at the terminal.
The run directory¶
A new run needs a run directory that is missing or empty. Testuary refuses a
directory that already holds files and deletes nothing in it, neither earlier
results nor logs nor unrelated files. Inside the project you test, only its
.testuary/ directory may hold results. A path that passes through a link or
a reparse point, at any level, is refused.
After a run the directory holds:
results/
tests.csv one row per target and repetition with its scores and counts
summary.json the summaries of all targets
completion.json the end-of-run summary the console printed
<target>/
phase_state.json which phases of each repetition are finished
summary.json the summary across the target's repetitions
r0/ the first repetition, r1/ the second, and so on
tests/ the delivered suite and its measurements
archive/ the suites and records of each phase
resolved_settings.json the settings the repetition used
summary.json the repetition's final values
time_limits.json the time limits the repetition reached, only when it reached one
logs/
run.log every log line of the run, as plain text
main.jsonl the run's structured log
test/<target>/r0/ the logs of each repetition, including the model request records
_work/ temporary work directory, empty after the run
Each target gets a directory named after its path, for example
src_main_java_example_Calculator for src/main/java/example/Calculator.java.
Each repetition of the target has a directory inside it, r0 for the first. A
single repetition is r0, and --runs 3 makes r0, r1 and r2. The
archive/ directory keeps the starter suites and the suites of the coverage
phase and the mutation phase, so you can compare them with the delivered suite.
The result files reference lists the columns of
tests.csv and the fields of the summaries.
Several repetitions¶
--runs and --run repeat each
target's phases, each repetition in its own directory. --run adds a
repetition later or runs a failed one again. Each repetition's
starter suites use their own
sampling seeds, so a
repetition draws the same seeds whether it runs alone or with others.
The delivered suite¶
For a Python target, the delivered suite holds the test file, the
conftest.py it was measured with and a pytest.ini. The conftest.py
makes the project importable from the directory the tests were measured in.
The pytest.ini holds only an empty [pytest] section. It makes the test
directory pytest's root directory, so that a pytest configuration in a
directory above it does not apply, and every Python target's suite gets it.
The project's own pytest configuration is not part of the delivered suite:
each test run reads an exact copy of it beside the run's copy of the suite.
Every question whether tests pass is answered by one normal test run. For
a Python target, the normal run is the run as delivered: the test files run
one after another, each in its own pytest session, its tests in the order
pytest collects them. Each session hashes strings with the fixed seed 1
(PYTHONHASHSEED=1). When it starts, the global random generators of
Python, NumPy and SymPy get the same seed, 1, and so does Hypothesis
unless the project's pytest configuration sets one. A generator that a
test makes itself is not seeded. For this the session imports NumPy and
SymPy where they are installed, which a user's run does not do. To repeat
a Python suite's check as Testuary ran it, run each test file in its own
pytest session with PYTHONHASHSEED=1 and these generators seeded with 1
when the session starts. For a Java or C file the normal run is the
language's own test run, which takes no seed. Nothing resets state between
tests, as in a user's run, so an accepted suite does not depend on a
cleaner that the user does not have. A test file of the delivered suite can need
its conftest.py: keep it with the tests. The starter suites' cleanup removes
the tests that make a suite fail its normal run, since the model never
works on those suites. When no starter suite passes and is fully measured,
the run stops. At the end of a loop, a suite that fails its normal run is
no candidate, no test is removed from it, and the record names the failing
tests. The suite the loop received stays a candidate, so the end of a loop
always has one to choose. A loop does not check the suite it inherits
before it starts: its first measurement shows the model what the suite
does. In the mutation phase, a suite that fails its normal run there, before
any edit, ends the phase as skipped. A pass holds for that run, not for
other test orders, subsets or environments.
A passing test run of a loop can carry a "STANDS ALONE" note for the model.
For Python it names the tests that left the module under test or a
library setting changed. With report_reverse_order_failures = true in
[coverage] or [mutation], a passing Python run of that loop is also
followed by a run of the file as delivered in reverse order, and the note
names the tests that fail there, or says why the file could not run in
that order. A language whose tests do not run in reverse order, such as
Java, gets no such run and no note. A note decides no test's outcome.
After a note naming tests, the model can change the tests before it
measures, and a loop at its goal ends only after a passing run without
such a note. The loop's request, token and time limits still end it, so a
delivered suite can keep such a test.
Put the delivered suite's test file next to the project's own tests, not inside its package.
A project's rules for its own modules can apply to a file placed there. For
example, xarray requires __slots__ on every subclass of its types defined in
an xarray.* module, and a test module inside the package becomes such a
module.
The end of a run¶
At the end, the run prints a summary. A target has a suite that meets the
minimums when every phase handed on a suite and the suite reaches the minimum
coverage and mutation score you configured (the settings minimum in
[coverage] and minimum in [mutation]). The summary's first line counts
the targets with a suite that meets the minimums, the targets without one and the
targets that failed to execute. One block per target follows with its outcome and
reason, the delivered suite, its line coverage, branch coverage and mutation score
when the target had one repetition, its model use, and any fallback (a step
skipped because the language lacks it) or warning of its repetitions. When
targets are unfinished and the run can be continued, the summary
ends with the exact command that continues it. The same list of unfinished
targets is in the log.
The exit status says how the command ended:
- Status 0: The run finished. A suite below a configured minimum, a phase that ended without a suite and a target an earlier run in this directory had finished are results, recorded in tests.csv.
- Status 1: A target failed to execute: it crashed, could not be set up, copied, restored or delivered, or some generated starter suites could not be checked, because running them timed out or did not start. The summary and the run's log say why.
- Status 2: A setting or option was refused, or a required tool was missing, before any work started.
- Status 130: The run was stopped with Ctrl+C. Work that already finished is kept, and the run can be continued in its run directory (at the terminal: --resume).
The replies of starter suites that the validity check could not judge are
kept and checked again with --resume. Outcomes says
how the terminal shows each target's outcome. A run you interrupt with Ctrl+C
prints "Stopped by you." with what you can do next.
Continuing a run¶
To continue an interrupted run, run the same command again with --resume and
the same run directory. For each repetition of each target, Testuary skips the
phases that are finished and resumes from the first one that is not. A phase
counts as finished only when its checkpoint, the hash of its recorded results and
the hash of the target's canonical source (its file in the source directory),
which the repetition recorded when the phase started, all match. A later phase is never trusted
past an earlier gap. A repetition that is finished is kept, so --resume with
a higher --runs count adds only the missing repetitions.
To reuse saved phases of an earlier run instead of running them again, use
--reuse with a comma list of phase names: starter for the starter
suites, coverage, mutation and finishing for Finishing. Reusing the starter suites,
the coverage phase or the mutation phase keeps their saved results. A phase is
reused with the phases it builds on: coverage reuses the starter suites
too, and mutation the coverage phase and the starter suites. For example,
--reuse coverage runs the mutation phase again on the coverage phase's saved
suite. finishing only skips Finishing and reuses no results. A reuse takes
the saved results of the first repetition, r0, for every repetition.
finishing skips Finishing, which only a run itself makes, so it works with
the first repetition alone. The
command line reference says which
options --resume combines with.
A run directory without generation records, or with records in an unknown format, cannot be resumed or reused. The run refuses before any work, says so, and asks for a new run in an empty run directory. A resume or reuse also refuses changed generation conditions, such as another provider, model, prompts or limits. Another address of the same model service is a warning, not a refusal, as models and prompts describes.
If the temporary work directory cannot be cleaned, the run stops rather than continuing over leftover work.
Logs¶
logs/run.log holds every log line of the run as plain text and can be read
while the run is going. The terminal names it from the moment it opens.
Each repetition of a target has its own logs, with the model conversations
of its loops. The result files reference
lists the log files and what each holds.
Model request records¶
Every model request sent is recorded with its outcome and whether the
model service reported its token usage: the loops' records are among the
repetition's logs, and the starter
suites' records are in their
archive. The
result files reference
describes the request and token columns of tests.csv.