Result files¶
Names in the records¶
Most record keys and tests.csv columns use the words of the guides. The
tables below give the meaning of those that do not: *_turns counts tool
calls, *_logical_requests counts model requests, and the status
budget_exhausted means that a request, token or time limit ended the loop.
The phases are starter, coverage, mutation and
finishing. The phase directories of a run's archive are archive/starter,
archive/coverage and archive/mutation, and a starter suite's directory is
archive/starter/suites/suite_NNN/. A phase-specific column starts with
its phase (coverage_requests, starter_status). The headline values,
line_coverage, branch_coverage and mutation_score, have no phase: they
describe the delivered suite. Repetitions count from 0: the first repetition's
directory is r0, its row in tests.csv has run 0, and the summaries name
it r0. Each target a run works on is numbered by its position in the
selection, from 1 (target_number).
Time columns of tests.csv end in _s (seconds) or _ms (milliseconds),
and the summaries name a time seconds. Many records hold t_mono_s, a
reading of the computer's monotonic clock in seconds, and clock_domain_id,
the clock it comes from. Readings with a start_, started_, ended_,
completed_ or failed_ prefix are the same pair for that moment. Two
readings compare only within one clock domain. schema_version names a
record's format. A demonstration's records also hold testuary_demo, which
Demonstrations describes.
The records that a user or an analysis reads are described field by field below. For the other files this page names the file and what it holds: their fields are working data that this page does not list.
tests.csv¶
results/tests.csv in the run directory has one row per target and repetition.
It is written as UTF-8 with a byte order mark and Windows line endings, so
spreadsheet programs open it correctly. A demonstration rewrites its table
with plain line endings when it adds its column demo. An empty
cell means the value is not known, not zero. A repetition that crashed still
writes its row: the phases it finished carry their values, the crashed
phase's status reads crashed with the error in error, and the columns
only Finishing writes stay empty. The headline columns:
| Column | Meaning |
|---|---|
target_number, run, file_path |
The target's position, the repetition (0 for the one in directory r0) and its path |
line_coverage, branch_coverage |
Final coverage of the delivered suite in percent, as the phase end that handed it on measured it, from Finishing. When Finishing did not run, as after a stop after the coverage phase or a reuse that skips it, the coverage phase's last values, measured on that loop's suite |
mutation_score |
The delivered suite's mutation score in percent, described below. When Finishing did not run, as after a stop after the mutation phase, it is the mutation phase's best score, which covers only the selected operators while the phase used the selective catalogue |
starter_status, coverage_status, mutation_status, finishing_status |
How each phase ended |
test_count_delivered, mutation_test_count |
Tests in the delivered suite as Finishing counted them (the coverage phase's suite when Finishing did not run) and in the mutation phase's own test file at the loop's end, before its phase end chose the suite it handed on (mutation_delivered_tests) |
starter_requests, coverage_requests, mutation_requests |
Requests sent, the ones sent again included |
starter_logical_requests, coverage_logical_requests, mutation_logical_requests |
Model requests counted against each phase's limit. They are fewer than the requests sent when a request was sent again: in the coverage and mutation phases after a connection error or another temporary failure, as count_retries in [coverage] decides, and in the starter phase when a request was sent again with half the output limit after the model's context was full |
*_input_tokens, *_output_tokens, *_cache_hit_tokens |
Input tokens, output tokens and cache-hit tokens per phase (starter_input_tokens and so on), as the model service reported them |
*_unknown_usage_requests, *_unknown_cache_requests |
Requests for which the model service did not report the token use, or how much of it came from its cache. The token columns miss the use of these requests |
duration_s_target |
The repetition's elapsed time in seconds: its own work, its phases and the work between them, a failed attempt included. Empty when the repetition crashed, or when it was stopped or crashed and a resume reused work it had done, a phase it had finished or the starter suites it had saved, since the time of that work is not known |
duration_s_file_shared |
The time the target's repetitions share, outside every repetition, such as resume planning and the first copy of the project. The same in each of the target's rows and added to no repetition. For a target with one repetition, it and duration_s_target add up to the target's time |
time_model_s, time_test_s, time_coverage_s, time_mutation_s, time_merge_s, time_cover_s, time_preparation_s, time_other_s |
Where the repetition's time went, in seconds: model requests (in the starter phase, whose requests run in parallel, the time some request is in flight), test runs, coverage and mutation measurement, combining suites, suite reduction, the copies of the project a measurement runs in, and the rest. No second is counted twice. Like duration_s_target, they cover the repetition's own work, a failed attempt's time included, and time_other_s holds its work between the phases. An empty cell means that work did not happen. All are empty when duration_s_target is empty because the time is not known |
The other columns:
| Column | Meaning |
|---|---|
score |
mutation_score when there is one, else 0 when the coverage phase ended with failing tests (coverage_status tests_failed), else empty |
error |
Notes on the row, such as a failure or a coverage phase that stopped after measurements without improvement. A loop that a request, token or time limit ended adds none: its status reads budget_exhausted |
below_minimum_reason |
Why a suite ended below its configured minimum, a finished result a resume does not continue. Empty otherwise |
mutation_switched, mutation_selective_final, mutation_full_at_switch |
Whether the mutation phase switched from the selective catalogue to the full catalogue, the score over the selective catalogue it had reached and the score over the full catalogue at the switch. Empty without a switch |
test_loc_delivered, assertion_count_delivered, parametrize_count_delivered, fixture_count_delivered |
Code lines, assertions, parametrized cases and fixtures of the delivered suite as Finishing counted them (the coverage phase's suite when Finishing did not run) |
mutation_test_loc, mutation_assertion_count, mutation_parametrize_count, mutation_fixture_count |
The same of the mutation phase's own test file at the loop's end |
test_complexity_sum_delivered, test_complexity_max_delivered |
The summed and the largest cyclomatic complexity of the delivered suite |
target_loc, target_complexity |
Code lines and cyclomatic complexity of the target |
source_observations |
The counts behind the columns above as JSON, each with its status, reason and parser |
lint_smells_target, lint_smells_tests |
Findings of the language's linter (Ruff for Python) in the target and in the tests, with [finishing] measure_lint = true. Empty for a language without a linter |
suite_timing_median_ms, suite_timing_samples_ms, suite_timing_run_count, suite_timing_warmup_count |
Timed runs of the delivered suite, with [timing] measure = true: the median and every sample in milliseconds (separated by semicolons), and the numbers of timed and warm-up runs |
suite_timing_requested_cpus, suite_timing_allowed_cpus |
The processors the timed runs asked for, and those a process started like them was allowed to use |
starter_suites_generated, starter_suites_passing |
Starter suites generated, and those whose tests passed |
starter_suites_unmeasured_coverage, starter_suites_unmeasured_mutation |
Passing starter suites that left without being fully measured: the coverage tool or the mutation tool could not measure them. Each was generated again while its attempts lasted, else left out |
starter_merged_tests, starter_delivered_tests, starter_delivered_from |
Tests in all passing starter suites together and in the suite the phase handed on, and which candidate it chose |
starter_baseline_line, starter_baseline_branch |
Coverage of the suite the starter phase handed on |
coverage_merged_tests, coverage_delivered_tests, coverage_delivered_from |
The same at the phase end of the coverage phase |
mutation_merged_tests, mutation_delivered_tests, mutation_delivered_from |
The same at the phase end of the mutation phase |
coverage_turns, mutation_turns |
Tool calls of the coverage phase and the mutation phase. Accepted edits are not counted |
starter_duration_s, coverage_duration_s, mutation_duration_s, finishing_duration_s |
Each phase's elapsed time in seconds, of the attempts whose work the row holds: an attempt that crashed, or that failed in a phase other than the starter phase, counts for nothing. starter_duration_s is empty when the starter ran again with the suites a stopped or crashed repetition saved, since the time of that work is not known |
coverage_agent_duration_s, mutation_agent_duration_s |
The time of each loop's model requests and tool calls, without its first measurement and its phase end |
finishing_mutation_score_full, finishing_mutation_score_selective |
The delivered suite's mutation score over the full catalogue and over the selective catalogue, from one run. In Java, PIT tests a class's mutants in one JVM without running its static initializers again, so an earlier mutant can change a later mutant's verdict: the subset's score can differ from a run with the selective catalogue alone |
finishing_mutation_score_lower_bound, finishing_untested |
For a mutation measurement in Finishing that left mutants untested: its score with every untested mutant counted as not killed, and how many mutants it left untested. Empty otherwise |
finishing_timeout_kills |
Kills that came from the per-mutant time limit. A value above 0 means the score depends on timing |
finishing_line_coverage, finishing_branch_coverage |
Coverage of the delivered suite from Finishing |
starter_original_*, coverage_original_* |
Line and branch coverage (_line, _branch) and the mutation score over the full catalogue and the selective catalogue (_mutation_full, _mutation_selective) that the starter phase and the coverage phase recorded for the suite each handed on, as [finishing] phase_scores selects. Nothing is measured again for them |
mutation_original_* |
The same values for the mutation phase: they repeat Finishing's measurement of the delivered suite, whatever phase_scores selects |
The completion summary at the end of a run shows the headline values of each target.
Starter suites¶
The starter_suite_line_*, starter_suite_branch_* and
starter_suite_mutation_* columns summarize the accepted starter suites
(mean, median, standard deviation, minimum and maximum). Each starter suite is
measured on its own for coverage. For mutation,
[starter] suite_mutation chooses how each suite's
score is obtained, and under both the columns hold the measured scores. A suite that
passes but that the coverage tool cannot measure, or of which the mutation
tool measures nothing, also after the tests a failure named were removed,
is not fully measured: it leaves the starter suites,
it is generated again within its attempts (refill_failed_suites) or left
out, and the remaining suites are measured again without it. Its
record names the classification coverage_unmeasured or
mutation_unmeasured and the reason. When the mutation tool's check run
before the first mutant stopped the suite because a test failed, the reason
names every stopping test, or says none named when the tool names none.
starter_suite_mutation_method names the method used: a derive or both
that could not pool the suites measured them and reads measure.
unavailable means the per-suite measurement produced no result, so the
columns are empty, and an empty value means no mutation was measured. Under
both the derivation record still holds the derived values.
Measurement and recovery
explains what the methods mean.
Each starter suite's directory in the starter phase's archive,
suites/suite_NNN/, holds the files of its latest attempt: the model's reply
(response.txt), the code taken from it (original with the language's
suffix), its record (original.json), the validity record (validity.json)
and, for an accepted suite, the cleaned suite. For an accepted suite this is the
accepted attempt. The files of every earlier attempt of the suite, which a later
attempt replaced, are kept in suites/suite_NNN/attempts/<n>/, where n is that
attempt's number for its suite. original.json records the number as
slot_attempt, and each attempt in suites/_generation.json records it too.
In a language without a collection step of its own, such as C, an attempt
discarded because its suite does not compile or link records
collection_failure_reason in the cleanup part of its original.json: the
failing run's reason with up to collection_errors of the compiler's lines.
For the suite as generated it is the first failing run's text, the text of the
run log's [starter] collection failed line. For a suite that fails to
compile after a cleanup round it is that round's run.
When a measurement of a starter suite removed tests, as
Tests removed after a failing measurement
describes, the entries go into cleanup.measurement_removals of the
attempt's original.json, in the form of the phase end's
measurement removals, with place
starter.zero_coverage_probe, starter.per_suite_evidence or
starter.per_suite_mutation. The cleaned suite is then the suite without
those tests, the suite's validity_derivations.json holds its
measurement_remove record, and for an accepted suite the slot in
suites/_generation.json names the changed suite in
measurement_removed_source.
The other files of a starter suite's directory:
| File | Holds |
|---|---|
cleaned.json |
The accepted suite's per-test data: each test's covered lines, branches and kills as tokens, each test's time, the suite's time and its covered lines and branches of the target. Working data |
accepted/<run>/<attempt>/<sha256> with the language's suffix |
The cleaned code an attempt accepted, named by the run, the attempt and the code's SHA-256, which the attempt's entry in _generation.json names |
unconfirmed with the language's suffix |
A cleaned suite whose last confirmation run gave no answer, kept so that --resume checks it again |
validity_derivations.json |
The validity derivations of the suite's cleanup, in the format of the validity records |
original.json, the record of the latest attempt:
| Key | Holds |
|---|---|
i, slot_attempt |
The starter suite's number and the attempt's number for that suite |
temperature, seed |
The sampling temperature and the seed of the attempt's request |
attempts, unknown_usage_requests |
How many requests the attempt sent, ones sent again included, and how many of them reported no token use |
input_tokens, output_tokens, cache_hit_tokens, cache_miss_tokens, unknown_cache_requests |
The attempt's token use as the model service reported it |
generation_seconds |
The time of the attempt's request |
extracted, response_sha256, response_chars |
Whether the reply held code, and the reply's SHA-256 and length |
source_sha256 |
The SHA-256 of the extracted code |
normalization |
Whether the code parsed (parse_ok, parse_error), the rule that removed same-named definitions an earlier one shadows (rule_version), whether it changed the code (changed, before_sha256, after_sha256) and what it removed (removed_bindings) |
cleanup |
The cleanup's outcome: disposition_basis and disposition_reason, the configured and used cleanup rounds, whether this was the slot's final attempt, the removals of measurement removals and the collection failure's reason |
validity_record |
The file of the validity record, validity.json, when the check wrote one |
classification |
The attempt's outcome, such as all_passing, discarded_syntax, discarded_collection, coverage_unmeasured or mutation_unmeasured |
suites/_generation.json, the starter suites' manifest. It is written after
every processed reply, so an interrupted generation keeps what it spent:
| Key | Holds |
|---|---|
requests_used, logical_requests, unknown_usage_requests, unknown_cache_requests |
Requests sent, model requests counted against the limit, and those without reported token use or cache use |
input_tokens, output_tokens, cache_hit_tokens |
The generation's token use |
generation_seconds |
The generation's elapsed time |
n_generated |
Starter suites generated |
n_discarded_syntax, n_discarded_collection, n_unsalvageable, n_zero_coverage, n_validation_unresolved |
Attempts discarded because their code did not parse, did not compile or collect, could not be repaired, covered no line of the target, or whose validity check gave no answer |
n_unmeasured_coverage, n_unmeasured_mutation |
Passing suites the coverage tool or the mutation tool could not measure |
transient_failures |
Temporary failures of requests |
processed_deliveries |
The replies already processed, by delivery, so a resume processes and counts each reply once |
generation_attempts_per_suite |
The attempts each starter suite spent |
generation_complete |
Whether the generation finished |
suite_slots |
One entry per requested suite, described below |
suite_slots holds requested_slots and slots, one entry for each starter
suite the run asked for, numbered from 1 (slot), also for those that never
started or could not be used. Its state is pending, active, accepted,
rejected or unavailable, and disposition_basis says why: code accepted
after cleanup (accepted_cleaned), a reply without code (no_code), a
rejection after judging (judged_rejection), or a check that could not run
(unavailable). terminal says whether the suite's outcome is final,
attempts_used counts its attempts, classification and reason repeat
its last outcome, attempts lists every attempt, terminal_attempt names
the attempt that ended it and accepted_source the accepted code. The
format's version is 1, and the run records each starter suite's final
outcome there. Each attempt keeps the identifier and the request number of
the model request that produced it. A repair that only checks the code again
updates that attempt and sends no new request. An accepted suite's entry
(base=starter_suites) names its starter suite and attempt and the SHA-256 of
the cleaned code it accepted, not of the model's raw reply.
The starter phase's archive holds starter_pool_attribution.json
(testuary.starter_pool_attribution/3), written only when each suite was
measured on its own. attribution_mode and basis name how the kills were
attributed, implementation how the suites were run, and union_mutants
the mutants with a result in any suite. Its slots list every requested
starter suite: whether it was accepted and measured, the outcome of
its mutation run, how many mutants it detected (killed: killed, timed out
or not viable) among those with a result (valid), and the verdicts
behind those counts.
starter_pool_kill_matrix.json merges the suites' results, in the format of
the kill matrices: a mutant counts as killed when any suite
killed it, and as untested when a suite has no result for it and no suite
killed it. Each mutant holds each suite's verdict in suite_verdicts, and
per_suite names how the suites were run and each suite's outcome. Under
derive it is the pooled run's matrix.
Under derive and both, starter_pool_derivation.json
(testuary.starter_pool_derivation/1) records the method requested and used,
why a derivation could not be made, the pooled run's file name per starter
suite, each suite's derived counts and, under both, its measured
counts and every mutant whose derived and measured status differ with the
rule that produced the derived status. starter_pool_derived_kill_matrices.json
holds each suite's derived matrix, and under both
starter_pool_pooled_kill_matrix.json holds the pooled run's.
The starter suites' model requests are recorded in
suites/request_records.jsonl, one line per request with its outcome and
token use, and suites/prompt_transport.jsonl, one line per request with the
roles and SHA-256 of the messages it sent. Both are working data.
Repetitions and their summaries¶
Results and logs shows the run directory's layout, with
a directory for each repetition of a target (r0, r1 and so on).
| File | Holds |
|---|---|
archive/<phase>/result<suffix> |
The suite the phase end handed on, with the language's suffix |
archive/<phase>/selected_kill_matrix.json |
Which tests of that suite kill which mutants, a kill matrix |
archive/<phase>/selected_coverage_matrix.json |
Which lines and branches each test of that suite covers, a coverage matrix |
archive/<phase>/consolidation_record.json |
The phase end's record, described under Phase records |
archive/<phase>/merged_kill_matrix.json, merged_coverage_matrix.json |
The kill matrix and the coverage matrix of the merged suite |
archive/<phase>/reduced_kill_matrix.json, reduced_coverage_matrix.json |
The kill matrix and the coverage matrix of the reduced suite. The coverage matrix also names the kept tests (kept_tests) and the tests a derived score was restricted to (derive_kept_tests) |
archive/<phase>/measured_reduced_kill_matrix.json, derived_reduced_kill_matrix.json |
The reduced suite's kill matrix as measured, and as derived from the merged suite's matrix for its tests, which Measurement and recovery explains |
archive/<phase>/candidate2_kill_matrix.json, candidate2_coverage_matrix.json |
The kill matrix and the coverage matrix of the phase's own new suite, measured alone before the merge |
archive/<phase>/validity_merged.json, validity_delivered.json, validity_derivations.json |
The validity records of the merged suite, of the delivered suite and of the steps that derived one suite from another |
archive/<phase>/candidate_<label>_removed with the language's suffix |
A candidate that lost tests to a measurement removal and was not delivered |
archive/coverage/candidate_agent with the language's suffix |
The coverage phase's own suite as its loop left it, before the phase end |
archive/mutation/pre_merge/candidate_inherited, candidate_agent with the language's suffix |
The mutation phase end's two inputs: the suite the mutation phase received and the mutation phase's own suite |
archive/<phase>/generation/record.json |
The phase's generation record |
tests/ |
The delivered suite: its test files and the setup files it was measured with, which Results and logs describes |
tests/kill_matrix.json |
Which tests of the delivered suite kill which mutants, from Finishing's mutation results: the mutation phase end's matrix when Finishing reuses it |
tests/coverage_matrix.json |
The delivered suite's coverage from Finishing, in the coverage matrices' second form |
time_limits.json |
Only when the repetition reached a time limit: one entry each time it reached one, its phase, its setting in section.name form and its value in seconds. The warnings of summary.json are made from it |
A repetition's summary.json holds its final values from its row in
tests.csv: line_coverage, branch_coverage and mutation_score in
percent, null when not measured, the status of the starter suites, the
coverage phase and the mutation phase, and fully_measured, which is true when the
repetition is fully measured: its final line coverage and mutation score were
measured. The target's summary.json, in the target's directory beside the
repetition directories, summarizes its repetitions: runs names them,
fully_measured_runs counts the fully measured ones, and each final value has the
number of repetitions that measured it with their mean, minimum and maximum.
Its starter_median holds the mean over the repetitions of the starter
suites' median of each value, before the loops.
A repetition's summary also holds each phase's values for the suite it handed
on with its model use and time (phases), the values of Finishing
(finishing), the accepted starter suites' statistics (starter) and where
the repetition's time went (time), each from its column above:
| Summary key | Column |
|---|---|
phases.<phase>.line_coverage, branch_coverage |
<phase>_original_line, <phase>_original_branch |
phases.<phase>.mutation_score, mutation_score_selective |
<phase>_original_mutation_full, <phase>_original_mutation_selective |
phases.<phase>.tests |
<phase>_delivered_tests |
phases.<phase>.logical_requests, physical_requests |
<phase>_logical_requests, <phase>_requests |
phases.<phase>.input_tokens, output_tokens, cached_input_tokens |
<phase>_input_tokens, <phase>_output_tokens, <phase>_cache_hit_tokens |
phases.<phase>.turns |
<phase>_turns (coverage and mutation) |
phases.<phase>.seconds |
<phase>_duration_s |
finishing.line_coverage, branch_coverage |
finishing_line_coverage, finishing_branch_coverage |
finishing.mutation_score, mutation_score_selective |
finishing_mutation_score_full, finishing_mutation_score_selective |
finishing.mutation_score_lower_bound, mutants_untested, seconds |
finishing_mutation_score_lower_bound, finishing_untested, finishing_duration_s |
starter.line_coverage, branch_coverage, mutation_score, each with mean, median, min, max and std |
starter_suite_line_*, starter_suite_branch_*, starter_suite_mutation_* with the same statistics |
time.total, model_requests, test_runs |
duration_s_target, time_model_s, time_test_s |
time.coverage_measurement, mutation_measurement |
time_coverage_s, time_mutation_s |
time.merging, cover_solver, project_copies, other |
time_merge_s, time_cover_s, time_preparation_s, time_other_s |
results/summary.json lists the summaries of every target in the run
directory (files), with each value's count, mean, median, lowest, highest
and standard deviation across the targets and the values themselves
(values), beside the same for the targets' starter medians
(starter_median). A repetition whose mutation measurement in Finishing left
mutants untested has no mutation score: its finishing holds the lower bound
(mutation_score_lower_bound, the untested mutants counted as not killed) and
the untested mutants (mutants_untested). The target's summary states the
lower bound across such repetitions, and the project's summary
the statistics of the lower bounds of the targets without a mutation score
(mutation_score_lower_bound).
A repetition's summary.json also lists warnings: one entry for each phase
and setting whose time limit the repetition reached. Each entry holds kind
(time_limit), phase (starter, coverage, mutation or finishing), the
setting in section.name form, such as
mutation.measurement_timeout_seconds, its value in seconds (unit
is s), how often the repetition reached it (count) and the sentence the
run's log writes (text). The list is empty when the repetition reached no
limit. Raising the named setting gives the next run more time.
After the time limits, the list holds one entry of kind loop_limit for
each loop (phase coverage or mutation) that one of its limits stopped
before its goal: its request, token or time limit. Its setting, such as
mutation.max_requests_per_phase, and value name the limit from the
loop's stop in phase_state.json, unit is s for the time limit and null
otherwise, and count is 1. A stop that names no setting leaves setting
and value null. The target's summary.json counts the repetitions with a
warning in warning_runs.
The end-of-run summary¶
results/completion.json, beside results/summary.json, holds the summary
the console prints when a run ends, which
Results and logs describes. It is
written in one step at the end of every run, with credentials removed from its
texts, and a resumed run writes it again.
read_completion reads it.
| Key | Holds |
|---|---|
schema_version |
The format's version |
provenance |
Where the values come from, such as observed for a run's own measurements and synthetic for a demonstration's |
completion.files |
One entry per selected target, described below |
completion.results_directory, log_path |
The run's results directory and its run.log |
completion.resume_command |
The exact command that continues the run, null when there is nothing to continue |
completion.cancelled |
Whether the run was stopped before it finished |
Each entry of files holds the target's path, its outcome (complete,
no_complete_suite, failed, cancelled, already_complete or
stopped) with its reason, the delivered suite's files (suite), its
metrics (each with name, value, unit, basis and goal), its model
usage (requests, tokens, cost and the request limit), whether a resume
continues it (resumable), and the sentences of its fallbacks and
warnings. Its warnings are the sentences of each repetition's
summary.json warnings. The console prints each warning as a line
Warning: with its sentence.
Phase records¶
When a removal ran at a phase end, as
Tests removed after a failing measurement
describes, the phase's consolidation_record.json holds
measurement_removals, one entry per round: the suite (place:
phase_end.candidate, phase_end.merged, phase_end.inherited or, for a
starter suite measured under the delivered name, starter.candidate), the
measurement, the round, the removed tests in removed, each with its
outcome (failed or error) where the run gave one, the names the suite
does not hold in unmatched, why a named case's whole test went in
detail, and, for a failure that removed nothing, why in unnamed. The
entries stand in the order the removals ran. A changed candidate that is not
delivered is kept in the archive as candidate_<label>_removed with the
test file's suffix, <label> the candidate's label in the selection, and a
removal in
the delivered suite's ancestry is a measurement_remove record of
validity_derivations.json.
phase_state.json¶
Each target's phase_state.json records which phases of each repetition are
finished:
| Key | Holds |
|---|---|
target_number, stem, target_file_path |
The target's number, its file name without the suffix, and its path |
starter, coverage, mutation, finishing |
Each phase's entry per repetition, keyed by the repetition's number ("0" for r0) |
pipeline |
Per repetition, the time of its work between its phases, in segments |
file |
The target's time across its repetitions in this run, the work they share included, in segments |
A phase's entry for a repetition holds:
| Key | Holds |
|---|---|
status |
in_progress while the phase runs or after the process stopped in it, complete, or failed with its reason and, when known, failure_kind |
target_file_path, target_source_sha256 |
The target's path and the SHA-256 of its raw bytes when the phase started, which recorded_source_sha256 reads |
resume_context |
Where the project ran, which a resume compares |
segments |
The phase's attempts, described below |
artifacts |
Each file the phase delivered, by its path in the target's directory, with its SHA-256, which a resume checks |
metrics |
The phase's values, from which its tests.csv columns come. A loop's hold its status, such as budget_exhausted, or skipped_inherited_slow for a mutation phase whose inherited suite ran longer than its time limit before any edit, and in error a stopped loop's reason, such as final_answers_without_measurement after max_final_answers_without_measurement stopped it |
generation_condition |
The phase's generation condition, when the phase recorded one |
started_t_mono_s, completed_t_mono_s, failed_t_mono_s |
When the phase started, completed or failed, each with its clock_domain_id |
Each entry of segments is one attempt: its outcome (ran, crashed,
abandoned or restored), its seconds, its model_seconds, its time per
operation (parts: model, test, coverage, mutation, merge,
cover and preparation) and the time no part claims (other_seconds), the
mutation tool's workers (engine_workers), whether the starter suites were
reused (starter_suites_reused), when it started and ended, and
discarded for an attempt that finished but whose work is not kept. A
failed phase's failure_kind is infrastructure, a model service or tool
problem, for a model service failure, exhausted retries, a crash in the phase
or a mutation tool that gave no usable result, and method, no usable suite
within the limits, when the starter suites produced no usable suite or one of
the coverage phase's limits ended it before a usable, measured suite. This
outcome is the method's own result. A phase without that field failed for a
reason the run cannot judge, such as a request too large for the model's
context or a validity check that gave no answer.
consolidation_record.json¶
Each phase end writes consolidation_record.json in its phase directory of
the archive:
| Key | Holds |
|---|---|
target_file |
The target |
delivered_from, winner_source_label |
The candidate the phase end delivered, such as reduced or merged, and for a single suite delivered as it is, that suite's label |
selected_source_sha256, execution_name |
The SHA-256 of the delivered suite and its file name |
generation_ancestry |
From which inputs the phase built the delivered suite, described under Generation records |
target_revision, measurement_identity |
The SHA-256 of the target's source and the identity of the measurements, which binds them to that source and to the settings they ran with |
structural_parser_identity, target_validator_identity |
The parser that read the tests' structure and the interpreter or compiler that checked them |
merged_source_sha256, reduced_source_sha256, merged_tests, delivered_tests |
The SHA-256 of the merged and the reduced suite, and the tests in the merged and in the delivered suite |
reduction_valid |
Whether the reduction produced a passing reduced suite. A reduction may keep every test |
repaired_tests, evidence_repairs |
The tests a repair added back to the reduction because its evidence missed something, and each repair round |
unreduced_fallback, fallback_reason, fallback_candidates, fallback_merged_stats |
When no valid reduced suite could be built: why, the candidates offered instead, and the merged suite's size |
selection_audit |
The selection, described below |
detail |
A note on the phase end, such as why it built no reduction |
derive_measure_delta, derive_measure_differences |
Under reduced_suite_mutation = both, the derived minus the measured mutation score of the reduced suite and every mutant whose two statuses differ |
reduction_solver, reduction_solver_metadata |
How the reduction was solved, exact or greedy, and the solver's settings and times |
consolidation_mutation_method |
The value of reduced_suite_mutation |
artifacts, validity_artifacts |
The files the phase end wrote in its directory, and of them the validity records |
collision_resolution, input_normalizations |
How the merge resolved same-named tests (renamed, deduplicated, dropped) and the shadowed definitions removed from its inputs |
mutation_measured, shipped_has_matrix |
Whether the merged or the delivered suite has a kill matrix, and whether the delivered one has |
measurement_removals |
The measurement removals, only when one ran |
mutation_origin_stops |
Only when the mutation tool's check run before the first mutant stopped a measurement because a test failed without a mutant: each stop with suite, the name of the measured suite's directory (such as merged), its reason and every stopping test in stopping_tests. In the starter phase it also lists the stops of the starter suites' own mutation runs, suite then naming the starter suite, such as suite_001. The run log names each stop on a [finishing] the mutation tool's origin check stopped line |
validity_policy_version |
The validity policy the delivered suite was checked under |
selection_audit holds:
| Key | Holds |
|---|---|
candidates |
Each candidate with its label, whether it is passing, its status, whether it is fully_measured, its mutation, branch and line scores, its runtime_ms, tests, source_sha256, execution_name, status_reason, removed_tests and repaired_tests, its target_revision, measurement_identity and suite_timing |
selector, active_axes |
The scores the selection compared, in order |
delivered_from, selection_basis, top_score_labels |
The chosen candidate, why (score, reduced_preference or timing_median) and the candidates with the best scores |
runtime_tie_break_basis, runtime_tie_break_candidate_order, runtime_tie_break_rounds |
For equal best scores, how the time decided (median or input_order), the order it took and its timed rounds |
input_evidence_missing_measurements, delivered_from_missing_from_merged_input |
The inputs without kills, lines or branches on record, and how many of the inputs' kills, lines and branches the delivered suite lacks |
reduction_missing_merged_lines, reduction_missing_merged_branches, reduction_missing_merged_kills |
How many of the merged suite's covered lines, taken branches and kills the reduced suite lacks |
reduction_measurement_issue |
Why a built reduction was excluded, when it was |
reduced_not_selected, merged_stats |
When the merged suite was chosen over a distinct reduction: true, and the merged suite's size |
Under reduced_suite_mutation = derive a reduced
candidate whose mutation score is derived from the merged suite's kill matrix
for its tests says so in its reason: derived, not measured. Its entry also
holds mutation_derived set to true, which no other candidate holds. When
the merged suite's per-test evidence gives other coverage than its coverage
run, in percentages, covered lines or taken arcs, merged_evidence_difference
in selection_audit records both sides, measured and per_test_evidence: each
side's line and branch coverage and the covered lines
(covered_lines_only) and taken arcs (taken_arcs_only) that only it has.
The merged suite keeps its coverage run's values. At the starter phase, when a
single starter suite is delivered and its acceptance run measures other line
or branch coverage than the evidence it was selected on,
delivered_coverage_difference records the sides selection and acceptance
in the same form, and the acceptance run's values are delivered.
The headline mutation_score in tests.csv is the delivered suite's mutation
score over all mutants from Finishing, also recorded as
finishing_mutation_score_full. finishing_mutation_measurement says how
that measurement ended. complete means it finished. incomplete means it
ended with mutants untested, as when its time limit ran out. failed means
it produced no record of which tests kill which mutants (the kill matrix). The
last two report no score, and Finishing is recorded as failed, so a resumed
run measures again. For an incomplete measurement, the counts of the mutants
with a result remain in finishing_killed, finishing_survived and
finishing_total. Such a measurement also records its lower bound, every
untested mutant counted as not killed, in
finishing_mutation_score_lower_bound, and its untested mutants in
finishing_untested. A later run that reuses the suite and measures no
mutation, as with [finishing] measure_mutation = false, keeps the score.
Generation records and the settings record¶
A repetition's resolved_settings.json holds the settings it used, without
secrets:
| Key | Holds |
|---|---|
sections |
The settings of every section, as the run resolved them, except those that configure no run ([web]). Of the provider sections ([openai], [ollama] and [vllm]), only that of the provider the run's model goes through, and none when a program supplies the model or the model does not resolve, since another provider's address can name a private host. The model catalogue is not in this file |
generated_test_validity |
The language adapter's validity policy (policy_version), none for an adapter without one |
effective |
What the settings alone do not show: the mutation phase's catalogue (mutation_catalogue, full or filtered) and Cosmic Ray's distributor (cosmic_ray_distributor) |
limits |
The limits of the coverage phase and the mutation phase |
model |
One entry per generation phase the run selected, with the resolved values and their sources. A model of a client factory names the factory's file and its SHA-256 (extension_file, extension_sha256) |
fallbacks |
The steps the run skips because the file's language lacks them, only when one applies |
absent_models_file |
The last file of a program's models_files (of testuary.settings.resolve_settings) when it does not exist, which reads as empty |
The model entries leave out reused phases, and a model a program supplies
records only its source. A client factory's distribution version is recorded
only when it is known. The file does not describe the ancestry of the
delivered suite. A resume keeps the existing file unchanged. Any other run
writes it afresh, so a run that reuses an earlier run's phases states the
settings its own phases ran under.
The starter suites' archive/starter/suites/generation_origin.json and each
phase's archive/<phase>/generation/record.json hold the generation
condition, the settings without secrets and links to the copied profile and
hints files. The generation origin's keys are format (1), condition, the
generation condition, which names the language package the run loaded
(language_package) and, where the tests' language differs from the
target's, that language (test_language, such as python for Python tests
of a C file), settings, the sections as in resolved_settings.json and,
of the model catalogue, only the run's model alias (models),
prompt_profile and hints (each with its path,
its SHA-256 and its copy), extensions (each loaded extension with its file,
SHA-256, distribution and version) and extension_hints (each extension's
hints with their SHA-256 and their copy). The copies stand beside the record,
named after it: <name>.profile.json, <name>.hints.json and
<name>.extension_hints_<n>.json. A starter suite reused from an
earlier run carries its origin's profile, hints and extension hints forward.
Model request records says where
the request records are. These are diagnostic records.
Reusing an earlier suite does not mean it was generated with the current profile.
The mutation tool that ran is the engine of [mutation] in the sections
of resolved_settings.json and of these records' settings, and the
engine of the generation condition's measurement settings, which
phase_state.json, the consolidation records and the chat logs copy:
cosmic_ray for Python, pit for Java and mull for C under the shipped
default. The measurement identities, such as config_sha256, hash the
mutation settings as configured, so they hold the shipped default
cosmic_ray for every language.
Generation records¶
Each phase's consolidation_record.json holds generation_ancestry (format
version 1, basis=construction_inputs): from which inputs the phase built
the suite it chose. It names the chosen source, the command and the run
apart from the model request counts, and keeps each input's origin and
parents in the record itself, so later records or profile files cannot
change it. Pruning, combining and suite reduction are construction steps:
they do not say which test the model wrote. A result reused from an earlier
run is named as reused, not as generated again. The settings alone do not
show what generated a chosen suite.
testuary.generation_evidence.read_suite_slots(manifest, suites_root),
read_cleaned_source(reference, suites_root) and
read_ancestry(record, source) read these records and check them. They
refuse an unknown version, an unsafe or changed reference, a cycle and
contradictory entries with GenerationEvidenceError. Missing ancestry stays
unavailable, and an edit whose earlier version is unknown stays partial. The
readers add nothing the records do not hold.
Matrix and validity formats¶
Kill matrices¶
Every file named *kill_matrix*.json above is a kill matrix in LangAdapter's
format, which
MutationRun.to_kill_matrix
describes:
| Key | Holds |
|---|---|
schema_version, target_file |
The format's version and the mutated file |
mutants |
One row per mutant: the engine's id, the mutant_id that names the same mutant in every run over the same source, its location (key, line), its operator and diff, its status, the tests that killed it (killer, killers, killing_tests, and unmatched_killers for names the suite does not hold), the kill's verification and the engine worker's outcome (worker_outcome) |
summary |
The counts of the mutants: total, detected, survived, timeout, non_viable, untested and unresolved |
catalogue_provenance |
The catalogue and operators requested, and the operators and mutants observed |
suite, execution, attribution |
The measured suite's files and SHA-256, where the run executed and how it was supervised, and whether it recorded the first or every killing test |
resolution |
How the matrix was obtained, such as measured, with its reason |
derived |
Only for a derived matrix: what it was derived from and how |
per_mutant_timeout |
The per-mutant time limit the run applied, and where it came from |
source_sha256, execution_name |
The SHA-256 of the measured suite and its file name |
t_mono_s, clock_domain_id |
When the matrix was written |
A matrix of a run that left mutants untested carries LangAdapter's incomplete stamp, and it reports no score.
Coverage matrices¶
The coverage matrices in a phase's archive hold, for one suite, these keys, not every matrix all of them:
| Key | Holds |
|---|---|
schema_version, target_file |
The format's version and the target |
source_sha256, execution_name |
The SHA-256 of the suite and its file name |
structural_parser_identity, target_validator_identity |
As in consolidation_record.json |
line, branch |
The suite's line and branch coverage in percent |
target_total_statements, covered_statement_lines |
The target's statements and the lines the suite covers |
target_total_branches, covered_branch_arcs |
The target's branches and the arcs the suite takes, as from->to |
per_test_tokens |
Each test's covered lines and branches as tokens, which the reduction reads |
per_test_runtime_ms, effective_cost_us |
Each test's measured time, and the cost the reduction weighs it with |
kept_tests, candidate_label |
In selected_coverage_matrix.json, the delivered suite's tests and the candidate it came from |
tests/coverage_matrix.json, from Finishing, has a second form: lines
(each line with its status, covered or missing), branches (each arc
from, to with its status, taken or missing), summary (the
covered and missing lines and branches and the target's lines), and the
suite's source_sha256 and execution_name.
Validity records¶
validity.json, validity_merged.json and validity_delivered.json each
hold one validity check of a suite under the language's validity policy:
| Key | Holds |
|---|---|
record_type, stage |
direct for a check, and where it ran, such as starter.accepted_suite, phase_end.merged or phase_end.delivered_assertion |
policy_version, target_identity |
The policy and the target it was checked against: its language, module, source directory, source path and SHA-256 |
analyzed_source_sha256 |
The SHA-256 of the checked suite |
findings, audit_flags |
What the check found against the policy, and what its screening flagged |
removed_tests, final_clean_source_sha256 |
The tests removed for a finding, and the SHA-256 of the suite without them |
validity_derivations.json holds records, one per step that made one
suite from another, such as a reduction (set_cover_remove_restore) or a
measurement removal (measurement_remove): record_type derivation, its
stage, operation, policy_version and target_identity, the SHA-256 of
the suite before (parent_source_sha256) and after
(child_source_sha256), and the tests removed and restored.
Logs¶
logs/run.log holds every log line of the run as plain text, and
logs/main.jsonl is the run's structured log. Each repetition of a target
has its logs in logs/test/<target>/r<k>/:
| File | Holds |
|---|---|
coverage.jsonl, mutation.jsonl, finishing.jsonl |
Each phase's structured log, one JSON object per line: its start, each logged line, a finished loop with its conversation, and its end. main.jsonl has the same form |
coverage_chat.json, mutation_chat.json |
Each loop's conversation with the model: the messages and metadata such as the SHA-256 of the tool descriptions and the generation condition |
coverage_chat_pre_retry_NNN.json, mutation_chat_pre_retry_NNN.json |
The conversation as it stood before a request was sent again after a failure |
coverage_usage.jsonl, mutation_usage.jsonl |
One line per request sent: its outcome, its token use and whether the model service reported it |
coverage_prompt_transport.jsonl, mutation_prompt_transport.jsonl |
One line per request: the roles and SHA-256 of the messages it sent and how they were sent |
coverage_trace.jsonl, mutation_trace.jsonl |
Only with enable_trace_log = true: each request, tool choice, final answer check, recovery and reasoning event |
generated_test_validity_writes.jsonl |
The validity record of each edit the loop wrote, in the format of the validity records |
generated_test_validity_rejections.jsonl |
Each edit the validity policy refused, with its findings |
coverage_generation_origin.json, mutation_generation_origin.json |
Each loop's generation origin, in the form of the generation records |
The fields of the logs are working data that this page does not list. Results and logs says how to use them.