What Testuary requires¶
Testuary, a consumer of LangAdapter, generates test suites with a model. This page states, family by family, which operations Testuary calls in which part of its run, which of its settings make an operation required, and what it does when your language lacks the operation.
Who needs what explains the tiers of a consumer's
profile and how langadapter check --for combines profiles.
Hooks and facts lists each family's hooks under the
family's heading.
When a language lacks an operation¶
An operation is missing when your adapter does not provide its hook. LangAdapter then answers without running anything:
- An operation on tests, such as
tests.benchmark, ends with the outcomeunsupported. - An operation on sources, such as
sources.skeleton, answers with the statusunsupported. mutations.engines()is empty.
Each consumer decides what such an answer means. It refuses the run before any work, it ends the work on one file, or it takes a fallback and goes on. For Testuary, the sections below say which, family by family.
Testuary's phases¶
A Testuary run has four phases. The starter phase, the coverage phase and
the mutation phase generate tests with a model. Each of them ends with a
phase end, which merges the phase's candidate suites, measures them,
reduces them to the tests that add coverage or kills, and chooses one
suite. Finishing measures the chosen suite and sends no model request. A
run that stops after a phase (--stop-after-phase) or reuses a phase's
results (--reuse) leaves the other phases out, and with them their
requirements.
The check before the first phase¶
Before any phase, a run reads the project's capability report for the families of Testuary's profile.
- Run tests is required in every run. Coverage is required when a generation phase or Finishing runs. Mutation is required when the mutation phase runs, when Finishing measures mutation, or when a generation phase runs while the phase ends measure mutation. A required family your language lacks refuses the run with "required tools are missing for this run", and no phase runs. A capability report that does not complete refuses the run the same way.
- An optional family your language lacks gives its fallback. The run lists
each such family under "Optional language adapter features this run goes
without" and records the fallback under
fallbacksin the run's records. Three fallbacks change Testuary's method, and the run's summary names them too: merging, validity screening and kill attribution. - The check asks for an optional family only where the run uses it: code
smells only with
[finishing] measure_linton, delivered runs only withreport_reverse_order_failureson, and kill attribution only when the run measures mutation. - A capability report cannot show instrumentation, versions or containers, so the check neither lists nor records their fallbacks.
testuary checkasks the same questions of a project without running a phase.
Families by phase¶
"Required" means that the phase cannot work without the family. "Optional" means that the phase uses the family when your language has it and takes the fallback below when it does not.
| Family | Starter phase | Coverage phase | Mutation phase | Phase ends | Finishing |
|---|---|---|---|---|---|
| run tests | required | required | required | required | required |
| coverage | required | required | required | required | required |
| mutation | not used | not used | required | required | required |
| load repair | optional | not used | not used | optional | not used |
| import pruning | optional | not used | not used | optional | not used |
| syntax repair | optional | optional | optional | optional | not used |
| skip removal | optional | not used | not used | not used | not used |
| import guidance | optional | not used | not used | not used | not used |
| kill attribution | not used | not used | not used | optional | not used |
| merging | optional | optional | optional | optional | not used |
| per-test evidence | not used | not used | not used | optional | not used |
| instrumentation | optional | optional | optional | optional | not used |
| delivered runs | optional | optional | optional | optional | optional |
| validity screening | optional | not used | not used | optional | not used |
| timing | not used | not used | not used | optional | optional |
| structural metrics | not used | optional | optional | not used | not used |
| code smells | not used | not used | not used | not used | optional |
- Run tests. Every phase runs suites with
tests.run_suite.sources.prune_testsremoves tests from a suite in the starter phase and at the phase ends, andsources.test_metricscounts a suite's tests, which a phase end needs for every suite it compares. - Coverage. Every phase measures the target's coverage with
tests.run_suiteand acoverage_target. - Mutation. The mutation phase measures mutants with the language's
engine, and so does its phase end. The starter and coverage phase ends
measure mutation while
[main] phase_end_mutationis on, and Finishing while[finishing] measure_mutationis on. An engine that cannot start there is an error, not a fallback. A run that measures mutation outside a container also asks your adapter whether it refuses the session directory or the number of workers, and a refusal stops the run before any phase. - Load repair and import pruning. In the starter phase and at the phase ends, they repair a generated suite that does not load. Without load repair, such a suite is discarded. Without import pruning, a suite with an import that does not resolve does not load and is discarded.
- Syntax repair.
sources.validate_source, elsesources.check_structure, judges a generated source in the starter phase and at the phase ends, andsources.check_structurejudges each edit in the coverage and mutation phases.sources.finalize_sourcecompletes a source before it is kept. Without the family, each source passes these checks as it is, and a starter suite that does not parse is discarded. - Skip removal. The starter phase removes whole-file skips. Without it, they stay, and a suite without a passing test is rejected.
- Import guidance. The starter prompt names the import line a test needs. Without it, the prompt asks the model to follow the language's import conventions.
- Kill attribution. The reductions at the phase ends that keep every mutant kill need each kill attributed to a test. Without it, those reductions are skipped.
- Merging. At each phase end,
sources.resolve_collisionssettles test names that occur in several candidates andsources.merge_sourcesmerges the candidates into one suite.sources.normalize_bindingsnormalizes a suite's sources in the starter phase and when the coverage and mutation phases start. Without merging, the phase end keeps the better single candidate, by mutation score, then branch coverage, then line coverage, then the timing tie-break. With[starter] suite_mutationset toderiveorboth, the starter phase end stages its suites as one pool withsources.stage_pool, and without it measures each suite alone. - Per-test evidence. At each phase end, each test's coverage and kills decide the reduction. Without it, the reduction is skipped and each phase chooses among unreduced suites, and the starter phase end scores each starter suite on its own measurements.
- Instrumentation.
prepare_suiteprepares the suite's directory when the coverage and mutation phases start. Without it, the project must resolve instead. A test whose run warns of leaked state, such as an unawaited coroutine, is removed by the starter phase's cleanup and counts as failing in the coverage and mutation phases. Without the family, such tests are neither removed nor counted as failing. The phase ends log the adapter's warnings of tests whose verdict depends on the order in which a mock's results are consumed. - Delivered runs. For a language whose runs as delivered take a
session seed (
seeds_delivered_runs), the normal run of every phase is a run as delivered. Withreport_reverse_order_failureson, the coverage and mutation phases add the reverse-order note to a passing run. Without the family, the normal run is the language's own test run, and the reverse-order note is off. For C, whose normal run runs each test file whole as one program (C projects), the normal run and the run as delivered run the tests the same way. - Validity screening. In the starter phase and at the phase ends, tests that read the implementation are rejected. Without it, they are not rejected.
- Timing. At each phase end,
tests.benchmarkdecides between suites that tie on every other measure. Finishing records the chosen suite's run time with[timing] measureon. Without timing, the input order decides a tie, and no run time is recorded. - Structural metrics. After a timeout in the coverage or mutation
phase,
sources.skeletonandsources.function_fingerprintsname the tests added since the suite last ran in time, as the suspects. Without them, the phase names no suspect. - Code smells. With
[finishing] measure_linton, Finishing counts a linter's findings in the chosen suite. Without it, those counts stay empty. - Versions and containers. No phase calls an operation of these families. Without versions, every runtime version is reported as untested. Without containers, suites run on this machine only.
Where a measuring run of coverage, per-test evidence or mutation fails a suite that passed the normal run, Testuary removes the tests the run names before it chooses a suite, as its Measurement and recovery page describes. One run decides says what your adapter does for it.
Every phase reads the language's facts, such as its test file names and the names and code fences of its prompts. A fact your language does not declare takes its default.
Testuary's settings¶
Testuary's Settings page gives each setting's default. With Testuary's packaged settings, a run of all four phases requires run tests, coverage and mutation, and its check asks for neither code smells nor delivered runs.
| Setting | What it changes |
|---|---|
[main] phase_end_mutation |
On, the starter and coverage phase ends measure mutation, so a run with a generation phase requires mutation. |
[finishing] measure_mutation |
On, Finishing measures mutation, so a run that reaches Finishing requires mutation. |
[mutation] engine |
The engine among those your language declares. A name your language does not declare is refused, except Testuary's default name, which takes your language's first engine. |
[starter] suite_mutation |
derive and both stage the starter suites as one pool, through merging. |
[timing] measure |
On, Finishing records the chosen suite's run time, through timing. |
[finishing] measure_lint |
On, Finishing counts code smells, and the check asks for them. |
report_reverse_order_failures in [coverage] and [mutation] |
On, that phase adds the reverse-order note, and the check asks for delivered runs. |