Skip to content

Outcomes

Every result of an operation that executes a tool carries exactly one outcome and a reason. A failure of the execution is an outcome, not an exception, unless it is uncertain whether a command the operation started has ended. Then the operation raises the original exception with its recovery handle, as Prepared copies describes. A value the adapter did not observe is absent from the result (None), never reported as zero, full coverage or success.

The categories

Outcome Meaning
completed The operation ran and its observation is valid. For a suite run, no collected test failed or raised an error. Skipped tests and expected failures do not change that.
test_failures The suite ran and at least one test failed or raised an error.
no_tests Nothing was collected.
collection_failed An error during collection stopped the run before its tests ran.
timed_out The operation reached its timeout.
infrastructure_failure A tool crashed, its report is unreadable, a session or file it needs is gone, or an instrument check failed.
unsupported The language, engine or context does not provide the operation.
missing_setup A tool or file the operation needs is absent from the project environment.

unsupported and missing_setup are different answers. The first says the adapter cannot do this for the context. The second says it could, once the named tool or file is present. Neither is a measured zero.

Scores and partial results

Failed or incomplete measurements never supply a complete score. A mutation run that its time limit ended (timed_out) has no score, though the mutants the tool reported stay in the result with their outcomes. A failed mutation run holds no mutants, counts or score, only its reason, its failure kind and the tests that stopped it. A warmed timing result that is not complete carries no summary statistics. Progress values, such as a mutation run's progress score, are for display only.

Readiness before running

The capability and setup reports use a related vocabulary for readiness: available, missing_setup and unsupported, or no readiness at all when it was not checked, which leaves it unknown, never assumed. Each unconfirmed setup item that the requested operations need names a next step, and those that only a real run can try say so. See Environment selection.