Skip to content

Runs

langadapter.runs

Suite runs: the request, the per-test outcomes, the coverage observation and the side effects a run applied.

A run executes the suite's files as they are and writes nothing into the project or the suite's directories, unless the request asks for a side effect. A run without a bytecode store compiles into a cache directory of its own, so a source always runs as it is written. The result states each side effect it applied. A value the run did not observe is absent (None), never zero.

PASSED module-attribute

PASSED = 'passed'

The outcome of one test. A test whose setup or teardown fails is an error, and an unexpected pass of a test marked to fail counts as passed.

FAILED module-attribute

FAILED = 'failed'

A test whose own code failed, such as an assertion.

ERROR module-attribute

ERROR = 'error'

A test whose setup or teardown failed.

SKIPPED module-attribute

SKIPPED = 'skipped'

A test the run skipped.

XFAILED module-attribute

XFAILED = 'xfailed'

A test marked to fail that failed.

WALL_TIME_SCOPE module-attribute

WALL_TIME_SCOPE = "the runner's process from start to exit: interpreter start, collection, the tests and, when coverage was requested, its tracing and data writing"

What a run's wall time covers.

KILLS_NOT_REQUESTED module-attribute

KILLS_NOT_REQUESTED = 'not_requested'

The evidence holds no kill items because the request gave no mutation run (PerTestEvidence.kills).

KILLS_USABLE module-attribute

KILLS_USABLE = 'usable'

Every killer of the mutation run maps to a test of the suite, and the evidence holds the kill items.

KILLS_UNUSABLE module-attribute

KILLS_UNUSABLE = 'unusable'

A killer of the mutation run maps to no test of the suite (PerTestEvidence.unmapped_killers), so the evidence holds no kill items.

WarmedSuiteBenchmarkRequest dataclass

WarmedSuiteBenchmarkRequest(suite: Suite, state_policy: str, iteration_timeout_s: float, total_timeout_s: float, startup_timeout_s: float = 60, forks: int = 3, warmup_runs: int = 5, runs: int = 10, serialize: bool = True, cpus: tuple[int, ...] | None = None, runner_options: object | None = None, on_progress: Callable[[WarmedBenchmarkProgress], None] | None = None)

A warmed timing of suite (tests.benchmark_warmed): forks fresh JVM processes, each running warmup_runs discarded iterations and then runs measured iterations of the suite.

  • suite: the suite to time.
  • state_policy: native_lifecycle, the only accepted value: each iteration runs the suite's own fixtures as JUnit runs them, and static state stays in the JVM.
  • iteration_timeout_s: the time limit of one iteration in seconds, until its record arrives.
  • total_timeout_s: the time limit of the whole operation in seconds, the build and every fork included.
  • startup_timeout_s: the time limit in seconds from the launch of a fork until its JVM reports that it is ready.
  • forks: the number of forks, at least 1.
  • warmup_runs: the discarded iterations of each fork, at least 1.
  • runs: the measured iterations of each fork, at least 1.
  • serialize: when true, the forks hold the process-wide benchmark lock, so they wait for the other benchmarks of this process.
  • cpus: the distinct processor numbers the forks run on, passed to taskset -c, or None for no processor set.
  • runner_options: None, since Java suite runs take no runner options.
  • on_progress: receives a WarmedBenchmarkProgress at each stage. Its failure adds a diagnostic and never changes the result.

suite instance-attribute

suite: Suite

state_policy instance-attribute

state_policy: str

iteration_timeout_s instance-attribute

iteration_timeout_s: float

total_timeout_s instance-attribute

total_timeout_s: float

startup_timeout_s class-attribute instance-attribute

startup_timeout_s: float = 60

forks class-attribute instance-attribute

forks: int = 3

warmup_runs class-attribute instance-attribute

warmup_runs: int = 5

runs class-attribute instance-attribute

runs: int = 10

serialize class-attribute instance-attribute

serialize: bool = True

cpus class-attribute instance-attribute

cpus: tuple[int, ...] | None = None

runner_options class-attribute instance-attribute

runner_options: object | None = None

on_progress class-attribute instance-attribute

on_progress: Callable[[WarmedBenchmarkProgress], None] | None = field(default=None, repr=False, compare=False)

validate

validate() -> None

Refuse a state policy other than native_lifecycle, an invalid count, timeout, serialize value, processor set, progress callback or suite with InvalidRequest, naming the field.

WarmedBenchmarkProgress dataclass

WarmedBenchmarkProgress(stage: str, fork: int | None, kind: str | None, repetition: int | None, requested_forks: int, requested_warmup_runs: int, requested_runs: int, forks_started: int, forks_completed: int, warmup_completed: int, measurements_completed: int, elapsed_ns: int | None = None, operation_elapsed_s: float | None = None, outcome: Outcome | None = None, reason: str = '')

One progress event of a warmed timing, as the request's on_progress receives it. It reports progress and is no evidence that an iteration passed.

  • stage: setup before the build, fork_start and fork_end around a fork, iteration_start and iteration_end around an iteration, and complete once the result is built.
  • fork: the 0-based fork of the event, None for setup and complete.
  • kind: warmup or measurement in an iteration event, else None.
  • repetition: the 0-based index of the iteration within its kind in an iteration event, else None.
  • requested_forks: the request's forks.
  • requested_warmup_runs: the request's warmup_runs.
  • requested_runs: the request's runs.
  • forks_started: the forks started so far.
  • forks_completed: the forks so far that finished every requested iteration with a complete record.
  • warmup_completed: the warm-up iterations so far with a complete record.
  • measurements_completed: the measured iterations so far with a complete record.
  • elapsed_ns: in an iteration_end event, the iteration's time in nanoseconds as the JVM measured it, else None.
  • operation_elapsed_s: the time since the operation started, in seconds.
  • outcome: the iteration's outcome at iteration_end, the fork's at fork_end and the result's at complete, else None.
  • reason: the fork's reason at fork_end and the result's at complete, else empty.

stage instance-attribute

stage: str

fork instance-attribute

fork: int | None

kind instance-attribute

kind: str | None

repetition instance-attribute

repetition: int | None

requested_forks instance-attribute

requested_forks: int

requested_warmup_runs instance-attribute

requested_warmup_runs: int

requested_runs instance-attribute

requested_runs: int

forks_started instance-attribute

forks_started: int

forks_completed instance-attribute

forks_completed: int

warmup_completed instance-attribute

warmup_completed: int

measurements_completed instance-attribute

measurements_completed: int

elapsed_ns class-attribute instance-attribute

elapsed_ns: int | None = None

operation_elapsed_s class-attribute instance-attribute

operation_elapsed_s: float | None = None

outcome class-attribute instance-attribute

outcome: Outcome | None = None

reason class-attribute instance-attribute

reason: str = ''

WarmedTestCounts dataclass

WarmedTestCounts(discovered: int | None = None, started: int | None = None, passed: int | None = None, skipped: int | None = None, aborted: int | None = None, failed: int | None = None, container_failures: int | None = None)

The tests of one warmed iteration by status, as the fork's JUnit listener counted them. Each count is None when test discovery did not finish.

  • discovered: the tests of the test plan, dynamic tests included.
  • started: the tests that started.
  • passed: the tests that succeeded.
  • skipped: the tests that were skipped.
  • aborted: the tests JUnit reported as aborted.
  • failed: the tests that failed.
  • container_failures: the test containers, such as test classes, that failed.

discovered class-attribute instance-attribute

discovered: int | None = None

started class-attribute instance-attribute

started: int | None = None

passed class-attribute instance-attribute

passed: int | None = None

skipped class-attribute instance-attribute

skipped: int | None = None

aborted class-attribute instance-attribute

aborted: int | None = None

failed class-attribute instance-attribute

failed: int | None = None

container_failures class-attribute instance-attribute

container_failures: int | None = None

WarmedTestObservation dataclass

WarmedTestObservation(unique_id: str, display_name: str, kind: str, status: str, reason: str = '')

One test or test container of a warmed iteration, as JUnit identified it.

  • unique_id: JUnit's unique identifier of it, never empty.
  • display_name: JUnit's display name of it.
  • kind: test or container.
  • status: successful, skipped, aborted or failed, or discovered when it neither finished nor was skipped.
  • reason: the reason of a skip, or the exception it finished with, else empty.

unique_id instance-attribute

unique_id: str

display_name instance-attribute

display_name: str

kind instance-attribute

kind: str

status instance-attribute

status: str

reason class-attribute instance-attribute

reason: str = ''

WarmedIteration dataclass

WarmedIteration(fork: int, kind: str, repetition: int, elapsed_ns: int | None, outcome: Outcome, report_complete: bool, counts: WarmedTestCounts, tests: tuple[WarmedTestObservation, ...], reason: str = '', diagnostics: tuple[str, ...] = ())

One warm-up or measured iteration of a fork: one execution of the suite's test plan in the fork's JVM.

  • fork: the 0-based fork it ran in.
  • kind: warmup or measurement.
  • repetition: its 0-based index within its kind in the fork.
  • elapsed_ns: the time of Launcher.execute in nanoseconds, which the JVM measured with System.nanoTime, None when the test plan did not execute. A zero adds a diagnostic on the timer's resolution.
  • outcome: the iteration's outcome as the fork reported it: completed, no_tests, test_failures, collection_failed or infrastructure_failure.
  • report_complete: whether the record is complete: the test plan finished, and JUnit identified each item as either a test or a container.
  • counts: the tests by status.
  • tests: each test and test container JUnit identified, each identifier once.
  • reason: why the outcome is not completed, else empty.
  • diagnostics: the stack trace of an error the fork caught, and the note on a zero time.

fork instance-attribute

fork: int

kind instance-attribute

kind: str

repetition instance-attribute

repetition: int

elapsed_ns instance-attribute

elapsed_ns: int | None

outcome instance-attribute

outcome: Outcome

report_complete instance-attribute

report_complete: bool

counts instance-attribute

counts: WarmedTestCounts

tests instance-attribute

tests: tuple[WarmedTestObservation, ...]

reason class-attribute instance-attribute

reason: str = ''

diagnostics class-attribute instance-attribute

diagnostics: tuple[str, ...] = ()

WarmedFork dataclass

WarmedFork(fork: int, warmup_completed: int, measurements_completed: int, complete: bool, disposal_confirmed: bool, allowed_cpus: tuple[int, ...] | None, samples_ns: tuple[int, ...], mean_ms: float | None = None, median_ms: float | None = None, min_ms: float | None = None, max_ms: float | None = None)

One fork of a warmed timing: a fresh JVM process in its own copy of the build, which ran the warm-up iterations and then the measured ones.

  • fork: its 0-based index.
  • warmup_completed: its warm-up iterations with a complete record.
  • measurements_completed: its measured iterations with a complete record.
  • complete: whether it finished and every requested iteration has a complete record.
  • disposal_confirmed: whether the end of its process was confirmed, which a complete fork requires.
  • allowed_cpus: the processors a process started like the fork may run on, read once before it: an allowed set, not the processors used, and None where the platform reports none or the reading failed.
  • samples_ns: the times in nanoseconds of its measured iterations with a complete record, in order.
  • mean_ms: the mean of samples_ns in milliseconds.
  • median_ms: the median of samples_ns in milliseconds.
  • min_ms: the shortest of samples_ns in milliseconds.
  • max_ms: the longest of samples_ns in milliseconds.

The four statistics are None unless the result holds its summaries (WarmedSuiteBenchmark.mean_of_fork_means_ms).

fork instance-attribute

fork: int

warmup_completed instance-attribute

warmup_completed: int

measurements_completed instance-attribute

measurements_completed: int

complete instance-attribute

complete: bool

disposal_confirmed instance-attribute

disposal_confirmed: bool

allowed_cpus instance-attribute

allowed_cpus: tuple[int, ...] | None

samples_ns instance-attribute

samples_ns: tuple[int, ...]

mean_ms class-attribute instance-attribute

mean_ms: float | None = None

median_ms class-attribute instance-attribute

median_ms: float | None = None

min_ms class-attribute instance-attribute

min_ms: float | None = None

max_ms class-attribute instance-attribute

max_ms: float | None = None

WarmedBenchmarkIdentity dataclass

WarmedBenchmarkIdentity(suite: tuple[FileIdentity, ...], project_inputs: tuple[FileIdentity, ...], toolchain: ToolchainIdentity, execution: ExecutionIdentity, build_tool: str | None, build_version: str | None, engine_version: str | None, jvm_args: tuple[str, ...], runner_options: tuple[tuple[str, str], ...], mode: str, timing_boundary: str, state_policy: str, forks: int, warmup_runs: int, runs: int, iteration_timeout_s: float, total_timeout_s: float, startup_timeout_s: float, requested_cpus: tuple[int, ...] | None, serialize: bool, schema_version: int = 1)

The functional settings and inputs of a warmed timing. Before its setup has run, the toolchain is unknown, and project_inputs, build_tool and engine_version are empty.

  • suite: the identities of the suite's files.
  • project_inputs: the identities of the project's main Java sources in the copy of the build.
  • toolchain: the JDK, with the versions of the JUnit Platform and, where the project uses it, JUnit 4.
  • execution: where the forks ran.
  • build_tool: the project's build tool, maven or gradle.
  • build_version: the build tool's version, which a warmed timing does not read: None.
  • engine_version: the version of the JUnit Platform that ran the tests.
  • jvm_args: the arguments of the fork JVMs: the project's test JVM arguments, then the context's interpreter flags.
  • runner_options: the runner options as pairs of texts, empty for Java.
  • mode: warmed_jvm.
  • timing_boundary: junit_launcher_execute: a sample times Launcher.execute of the test plan.
  • state_policy: the request's state_policy.
  • forks: the request's forks.
  • warmup_runs: the request's warmup_runs.
  • runs: the request's runs.
  • iteration_timeout_s: the request's iteration_timeout_s.
  • total_timeout_s: the request's total_timeout_s.
  • startup_timeout_s: the request's startup_timeout_s.
  • requested_cpus: the request's cpus.
  • serialize: the request's serialize.
  • schema_version: the version of this record's form, 1.

suite instance-attribute

suite: tuple[FileIdentity, ...]

project_inputs instance-attribute

project_inputs: tuple[FileIdentity, ...]

toolchain instance-attribute

toolchain: ToolchainIdentity

execution instance-attribute

execution: ExecutionIdentity

build_tool instance-attribute

build_tool: str | None

build_version instance-attribute

build_version: str | None

engine_version instance-attribute

engine_version: str | None

jvm_args instance-attribute

jvm_args: tuple[str, ...]

runner_options instance-attribute

runner_options: tuple[tuple[str, str], ...]

mode instance-attribute

mode: str

timing_boundary instance-attribute

timing_boundary: str

state_policy instance-attribute

state_policy: str

forks instance-attribute

forks: int

warmup_runs instance-attribute

warmup_runs: int

runs instance-attribute

runs: int

iteration_timeout_s instance-attribute

iteration_timeout_s: float

total_timeout_s instance-attribute

total_timeout_s: float

startup_timeout_s instance-attribute

startup_timeout_s: float

requested_cpus instance-attribute

requested_cpus: tuple[int, ...] | None

serialize instance-attribute

serialize: bool

schema_version class-attribute instance-attribute

schema_version: int = 1

WarmedSuiteBenchmark dataclass

WarmedSuiteBenchmark(outcome: Outcome, reason: str, validity: str, requested_forks: int, requested_warmup_runs: int, requested_runs: int, forks_started: int, forks_completed: int, warmup_completed: int, measurements_completed: int, iterations: tuple[WarmedIteration, ...], forks: tuple[WarmedFork, ...], operation_elapsed_s: float | None, identity: WarmedBenchmarkIdentity, requested_cpus: tuple[int, ...] | None, allowed_cpus: tuple[int, ...] | None, cancelled: bool, disposal_confirmed: bool, diagnostics: tuple[str, ...] = (), mode: str = 'warmed_jvm', timing_boundary: str = 'junit_launcher_execute', clock: str = 'java.lang.System.nanoTime', unit: str = 'ns', state_policy: str = 'native_lifecycle', comparison: str = 'not_assessed', mean_of_fork_means_ms: float | None = None, median_of_fork_medians_ms: float | None = None, sample_min_ms: float | None = None, sample_max_ms: float | None = None, between_fork_mean_sd_ms: float | None = None)

The result of a warmed timing (tests.benchmark_warmed): the record of every iteration, the samples of each fork and, for a complete observation, summary statistics in milliseconds.

  • outcome: the operation's outcome.
  • reason: why the outcome is not completed or the result is not complete, else empty.
  • validity: complete when every requested fork finished every iteration with a time, non_repeatable when the tests that ran or their outcomes changed between iterations, incomplete when records are missing, and no_samples when no fork ran.
  • requested_forks: the request's forks.
  • requested_warmup_runs: the request's warmup_runs.
  • requested_runs: the request's runs.
  • forks_started: the forks whose records were received.
  • forks_completed: the forks that finished every requested iteration with a complete record.
  • warmup_completed: the warm-up iterations with a complete record, over all forks.
  • measurements_completed: the measured iterations with a complete record, over all forks.
  • iterations: the record of every iteration, fork by fork.
  • forks: the record of each fork.
  • operation_elapsed_s: the operation's elapsed time in seconds, None when the language does not offer the operation or the adapter refused to start it.
  • identity: the functional settings and inputs.
  • requested_cpus: the request's cpus.
  • allowed_cpus: the processors every fork was allowed alike (WarmedFork.allowed_cpus), None when they differ or none was read.
  • cancelled: whether the operation was cancelled. A cancelled operation raises before its result is built, so a result holds False.
  • disposal_confirmed: whether the end of every fork's process was confirmed.
  • diagnostics: notes on the run, such as a progress observer's failure, a failed cleanup or the output of a failed setup step.
  • mode: warmed_jvm.
  • timing_boundary: junit_launcher_execute: a sample times Launcher.execute of the test plan.
  • clock: java.lang.System.nanoTime, the clock of the samples.
  • unit: ns, the unit of the samples.
  • state_policy: the request's state_policy, native_lifecycle.
  • comparison: always not_assessed.
  • mean_of_fork_means_ms: the mean of the forks' mean_ms.
  • median_of_fork_medians_ms: the median of the forks' median_ms.
  • sample_min_ms: the shortest sample of all forks in milliseconds.
  • sample_max_ms: the longest sample of all forks in milliseconds.
  • between_fork_mean_sd_ms: the sample standard deviation of the forks' mean_ms, None with a single fork.

The five summaries and the forks' statistics are present only when validity is complete, outcome is completed, the operation was not cancelled and the end of every fork was confirmed.

outcome instance-attribute

outcome: Outcome

reason instance-attribute

reason: str

validity instance-attribute

validity: str

requested_forks instance-attribute

requested_forks: int

requested_warmup_runs instance-attribute

requested_warmup_runs: int

requested_runs instance-attribute

requested_runs: int

forks_started instance-attribute

forks_started: int

forks_completed instance-attribute

forks_completed: int

warmup_completed instance-attribute

warmup_completed: int

measurements_completed instance-attribute

measurements_completed: int

iterations instance-attribute

iterations: tuple[WarmedIteration, ...]

forks instance-attribute

forks: tuple[WarmedFork, ...]

operation_elapsed_s instance-attribute

operation_elapsed_s: float | None

identity instance-attribute

identity: WarmedBenchmarkIdentity

requested_cpus instance-attribute

requested_cpus: tuple[int, ...] | None

allowed_cpus instance-attribute

allowed_cpus: tuple[int, ...] | None

cancelled instance-attribute

cancelled: bool

disposal_confirmed instance-attribute

disposal_confirmed: bool

diagnostics class-attribute instance-attribute

diagnostics: tuple[str, ...] = ()

mode class-attribute instance-attribute

mode: str = 'warmed_jvm'

timing_boundary class-attribute instance-attribute

timing_boundary: str = 'junit_launcher_execute'

clock class-attribute instance-attribute

clock: str = 'java.lang.System.nanoTime'

unit class-attribute instance-attribute

unit: str = 'ns'

state_policy class-attribute instance-attribute

state_policy: str = 'native_lifecycle'

comparison class-attribute instance-attribute

comparison: str = 'not_assessed'

mean_of_fork_means_ms class-attribute instance-attribute

mean_of_fork_means_ms: float | None = None

median_of_fork_medians_ms class-attribute instance-attribute

median_of_fork_medians_ms: float | None = None

sample_min_ms class-attribute instance-attribute

sample_min_ms: float | None = None

sample_max_ms class-attribute instance-attribute

sample_max_ms: float | None = None

between_fork_mean_sd_ms class-attribute instance-attribute

between_fork_mean_sd_ms: float | None = None

SuiteRunRequest dataclass

SuiteRunRequest(suite: Suite, timeout_s: float, coverage_target: Optional[str] = None, runner_options: Any = None, module_under_test: Optional[str] = None, isolation_config: bool = False, remove_created_files: bool = False, record_leaks: bool = False, session_seed: Optional[int] = None, reverse_order: bool = False, timeout_journal: bool = False)

A run of suite with a timeout in seconds. coverage_target names a file (relative to the project root, or absolute under it) whose coverage the run measures. runner_options are the language's own (for Python, PytestOptions). The side effects are off by default:

  • isolation_config: the project's test configuration, copied into the suite root and used from there.
  • remove_created_files: before the run, removes the files that earlier runs of this adapter with this option created in the project tree, not files written between runs.
  • timeout_journal: the per-test journal, from which the result names the test that was running when a timeout or a crash ended the run (SuiteRun.active_at_end), written by a plugin of the adapter, or by the determinism instrument a suite's setup file carries.

record_leaks records the tests that added, rebound or removed a name of the module under test or left xarray's options or numpy's error or print settings changed (SuiteRun.leaks): the determinism instrument records them when the suite's setup file carries one, else a read-only observer loaded with the run, which compares each test's start and end and restores nothing. module_under_test (by default the coverage target) is the module it compares. session_seed runs the suite as delivered: with Python's hash seed, the global generators of random, numpy and sympy, and hypothesis's seed option set to it once when the session starts, 0 to 2**32 - 1 as Python's hash seed allows. reverse_order runs the tests in the reverse of their collected order. timeout_s is the timeout in seconds.

suite instance-attribute

suite: Suite

timeout_s instance-attribute

timeout_s: float

coverage_target class-attribute instance-attribute

coverage_target: Optional[str] = None

runner_options class-attribute instance-attribute

runner_options: Any = None

module_under_test class-attribute instance-attribute

module_under_test: Optional[str] = None

isolation_config class-attribute instance-attribute

isolation_config: bool = False

remove_created_files class-attribute instance-attribute

remove_created_files: bool = False

record_leaks class-attribute instance-attribute

record_leaks: bool = False

session_seed class-attribute instance-attribute

session_seed: Optional[int] = None

reverse_order class-attribute instance-attribute

reverse_order: bool = False

timeout_journal class-attribute instance-attribute

timeout_journal: bool = False

validate

validate() -> None

Refuse an invalid suite, timeout or session seed with InvalidRequest, naming the field.

CollectionRequest dataclass

CollectionRequest(suite: Suite, timeout_s: float, runner_options: Any = None, isolation_config: bool = False)

Collect the listed suite without running test bodies. Collection still runs imports, setup files and collection hooks. Configuration isolation is a side effect, as for a suite run.

  • suite: the suite to collect.
  • timeout_s: the timeout of the collection in seconds.
  • runner_options: the language's own options (for Python, PytestOptions).
  • isolation_config: copies the project's test configuration into the suite root and uses it from there.

suite instance-attribute

suite: Suite

timeout_s instance-attribute

timeout_s: float

runner_options class-attribute instance-attribute

runner_options: Any = None

isolation_config class-attribute instance-attribute

isolation_config: bool = False

validate

validate() -> None

Refuse an invalid suite or timeout with InvalidRequest, naming the field.

TestOutcome dataclass

TestOutcome(test: TestId, outcome: str, time_s: Optional[float] = None, message: str = '', trace: str = '', exception: str = '')

One test's outcome and its time in seconds as the runner reported it (setup, call and teardown), absent when the report has none. A failed or erroneous test's message is its report's message, trace the report's text (the exception, its message and the stack, as the runner wrote them) and exception the exception's type where the report names it. time_s is that time in seconds.

test instance-attribute

test: TestId

outcome instance-attribute

outcome: str

time_s class-attribute instance-attribute

time_s: Optional[float] = None

message class-attribute instance-attribute

message: str = ''

trace class-attribute instance-attribute

trace: str = ''

exception class-attribute instance-attribute

exception: str = ''

RunCounts dataclass

RunCounts(collected: Optional[int] = None, passed: Optional[int] = None, failed: Optional[int] = None, errors: Optional[int] = None, skipped: Optional[int] = None)

The run's tests by outcome, each absent when the runner did not report it. collected counts every test with an outcome. A test marked to fail that failed counts as skipped, as the runner's report has it.

collected class-attribute instance-attribute

collected: Optional[int] = None

passed class-attribute instance-attribute

passed: Optional[int] = None

failed class-attribute instance-attribute

failed: Optional[int] = None

errors class-attribute instance-attribute

errors: Optional[int] = None

skipped class-attribute instance-attribute

skipped: Optional[int] = None

CoverageObservation dataclass

CoverageObservation(target: FileIdentity, statements: Tuple[int, ...], covered: Tuple[int, ...], branches: Tuple[BranchItem, ...], taken: Tuple[BranchItem, ...], traced_lines: Optional[Tuple[int, ...]] = None)

The coverage of one target: its statements and the statements the run covered, and its branches and the branches the run took, in the language's own representation. For Python these are coverage.py's arcs between lines, where a negative destination leaves the code object that starts at that line. Percentages are derived from these sets.

target instance-attribute

target: FileIdentity

statements instance-attribute

statements: Tuple[int, ...]

covered instance-attribute

covered: Tuple[int, ...]

branches instance-attribute

branches: Tuple[BranchItem, ...]

taken instance-attribute

taken: Tuple[BranchItem, ...]

traced_lines class-attribute instance-attribute

traced_lines: Optional[Tuple[int, ...]] = None

Raw lines traced by the native instrument, including excluded lines. These do not contribute to line_percent or the statement matrix.

line_percent property

line_percent: Optional[float]

The share of the target's statements the suite covered, in percent, None without statements.

branch_percent property

branch_percent: Optional[float]

The share of the target's branches the suite took, in percent, None without branches.

missing

missing() -> Tuple[int, ...]

The target's statements the suite did not cover, in order.

missing_branches

missing_branches() -> Tuple[BranchItem, ...]

The target's branches the suite did not take, in order.

to_coverage_matrix

to_coverage_matrix() -> Dict[str, Any]

The observation as a coverage matrix (schema 1): every statement covered or missing, every branch taken or missing, and the counts. The arcs keep the tool's destinations.

SideEffects dataclass

SideEffects(instrument: Optional[InstrumentIdentity] = None, instrument_source: Optional[str] = None, files_written: Tuple[str, ...] = (), files_removed: Tuple[str, ...] = (), leak_record: bool = False, session_seed: Optional[int] = None, reverse_order: bool = False, timeout_journal: bool = False)

What a run did beyond executing the suite: the instrument that ran with it and where it came from (instrument_source: plugin, loaded from the adapter's own directory, or suite, the suite's setup file carrying it), the files it wrote into the suite's root, by name (files_written), the files it removed, relative to the project root (files_removed), whether the leak record was taken (leak_record), the session seed of a run as delivered, whether the order was reversed and whether the run wrote the per-test timeout journal.

instrument class-attribute instance-attribute

instrument: Optional[InstrumentIdentity] = None

instrument_source class-attribute instance-attribute

instrument_source: Optional[str] = None

files_written class-attribute instance-attribute

files_written: Tuple[str, ...] = ()

files_removed class-attribute instance-attribute

files_removed: Tuple[str, ...] = ()

leak_record class-attribute instance-attribute

leak_record: bool = False

session_seed class-attribute instance-attribute

session_seed: Optional[int] = None

reverse_order class-attribute instance-attribute

reverse_order: bool = False

timeout_journal class-attribute instance-attribute

timeout_journal: bool = False

Leak dataclass

Leak(test: TestId, added: Tuple[str, ...] = (), rebound: Tuple[str, ...] = (), removed: Tuple[str, ...] = (), xarray_options: Tuple[str, ...] = (), numpy_errors: Tuple[str, ...] = (), numpy_print: Tuple[str, ...] = ())

A test that left shared state changed, as the leak record (the instrument's, or without it the read-only observer's) found it once the test's own fixtures were torn down: the names of the module under test it added, rebound to another object and removed (names that begin and end with a double underscore left out), and the xarray options and numpy error and print settings it left changed. xarray_options holds the names of the xarray options it changed, numpy_errors those of numpy's error settings and numpy_print those of numpy's print options.

test instance-attribute

test: TestId

added class-attribute instance-attribute

added: Tuple[str, ...] = ()

rebound class-attribute instance-attribute

rebound: Tuple[str, ...] = ()

removed class-attribute instance-attribute

removed: Tuple[str, ...] = ()

xarray_options class-attribute instance-attribute

xarray_options: Tuple[str, ...] = ()

numpy_errors class-attribute instance-attribute

numpy_errors: Tuple[str, ...] = ()

numpy_print class-attribute instance-attribute

numpy_print: Tuple[str, ...] = ()

RunDiagnostic dataclass

RunDiagnostic(kind: str, message: str, test: Optional[TestId] = None, native_test: Optional[str] = None)

A native diagnostic kept independently of the bounded process log.

kind names what it reports:

  • unsound_test: a warning that marks a test leaving shared state behind (LanguageFacts.unsound_test_patterns).
  • active_at_end: a test the timeout journal shows active at its last update. journal_unavailable: why the journal could not be read.
  • coverage_counts_inconsistent: gcovr refused a C run's coverage counters because gcov read a negative count, or one of 2^32 or more, from them. message is gcovr's message. The run has no coverage, and its tests' outcomes stay as they ran. A run whose tests all passed is then an infrastructure failure with that message as its reason, and a run with another outcome, such as failing tests, keeps it. A C run of whole programs then holds one more such diagnostic for each test whose own counts gcovr refuses, with test and that test's message.
  • compile_error: a C test file did not compile or link, and the run is collection_failed. message is the step that failed, such as "the test file test_calc.c does not compile". The compiler's or linker's error lines are the run's reason_detail.

test is absent when attribution is unavailable. native_test retains the tool's original attribution without claiming a matched test.

kind instance-attribute

kind: str

message instance-attribute

message: str

test class-attribute instance-attribute

test: Optional[TestId] = None

native_test class-attribute instance-attribute

native_test: Optional[str] = None

SuiteRun dataclass

SuiteRun(outcome: Outcome, reason: str, counts: RunCounts, tests: Tuple[TestOutcome, ...], unmatched: Tuple[str, ...], wall_time_s: Optional[float], output: str, coverage: Optional[CoverageObservation], identity: ObservationIdentity, side_effects: SideEffects = SideEffects(), leaks: Optional[Tuple[Leak, ...]] = None, active_at_end: Optional[TestId] = None, returncode: Optional[int] = None, diagnostics: Tuple[RunDiagnostic, ...] = (), reason_detail: Tuple[str, ...] = (), coverage_report_s: Optional[float] = None, coverage_statements_s: Optional[float] = None, runner_counts: Optional[RunCounts] = None)

The result of a suite run. tests holds every test the runner reported, and unmatched the report entries that map to no test of the suite. wall_time_s covers WALL_TIME_SCOPE. output is the runner's output, whole up to four million characters. reason_detail holds the complete lines a reason summarizes (a Python runner's output lines when it exited without a usable verdict). reason_text shows the reason with some or all of them. coverage is absent when not requested or not measured. leaks holds the tests the leak record names, in the order they finished. It is absent when the record was not requested or could not be taken (the suite's own instrument does not record leaks, or the run ended before loading it). active_at_end is the test the timeout journal shows running when the run ended before that test finished, as on a timeout or a crash: an observation, not proof that this test alone ends the run. It is absent without the journal, when every started test finished, and when the run ended between tests. coverage_report_s is the time of reading the coverage data after the runner exited (coverage.py's JSON report, JaCoCo's analysis), and coverage_statements_s the time of reading coverage.py's statements of the target from that data. Each is absent when its step did not run. runner_counts holds the counts the runner's own summary gives, where the language has one (pytest's): a test whose teardown also fails counts as one failure and one error, and an unexpected pass of a test marked to fail as neither. collected counts the tests that passed, failed or raised an error, less one per file the runner could not collect. It is None where the runner gave no counts (a run that ended before its session did). side_effects states what the run did beyond executing the suite.

outcome instance-attribute

outcome: Outcome

reason instance-attribute

reason: str

counts instance-attribute

counts: RunCounts

tests instance-attribute

tests: Tuple[TestOutcome, ...]

unmatched instance-attribute

unmatched: Tuple[str, ...]

wall_time_s instance-attribute

wall_time_s: Optional[float]

output instance-attribute

output: str

coverage instance-attribute

coverage: Optional[CoverageObservation]

identity instance-attribute

identity: ObservationIdentity

side_effects class-attribute instance-attribute

side_effects: SideEffects = field(default_factory=SideEffects)

leaks class-attribute instance-attribute

leaks: Optional[Tuple[Leak, ...]] = None

active_at_end class-attribute instance-attribute

active_at_end: Optional[TestId] = None

returncode class-attribute instance-attribute

returncode: Optional[int] = None

diagnostics class-attribute instance-attribute

diagnostics: Tuple[RunDiagnostic, ...] = ()

reason_detail class-attribute instance-attribute

reason_detail: Tuple[str, ...] = ()

coverage_report_s class-attribute instance-attribute

coverage_report_s: Optional[float] = None

coverage_statements_s class-attribute instance-attribute

coverage_statements_s: Optional[float] = None

runner_counts class-attribute instance-attribute

runner_counts: Optional[RunCounts] = None

reason_text

reason_text(lines: Optional[int] = 3) -> str

The reason followed by the last lines lines of its detail (all of them for None), as "reason: line | line". Three lines by default.

failing_tests

failing_tests() -> Tuple[TestId, ...]

The tests that failed or raised an error.

outcome_of

outcome_of(test: TestId) -> Optional[str]

The outcome of test in this run, None when the run did not report it.

CollectionRun dataclass

CollectionRun(outcome: Outcome, reason: str, identity: ObservationIdentity, output: str, diagnostic: str = '', returncode: Optional[int] = None, wall_time_s: Optional[float] = None, side_effects: SideEffects = SideEffects(), files: Tuple[str, ...] = ())

A collect-only observation, not a verdict that tests passed.

diagnostic retains the collection exception summary even when it lies outside bounded output. Neither coverage nor per-test execution is measured. files holds the test files the runner collected tests from or could not collect, POSIX and relative to the suite's root, where the runner names them (pytest's). side_effects states what the collection did beyond collecting the suite.

outcome instance-attribute

outcome: Outcome

reason instance-attribute

reason: str

identity instance-attribute

identity: ObservationIdentity

output instance-attribute

output: str

diagnostic class-attribute instance-attribute

diagnostic: str = ''

returncode class-attribute instance-attribute

returncode: Optional[int] = None

wall_time_s class-attribute instance-attribute

wall_time_s: Optional[float] = None

side_effects class-attribute instance-attribute

side_effects: SideEffects = field(default_factory=SideEffects)

files class-attribute instance-attribute

files: Tuple[str, ...] = ()

PerTestEvidenceRequest dataclass

PerTestEvidenceRequest(suite: Suite, target: str, timeout_s: float, warmup: bool = True, mutation_run: Any = None, runner_options: Any = None, module_under_test: Optional[str] = None, isolation_config: bool = False, remove_created_files: bool = False)

Per-test evidence of suite for target (a file relative to the project root, or absolute under it): the target's lines and arcs each test covers, its time, and, from mutation_run (a MutationRun of the same suite), the mutants it kills. warmup runs the suite once before the measured run and discards it. The other fields are those of a SuiteRunRequest, and module_under_test defaults to the target.

  • timeout_s: the timeout in seconds.
  • runner_options: the language's own options (for Python, PytestOptions).
  • isolation_config: copies the project's test configuration into the suite root and uses it from there.
  • remove_created_files: before the run, removes the files that earlier runs of this adapter with this option created in the project tree.

suite instance-attribute

suite: Suite

target instance-attribute

target: str

timeout_s instance-attribute

timeout_s: float

warmup class-attribute instance-attribute

warmup: bool = True

mutation_run class-attribute instance-attribute

mutation_run: Any = None

runner_options class-attribute instance-attribute

runner_options: Any = None

module_under_test class-attribute instance-attribute

module_under_test: Optional[str] = None

isolation_config class-attribute instance-attribute

isolation_config: bool = False

remove_created_files class-attribute instance-attribute

remove_created_files: bool = False

validate

validate() -> None

Refuse an invalid suite or timeout, or a missing target, with InvalidRequest, naming the field.

TestEvidence dataclass

TestEvidence(test: TestId, items: FrozenSet[EvidenceItem], time_ms: Optional[float])

One test's evidence: the items it covers or kills, and its time in milliseconds from the runner's report (setup, call and teardown), absent when the report has no time for it. A parametrized function is one test here, its cases' times summed, since coverage is recorded per function. time_ms is that time.

test instance-attribute

test: TestId

items instance-attribute

items: FrozenSet[EvidenceItem]

time_ms instance-attribute

time_ms: Optional[float]

tokens

tokens() -> FrozenSet[str]

The test's items as canonical tokens.

ContextEvidence dataclass

ContextEvidence(context: str, items: FrozenSet[EvidenceItem])

Items recorded under a native context that names no reported test.

These are diagnostic observations, not attribution to an invented test.

context instance-attribute

context: str

items instance-attribute

items: FrozenSet[EvidenceItem]

PerTestEvidence dataclass

PerTestEvidence(outcome: Outcome, reason: str, tests: Tuple[TestEvidence, ...], untimed: Tuple[TestId, ...], unmatched_contexts: Tuple[str, ...], kills: str, unmapped_killers: Tuple[str, ...], suite_time_ms: Optional[float], coverage: Optional[CoverageObservation], identity: ObservationIdentity, side_effects: SideEffects = SideEffects(), unmatched_items: Tuple[ContextEvidence, ...] = (), outside_items: FrozenSet[EvidenceItem] = frozenset(), stopping_tests: Tuple[TestId, ...] = ())

Per-test evidence of a green run. tests holds every test the runner reported, untimed the tests without a time, and unmatched_contexts the coverage contexts that name no test of the suite (their items belong to no test). Code that runs when a module is imported is excluded from every test's items, since it runs whenever any kept test imports the module. outside_items are the target's lines and branches the suite covered that no test's items hold (items_outside_tests). kills says whether kill items are present: usable when every killer of the mutation run maps to a test of the suite, else unusable with unmapped_killers and no kill items, or not_requested. coverage is the suite's coverage of the target. A run that is not green carries no evidence. stopping_tests are the tests a measurement's failure is pinned on, in the order its run met them, each once: a test that did not pass in the measuring run, or a test whose own coverage counts in that run cannot be read or are missing. Each is the test id the suite run gives that test. Empty when the evidence is complete or no test can be named. suite_time_ms is the suite's time in milliseconds: for Python and Java the sum of the tests' times, None when no test has one, and for C the elapsed time of the measured runs. side_effects states what the runs did beyond executing the suite.

outcome instance-attribute

outcome: Outcome

reason instance-attribute

reason: str

tests instance-attribute

tests: Tuple[TestEvidence, ...]

untimed instance-attribute

untimed: Tuple[TestId, ...]

unmatched_contexts instance-attribute

unmatched_contexts: Tuple[str, ...]

kills instance-attribute

kills: str

unmapped_killers instance-attribute

unmapped_killers: Tuple[str, ...]

suite_time_ms instance-attribute

suite_time_ms: Optional[float]

coverage instance-attribute

coverage: Optional[CoverageObservation]

identity instance-attribute

identity: ObservationIdentity

side_effects class-attribute instance-attribute

side_effects: SideEffects = field(default_factory=SideEffects)

unmatched_items class-attribute instance-attribute

unmatched_items: Tuple[ContextEvidence, ...] = ()

outside_items class-attribute instance-attribute

outside_items: FrozenSet[EvidenceItem] = frozenset()

stopping_tests class-attribute instance-attribute

stopping_tests: Tuple[TestId, ...] = ()

tokens

tokens() -> Dict[TestId, FrozenSet[str]]

Each test's evidence as canonical tokens.

BenchmarkProgress dataclass

BenchmarkProgress(kind: str, index: int, total: int, elapsed_ms: float)

One finished repetition: its kind, 1-based index, total and time.

The event precedes classification of the repetition's exit status. It is progress, not evidence that the repetition passed. kind is warmup or measured, and elapsed_ms is the repetition's time in milliseconds.

kind instance-attribute

kind: str

index instance-attribute

index: int

total instance-attribute

total: int

elapsed_ms instance-attribute

elapsed_ms: float

BenchmarkRequest dataclass

BenchmarkRequest(suite: Suite, runs: int, timeout_s: float, warmup_runs: int = 1, serialize: bool = True, cpus: Optional[str] = None, runner_options: Any = None, module_under_test: Optional[str] = None, isolation_config: bool = False, remove_created_files: bool = False, progress: Optional[Callable[[BenchmarkProgress], None]] = None)

The timing of suite: warmup_runs repetitions that are discarded, then runs measured ones, each limited to timeout_s. The placement: serialize waits for the other benchmarks of this process, and cpus is the processor set the repetitions run on where the project runs, as taskset -c takes it (0-3,8). The other fields are those of a SuiteRunRequest, and a benchmark measures no coverage. progress receives an event after each warm-up repetition, and after the first, the last and every max(1, runs // 4)-th measured repetition. A callback failure never changes the measured result.

  • runner_options: the language's own options (for Python, PytestOptions).
  • module_under_test: the module that an instrument carried by a setup file of the suite restores before each test.
  • isolation_config: copies the project's test configuration into the suite root and uses it from there.
  • remove_created_files: before the repetitions, removes the files that earlier runs of this adapter with this option created in the project tree.

suite instance-attribute

suite: Suite

runs instance-attribute

runs: int

timeout_s instance-attribute

timeout_s: float

warmup_runs class-attribute instance-attribute

warmup_runs: int = 1

serialize class-attribute instance-attribute

serialize: bool = True

cpus class-attribute instance-attribute

cpus: Optional[str] = None

runner_options class-attribute instance-attribute

runner_options: Any = None

module_under_test class-attribute instance-attribute

module_under_test: Optional[str] = None

isolation_config class-attribute instance-attribute

isolation_config: bool = False

remove_created_files class-attribute instance-attribute

remove_created_files: bool = False

progress class-attribute instance-attribute

progress: Optional[Callable[[BenchmarkProgress], None]] = field(default=None, repr=False, compare=False)

validate

validate() -> None

Refuse an invalid suite, repetition count, timeout, processor list or progress callback with InvalidRequest, naming the field.

Benchmark dataclass

Benchmark(outcome: Outcome, reason: str, times_ms: Tuple[float, ...], warmup_runs: int, cpus: Optional[str], includes: str, identity: ObservationIdentity, side_effects: SideEffects = SideEffects(), allowed_cpus: Optional[Tuple[int, ...]] = None, returncode: Optional[int] = None)

The measured repetitions' times in milliseconds, and what they cover (includes). A time is the elapsed time of the processes includes names, as their supervisor timed them where it does, else as timed around them. warmup_runs counts the repetitions discarded before them, cpus is the processor set given, and allowed_cpus the affinity of a process started like them, read once before them: an allowed set, not the processors used, and None where the platform reports none or the reading failed. Only a benchmark whose every repetition passed has statistics. One that ended early (a repetition that did not pass, the timeout) keeps the times of the measured repetitions that passed before it, and its outcome says why. returncode is the final repetition's exit code, absent when it did not exit normally (such as a timeout) or the request could not be executed. times_ms holds the measured times in order, and side_effects states what the repetitions did beyond executing the suite.

outcome instance-attribute

outcome: Outcome

reason instance-attribute

reason: str

times_ms instance-attribute

times_ms: Tuple[float, ...]

warmup_runs instance-attribute

warmup_runs: int

cpus instance-attribute

cpus: Optional[str]

includes instance-attribute

includes: str

identity instance-attribute

identity: ObservationIdentity

side_effects class-attribute instance-attribute

side_effects: SideEffects = field(default_factory=SideEffects)

allowed_cpus class-attribute instance-attribute

allowed_cpus: Optional[Tuple[int, ...]] = None

returncode class-attribute instance-attribute

returncode: Optional[int] = None

median_ms property

median_ms: Optional[float]

The median of times_ms, None unless the outcome is completed with times.

mean_ms property

mean_ms: Optional[float]

The mean of times_ms, None unless the outcome is completed with times.

min_ms property

min_ms: Optional[float]

The shortest of times_ms, None unless the outcome is completed with times.

max_ms property

max_ms: Optional[float]

The longest of times_ms, None unless the outcome is completed with times.

SuiteProfileRequest dataclass

SuiteProfileRequest(suite: Suite, timeout_s: float, runner_options: Any = None)

One profiled run of suite, limited to timeout_s seconds, with the language's runner_options (for Python, PytestOptions).

suite instance-attribute

suite: Suite

timeout_s instance-attribute

timeout_s: float

runner_options class-attribute instance-attribute

runner_options: Any = None

validate

validate() -> None

Refuse an invalid suite or timeout with InvalidRequest, naming the field.

ProfiledFunction dataclass

ProfiledFunction(path: str, name: str, start_line: int, calls: int, own_time_s: float, inclusive_time_s: float, share: float)

One function a profiled run entered: its file (path, relative to the project root where the file lies inside it, else as the profiler names it), name and start_line, its calls, own_time_s (its own code), inclusive_time_s (with its callees) and share, its own time divided by the run's total own time. Times are seconds of the profiled run, the profiler's overhead included.

path instance-attribute

path: str

name instance-attribute

name: str

start_line instance-attribute

start_line: int

calls instance-attribute

calls: int

own_time_s instance-attribute

own_time_s: float

inclusive_time_s instance-attribute

inclusive_time_s: float

share instance-attribute

share: float

SuiteProfile dataclass

SuiteProfile(outcome: Outcome, reason: str, profiler: Optional[str], profiler_version: Optional[str], kind: Optional[str], total_own_time_s: Optional[float], functions: Tuple[ProfiledFunction, ...] = ())

Where one profiled run of a suite spent its time. A suite profile attributes time but does not time the suite: its times include the profiler's overhead, and profiler, profiler_version and kind say how they arose. A deterministic profiler adds a cost at every call and return, so often-called functions appear more expensive. total_own_time_s is the profiled run's total own time, not the suite's time, which a benchmark measures. Only a run whose every test passed has functions, most own time first. Otherwise outcome and reason say why.

outcome instance-attribute

outcome: Outcome

reason instance-attribute

reason: str

profiler instance-attribute

profiler: Optional[str]

profiler_version instance-attribute

profiler_version: Optional[str]

kind instance-attribute

kind: Optional[str]

total_own_time_s instance-attribute

total_own_time_s: Optional[float]

functions class-attribute instance-attribute

functions: Tuple[ProfiledFunction, ...] = ()

with_detail

with_detail(reason: str, detail: Tuple[str, ...], lines: Optional[int] = 3) -> str

reason followed by the last lines lines of detail (all of them for None), as "reason: line | line".

items_outside_tests

items_outside_tests(coverage: Optional[CoverageObservation], tests: Sequence[TestEvidence], file: str) -> FrozenSet[EvidenceItem]

The target's lines (file names it as the tests' items do) and branches that the suite covered and no test's items hold: code that ran outside every test, at import, in a fixture, setUp or tearDown, in another thread, in a class's static initialization or a @BeforeAll. A reduction that keeps every test's items keeps these only through the code that runs them. Empty without the suite's coverage.

summarize

summarize(result: WarmedSuiteBenchmark) -> WarmedSuiteBenchmark

Retain raw evidence and summarize only a globally eligible observation.