Runs¶
langadapter.runs ¶
Suite runs: the request, the per-test outcomes, the coverage observation and the side effects a run applied.
A run executes the suite's files as they are and writes nothing into the
project or the suite's directories, unless the request asks for a side
effect. A run without a bytecode store compiles into a cache directory of its
own, so a source always runs as it is written. The result states each side
effect it applied. A value the run did not observe is absent (None),
never zero.
PASSED
module-attribute
¶
PASSED = 'passed'
The outcome of one test. A test whose setup or teardown fails is an error, and an unexpected pass of a test marked to fail counts as passed.
WALL_TIME_SCOPE
module-attribute
¶
WALL_TIME_SCOPE = "the runner's process from start to exit: interpreter start, collection, the tests and, when coverage was requested, its tracing and data writing"
What a run's wall time covers.
KILLS_NOT_REQUESTED
module-attribute
¶
KILLS_NOT_REQUESTED = 'not_requested'
The evidence holds no kill items because the request gave no mutation
run (PerTestEvidence.kills).
KILLS_USABLE
module-attribute
¶
KILLS_USABLE = 'usable'
Every killer of the mutation run maps to a test of the suite, and the evidence holds the kill items.
KILLS_UNUSABLE
module-attribute
¶
KILLS_UNUSABLE = 'unusable'
A killer of the mutation run maps to no test of the suite
(PerTestEvidence.unmapped_killers), so the evidence holds no kill
items.
WarmedSuiteBenchmarkRequest
dataclass
¶
WarmedSuiteBenchmarkRequest(suite: Suite, state_policy: str, iteration_timeout_s: float, total_timeout_s: float, startup_timeout_s: float = 60, forks: int = 3, warmup_runs: int = 5, runs: int = 10, serialize: bool = True, cpus: tuple[int, ...] | None = None, runner_options: object | None = None, on_progress: Callable[[WarmedBenchmarkProgress], None] | None = None)
A warmed timing of suite (tests.benchmark_warmed): forks
fresh JVM processes, each running warmup_runs discarded iterations
and then runs measured iterations of the suite.
suite: the suite to time.state_policy:native_lifecycle, the only accepted value: each iteration runs the suite's own fixtures as JUnit runs them, and static state stays in the JVM.iteration_timeout_s: the time limit of one iteration in seconds, until its record arrives.total_timeout_s: the time limit of the whole operation in seconds, the build and every fork included.startup_timeout_s: the time limit in seconds from the launch of a fork until its JVM reports that it is ready.forks: the number of forks, at least 1.warmup_runs: the discarded iterations of each fork, at least 1.runs: the measured iterations of each fork, at least 1.serialize: when true, the forks hold the process-wide benchmark lock, so they wait for the other benchmarks of this process.cpus: the distinct processor numbers the forks run on, passed totaskset -c, or None for no processor set.runner_options: None, since Java suite runs take no runner options.on_progress: receives aWarmedBenchmarkProgressat each stage. Its failure adds a diagnostic and never changes the result.
on_progress
class-attribute
instance-attribute
¶
on_progress: Callable[[WarmedBenchmarkProgress], None] | None = field(default=None, repr=False, compare=False)
validate ¶
validate() -> None
Refuse a state policy other than native_lifecycle, an invalid
count, timeout, serialize value, processor set, progress callback
or suite with InvalidRequest, naming the field.
WarmedBenchmarkProgress
dataclass
¶
WarmedBenchmarkProgress(stage: str, fork: int | None, kind: str | None, repetition: int | None, requested_forks: int, requested_warmup_runs: int, requested_runs: int, forks_started: int, forks_completed: int, warmup_completed: int, measurements_completed: int, elapsed_ns: int | None = None, operation_elapsed_s: float | None = None, outcome: Outcome | None = None, reason: str = '')
One progress event of a warmed timing, as the request's
on_progress receives it. It reports progress and is no evidence that
an iteration passed.
stage:setupbefore the build,fork_startandfork_endaround a fork,iteration_startanditeration_endaround an iteration, andcompleteonce the result is built.fork: the 0-based fork of the event, None forsetupandcomplete.kind:warmupormeasurementin an iteration event, else None.repetition: the 0-based index of the iteration within its kind in an iteration event, else None.requested_forks: the request'sforks.requested_warmup_runs: the request'swarmup_runs.requested_runs: the request'sruns.forks_started: the forks started so far.forks_completed: the forks so far that finished every requested iteration with a complete record.warmup_completed: the warm-up iterations so far with a complete record.measurements_completed: the measured iterations so far with a complete record.elapsed_ns: in aniteration_endevent, the iteration's time in nanoseconds as the JVM measured it, else None.operation_elapsed_s: the time since the operation started, in seconds.outcome: the iteration's outcome atiteration_end, the fork's atfork_endand the result's atcomplete, else None.reason: the fork's reason atfork_endand the result's atcomplete, else empty.
WarmedTestCounts
dataclass
¶
WarmedTestCounts(discovered: int | None = None, started: int | None = None, passed: int | None = None, skipped: int | None = None, aborted: int | None = None, failed: int | None = None, container_failures: int | None = None)
The tests of one warmed iteration by status, as the fork's JUnit listener counted them. Each count is None when test discovery did not finish.
discovered: the tests of the test plan, dynamic tests included.started: the tests that started.passed: the tests that succeeded.skipped: the tests that were skipped.aborted: the tests JUnit reported as aborted.failed: the tests that failed.container_failures: the test containers, such as test classes, that failed.
WarmedTestObservation
dataclass
¶
WarmedTestObservation(unique_id: str, display_name: str, kind: str, status: str, reason: str = '')
One test or test container of a warmed iteration, as JUnit identified it.
unique_id: JUnit's unique identifier of it, never empty.display_name: JUnit's display name of it.kind:testorcontainer.status:successful,skipped,abortedorfailed, ordiscoveredwhen it neither finished nor was skipped.reason: the reason of a skip, or the exception it finished with, else empty.
WarmedIteration
dataclass
¶
WarmedIteration(fork: int, kind: str, repetition: int, elapsed_ns: int | None, outcome: Outcome, report_complete: bool, counts: WarmedTestCounts, tests: tuple[WarmedTestObservation, ...], reason: str = '', diagnostics: tuple[str, ...] = ())
One warm-up or measured iteration of a fork: one execution of the suite's test plan in the fork's JVM.
fork: the 0-based fork it ran in.kind:warmupormeasurement.repetition: its 0-based index within its kind in the fork.elapsed_ns: the time ofLauncher.executein nanoseconds, which the JVM measured withSystem.nanoTime, None when the test plan did not execute. A zero adds a diagnostic on the timer's resolution.outcome: the iteration's outcome as the fork reported it:completed,no_tests,test_failures,collection_failedorinfrastructure_failure.report_complete: whether the record is complete: the test plan finished, and JUnit identified each item as either a test or a container.counts: the tests by status.tests: each test and test container JUnit identified, each identifier once.reason: why the outcome is notcompleted, else empty.diagnostics: the stack trace of an error the fork caught, and the note on a zero time.
WarmedFork
dataclass
¶
WarmedFork(fork: int, warmup_completed: int, measurements_completed: int, complete: bool, disposal_confirmed: bool, allowed_cpus: tuple[int, ...] | None, samples_ns: tuple[int, ...], mean_ms: float | None = None, median_ms: float | None = None, min_ms: float | None = None, max_ms: float | None = None)
One fork of a warmed timing: a fresh JVM process in its own copy of the build, which ran the warm-up iterations and then the measured ones.
fork: its 0-based index.warmup_completed: its warm-up iterations with a complete record.measurements_completed: its measured iterations with a complete record.complete: whether it finished and every requested iteration has a complete record.disposal_confirmed: whether the end of its process was confirmed, which a complete fork requires.allowed_cpus: the processors a process started like the fork may run on, read once before it: an allowed set, not the processors used, and None where the platform reports none or the reading failed.samples_ns: the times in nanoseconds of its measured iterations with a complete record, in order.mean_ms: the mean ofsamples_nsin milliseconds.median_ms: the median ofsamples_nsin milliseconds.min_ms: the shortest ofsamples_nsin milliseconds.max_ms: the longest ofsamples_nsin milliseconds.
The four statistics are None unless the result holds its summaries
(WarmedSuiteBenchmark.mean_of_fork_means_ms).
WarmedBenchmarkIdentity
dataclass
¶
WarmedBenchmarkIdentity(suite: tuple[FileIdentity, ...], project_inputs: tuple[FileIdentity, ...], toolchain: ToolchainIdentity, execution: ExecutionIdentity, build_tool: str | None, build_version: str | None, engine_version: str | None, jvm_args: tuple[str, ...], runner_options: tuple[tuple[str, str], ...], mode: str, timing_boundary: str, state_policy: str, forks: int, warmup_runs: int, runs: int, iteration_timeout_s: float, total_timeout_s: float, startup_timeout_s: float, requested_cpus: tuple[int, ...] | None, serialize: bool, schema_version: int = 1)
The functional settings and inputs of a warmed timing. Before its
setup has run, the toolchain is unknown, and project_inputs,
build_tool and engine_version are empty.
suite: the identities of the suite's files.project_inputs: the identities of the project's main Java sources in the copy of the build.toolchain: the JDK, with the versions of the JUnit Platform and, where the project uses it, JUnit 4.execution: where the forks ran.build_tool: the project's build tool,mavenorgradle.build_version: the build tool's version, which a warmed timing does not read: None.engine_version: the version of the JUnit Platform that ran the tests.jvm_args: the arguments of the fork JVMs: the project's test JVM arguments, then the context's interpreter flags.runner_options: the runner options as pairs of texts, empty for Java.mode:warmed_jvm.timing_boundary:junit_launcher_execute: a sample timesLauncher.executeof the test plan.state_policy: the request'sstate_policy.forks: the request'sforks.warmup_runs: the request'swarmup_runs.runs: the request'sruns.iteration_timeout_s: the request'siteration_timeout_s.total_timeout_s: the request'stotal_timeout_s.startup_timeout_s: the request'sstartup_timeout_s.requested_cpus: the request'scpus.serialize: the request'sserialize.schema_version: the version of this record's form, 1.
WarmedSuiteBenchmark
dataclass
¶
WarmedSuiteBenchmark(outcome: Outcome, reason: str, validity: str, requested_forks: int, requested_warmup_runs: int, requested_runs: int, forks_started: int, forks_completed: int, warmup_completed: int, measurements_completed: int, iterations: tuple[WarmedIteration, ...], forks: tuple[WarmedFork, ...], operation_elapsed_s: float | None, identity: WarmedBenchmarkIdentity, requested_cpus: tuple[int, ...] | None, allowed_cpus: tuple[int, ...] | None, cancelled: bool, disposal_confirmed: bool, diagnostics: tuple[str, ...] = (), mode: str = 'warmed_jvm', timing_boundary: str = 'junit_launcher_execute', clock: str = 'java.lang.System.nanoTime', unit: str = 'ns', state_policy: str = 'native_lifecycle', comparison: str = 'not_assessed', mean_of_fork_means_ms: float | None = None, median_of_fork_medians_ms: float | None = None, sample_min_ms: float | None = None, sample_max_ms: float | None = None, between_fork_mean_sd_ms: float | None = None)
The result of a warmed timing (tests.benchmark_warmed): the
record of every iteration, the samples of each fork and, for a complete
observation, summary statistics in milliseconds.
outcome: the operation's outcome.reason: why the outcome is notcompletedor the result is notcomplete, else empty.validity:completewhen every requested fork finished every iteration with a time,non_repeatablewhen the tests that ran or their outcomes changed between iterations,incompletewhen records are missing, andno_sampleswhen no fork ran.requested_forks: the request'sforks.requested_warmup_runs: the request'swarmup_runs.requested_runs: the request'sruns.forks_started: the forks whose records were received.forks_completed: the forks that finished every requested iteration with a complete record.warmup_completed: the warm-up iterations with a complete record, over all forks.measurements_completed: the measured iterations with a complete record, over all forks.iterations: the record of every iteration, fork by fork.forks: the record of each fork.operation_elapsed_s: the operation's elapsed time in seconds, None when the language does not offer the operation or the adapter refused to start it.identity: the functional settings and inputs.requested_cpus: the request'scpus.allowed_cpus: the processors every fork was allowed alike (WarmedFork.allowed_cpus), None when they differ or none was read.cancelled: whether the operation was cancelled. A cancelled operation raises before its result is built, so a result holds False.disposal_confirmed: whether the end of every fork's process was confirmed.diagnostics: notes on the run, such as a progress observer's failure, a failed cleanup or the output of a failed setup step.mode:warmed_jvm.timing_boundary:junit_launcher_execute: a sample timesLauncher.executeof the test plan.clock:java.lang.System.nanoTime, the clock of the samples.unit:ns, the unit of the samples.state_policy: the request'sstate_policy,native_lifecycle.comparison: alwaysnot_assessed.mean_of_fork_means_ms: the mean of the forks'mean_ms.median_of_fork_medians_ms: the median of the forks'median_ms.sample_min_ms: the shortest sample of all forks in milliseconds.sample_max_ms: the longest sample of all forks in milliseconds.between_fork_mean_sd_ms: the sample standard deviation of the forks'mean_ms, None with a single fork.
The five summaries and the forks' statistics are present only when
validity is complete, outcome is completed, the
operation was not cancelled and the end of every fork was confirmed.
timing_boundary
class-attribute
instance-attribute
¶
timing_boundary: str = 'junit_launcher_execute'
mean_of_fork_means_ms
class-attribute
instance-attribute
¶
mean_of_fork_means_ms: float | None = None
median_of_fork_medians_ms
class-attribute
instance-attribute
¶
median_of_fork_medians_ms: float | None = None
between_fork_mean_sd_ms
class-attribute
instance-attribute
¶
between_fork_mean_sd_ms: float | None = None
SuiteRunRequest
dataclass
¶
SuiteRunRequest(suite: Suite, timeout_s: float, coverage_target: Optional[str] = None, runner_options: Any = None, module_under_test: Optional[str] = None, isolation_config: bool = False, remove_created_files: bool = False, record_leaks: bool = False, session_seed: Optional[int] = None, reverse_order: bool = False, timeout_journal: bool = False)
A run of suite with a timeout in seconds. coverage_target
names a file (relative to the project root, or absolute under it) whose
coverage the run measures. runner_options are the language's own
(for Python, PytestOptions). The side effects are off by default:
isolation_config: the project's test configuration, copied into the suite root and used from there.remove_created_files: before the run, removes the files that earlier runs of this adapter with this option created in the project tree, not files written between runs.timeout_journal: the per-test journal, from which the result names the test that was running when a timeout or a crash ended the run (SuiteRun.active_at_end), written by a plugin of the adapter, or by the determinism instrument a suite's setup file carries.
record_leaks records the tests that added, rebound or removed a name
of the module under test or left xarray's options or numpy's error or
print settings changed (SuiteRun.leaks): the determinism instrument
records them when the suite's setup file carries one, else a read-only
observer loaded with the run, which compares each test's start and end
and restores nothing. module_under_test (by default the coverage
target) is the module it compares. session_seed runs the suite as
delivered: with Python's hash seed, the global
generators of random, numpy and sympy, and hypothesis's seed option
set to it once when the session starts, 0 to 2**32 - 1 as Python's
hash seed allows. reverse_order runs the tests
in the reverse of their collected order. timeout_s is the timeout in
seconds.
validate ¶
validate() -> None
Refuse an invalid suite, timeout or session seed with
InvalidRequest, naming the field.
CollectionRequest
dataclass
¶
CollectionRequest(suite: Suite, timeout_s: float, runner_options: Any = None, isolation_config: bool = False)
Collect the listed suite without running test bodies. Collection still runs imports, setup files and collection hooks. Configuration isolation is a side effect, as for a suite run.
suite: the suite to collect.timeout_s: the timeout of the collection in seconds.runner_options: the language's own options (for Python,PytestOptions).isolation_config: copies the project's test configuration into the suite root and uses it from there.
validate ¶
validate() -> None
Refuse an invalid suite or timeout with InvalidRequest,
naming the field.
TestOutcome
dataclass
¶
TestOutcome(test: TestId, outcome: str, time_s: Optional[float] = None, message: str = '', trace: str = '', exception: str = '')
One test's outcome and its time in seconds as the runner reported
it (setup, call and teardown), absent when the report has none. A
failed or erroneous test's message is its report's message,
trace the report's text (the exception, its message and the
stack, as the runner wrote them) and exception the exception's
type where the report names it. time_s is that time in seconds.
RunCounts
dataclass
¶
RunCounts(collected: Optional[int] = None, passed: Optional[int] = None, failed: Optional[int] = None, errors: Optional[int] = None, skipped: Optional[int] = None)
The run's tests by outcome, each absent when the runner did not
report it. collected counts every test with an outcome. A test
marked to fail that failed counts as skipped, as the runner's report
has it.
CoverageObservation
dataclass
¶
CoverageObservation(target: FileIdentity, statements: Tuple[int, ...], covered: Tuple[int, ...], branches: Tuple[BranchItem, ...], taken: Tuple[BranchItem, ...], traced_lines: Optional[Tuple[int, ...]] = None)
The coverage of one target: its statements and the statements the run covered, and its branches and the branches the run took, in the language's own representation. For Python these are coverage.py's arcs between lines, where a negative destination leaves the code object that starts at that line. Percentages are derived from these sets.
traced_lines
class-attribute
instance-attribute
¶
traced_lines: Optional[Tuple[int, ...]] = None
Raw lines traced by the native instrument, including excluded lines.
These do not contribute to line_percent or the statement matrix.
line_percent
property
¶
line_percent: Optional[float]
The share of the target's statements the suite covered, in percent, None without statements.
branch_percent
property
¶
branch_percent: Optional[float]
The share of the target's branches the suite took, in percent, None without branches.
missing_branches ¶
missing_branches() -> Tuple[BranchItem, ...]
The target's branches the suite did not take, in order.
to_coverage_matrix ¶
to_coverage_matrix() -> Dict[str, Any]
The observation as a coverage matrix (schema 1): every statement covered or missing, every branch taken or missing, and the counts. The arcs keep the tool's destinations.
SideEffects
dataclass
¶
SideEffects(instrument: Optional[InstrumentIdentity] = None, instrument_source: Optional[str] = None, files_written: Tuple[str, ...] = (), files_removed: Tuple[str, ...] = (), leak_record: bool = False, session_seed: Optional[int] = None, reverse_order: bool = False, timeout_journal: bool = False)
What a run did beyond executing the suite: the instrument that ran
with it and where it came from (instrument_source: plugin, loaded
from the adapter's own directory, or suite, the suite's setup file
carrying it), the files it wrote into the suite's root, by name
(files_written), the files it removed, relative to the project root
(files_removed), whether the leak record was taken (leak_record),
the session seed of a run as delivered, whether the order was reversed
and whether the run wrote the per-test timeout journal.
Leak
dataclass
¶
Leak(test: TestId, added: Tuple[str, ...] = (), rebound: Tuple[str, ...] = (), removed: Tuple[str, ...] = (), xarray_options: Tuple[str, ...] = (), numpy_errors: Tuple[str, ...] = (), numpy_print: Tuple[str, ...] = ())
A test that left shared state changed, as the leak record (the
instrument's, or without it the read-only observer's) found it once the
test's own fixtures were torn down: the names of the module under test
it added, rebound to another
object and removed (names that begin and end with a double underscore
left out), and the xarray options and numpy error and print settings it
left changed. xarray_options holds the names of the xarray options
it changed, numpy_errors those of numpy's error settings and
numpy_print those of numpy's print options.
RunDiagnostic
dataclass
¶
RunDiagnostic(kind: str, message: str, test: Optional[TestId] = None, native_test: Optional[str] = None)
A native diagnostic kept independently of the bounded process log.
kind names what it reports:
unsound_test: a warning that marks a test leaving shared state behind (LanguageFacts.unsound_test_patterns).active_at_end: a test the timeout journal shows active at its last update.journal_unavailable: why the journal could not be read.coverage_counts_inconsistent: gcovr refused a C run's coverage counters because gcov read a negative count, or one of 2^32 or more, from them.messageis gcovr's message. The run has no coverage, and its tests' outcomes stay as they ran. A run whose tests all passed is then an infrastructure failure with that message as its reason, and a run with another outcome, such as failing tests, keeps it. A C run of whole programs then holds one more such diagnostic for each test whose own counts gcovr refuses, withtestand that test's message.compile_error: a C test file did not compile or link, and the run iscollection_failed.messageis the step that failed, such as "the test file test_calc.c does not compile". The compiler's or linker's error lines are the run'sreason_detail.
test is absent when attribution is unavailable. native_test
retains the tool's original attribution without claiming a matched test.
SuiteRun
dataclass
¶
SuiteRun(outcome: Outcome, reason: str, counts: RunCounts, tests: Tuple[TestOutcome, ...], unmatched: Tuple[str, ...], wall_time_s: Optional[float], output: str, coverage: Optional[CoverageObservation], identity: ObservationIdentity, side_effects: SideEffects = SideEffects(), leaks: Optional[Tuple[Leak, ...]] = None, active_at_end: Optional[TestId] = None, returncode: Optional[int] = None, diagnostics: Tuple[RunDiagnostic, ...] = (), reason_detail: Tuple[str, ...] = (), coverage_report_s: Optional[float] = None, coverage_statements_s: Optional[float] = None, runner_counts: Optional[RunCounts] = None)
The result of a suite run. tests holds every test the runner
reported, and unmatched the report entries that map to no test of
the suite. wall_time_s covers WALL_TIME_SCOPE. output is the
runner's output, whole up to four million characters. reason_detail
holds the complete lines a reason summarizes (a Python runner's
output lines when it exited without a usable verdict). reason_text
shows the reason with some or all of them. coverage is absent when
not requested or not measured. leaks holds the tests the leak record
names, in the order they finished. It is absent when the record was not
requested or could not be taken (the suite's own instrument does not
record leaks, or the run ended before loading it). active_at_end is
the test the timeout journal shows running when the run ended before
that test finished, as on a timeout or a crash: an observation, not
proof that this test alone ends the run. It is absent without the
journal, when every started test finished, and when the run ended
between tests. coverage_report_s is the time of reading the coverage
data after the runner exited (coverage.py's JSON report, JaCoCo's
analysis), and coverage_statements_s the time of reading coverage.py's
statements of the target from that data. Each is absent when its step
did not run. runner_counts holds the counts the runner's own summary
gives, where the language has one (pytest's): a test whose teardown
also fails counts as one failure and one error, and an unexpected pass
of a test marked to fail as neither. collected counts the tests
that passed, failed or raised an error, less one per file the runner
could not collect. It is None where the runner gave no counts (a run
that ended before its session did). side_effects states what the
run did beyond executing the suite.
side_effects
class-attribute
instance-attribute
¶
side_effects: SideEffects = field(default_factory=SideEffects)
coverage_statements_s
class-attribute
instance-attribute
¶
coverage_statements_s: Optional[float] = None
reason_text ¶
reason_text(lines: Optional[int] = 3) -> str
The reason followed by the last lines lines of its detail
(all of them for None), as "reason: line | line". Three lines by
default.
outcome_of ¶
outcome_of(test: TestId) -> Optional[str]
The outcome of test in this run, None when the run did not
report it.
CollectionRun
dataclass
¶
CollectionRun(outcome: Outcome, reason: str, identity: ObservationIdentity, output: str, diagnostic: str = '', returncode: Optional[int] = None, wall_time_s: Optional[float] = None, side_effects: SideEffects = SideEffects(), files: Tuple[str, ...] = ())
A collect-only observation, not a verdict that tests passed.
diagnostic retains the collection exception summary even when it lies
outside bounded output. Neither coverage nor per-test execution is measured.
files holds the test files the runner collected tests from or could
not collect, POSIX and relative to the suite's root, where the runner
names them (pytest's). side_effects states what the collection did
beyond collecting the suite.
side_effects
class-attribute
instance-attribute
¶
side_effects: SideEffects = field(default_factory=SideEffects)
PerTestEvidenceRequest
dataclass
¶
PerTestEvidenceRequest(suite: Suite, target: str, timeout_s: float, warmup: bool = True, mutation_run: Any = None, runner_options: Any = None, module_under_test: Optional[str] = None, isolation_config: bool = False, remove_created_files: bool = False)
Per-test evidence of suite for target (a file relative to the
project root, or absolute under it): the target's lines and arcs each
test covers, its time, and, from mutation_run (a MutationRun of
the same suite), the mutants it kills. warmup runs the suite once
before the measured run and discards it. The other fields are those of
a SuiteRunRequest, and module_under_test defaults to the target.
timeout_s: the timeout in seconds.runner_options: the language's own options (for Python,PytestOptions).isolation_config: copies the project's test configuration into the suite root and uses it from there.remove_created_files: before the run, removes the files that earlier runs of this adapter with this option created in the project tree.
validate ¶
validate() -> None
Refuse an invalid suite or timeout, or a missing target, with
InvalidRequest, naming the field.
TestEvidence
dataclass
¶
TestEvidence(test: TestId, items: FrozenSet[EvidenceItem], time_ms: Optional[float])
One test's evidence: the items it covers or kills, and its time in
milliseconds from the runner's report (setup, call and teardown),
absent when the report has no time for it. A parametrized function is
one test here, its cases' times summed, since coverage is recorded per
function. time_ms is that time.
ContextEvidence
dataclass
¶
ContextEvidence(context: str, items: FrozenSet[EvidenceItem])
PerTestEvidence
dataclass
¶
PerTestEvidence(outcome: Outcome, reason: str, tests: Tuple[TestEvidence, ...], untimed: Tuple[TestId, ...], unmatched_contexts: Tuple[str, ...], kills: str, unmapped_killers: Tuple[str, ...], suite_time_ms: Optional[float], coverage: Optional[CoverageObservation], identity: ObservationIdentity, side_effects: SideEffects = SideEffects(), unmatched_items: Tuple[ContextEvidence, ...] = (), outside_items: FrozenSet[EvidenceItem] = frozenset(), stopping_tests: Tuple[TestId, ...] = ())
Per-test evidence of a green run. tests holds every test the
runner reported, untimed the tests without a time, and
unmatched_contexts the coverage contexts that name no test of the
suite (their items belong to no test). Code that runs when a module is
imported is excluded from every test's items, since it runs whenever
any kept test imports the module.
outside_items are the target's lines and branches the suite covered
that no test's items hold (items_outside_tests).
kills says whether kill items are present: usable when every
killer of the mutation run maps to a test of the suite, else
unusable with unmapped_killers and no kill items, or
not_requested. coverage is the suite's coverage of the target.
A run that is not green carries no evidence. stopping_tests are the
tests a measurement's failure is pinned on, in the order its run met
them, each once: a test that did not pass in the measuring run, or a
test whose own coverage counts in that run cannot be read or are
missing. Each is the test id the suite run gives that test. Empty when
the evidence is complete or no test can be named. suite_time_ms is
the suite's time in milliseconds: for Python and Java the sum of the
tests' times, None when no test has one, and for C the elapsed time of
the measured runs. side_effects states what the runs did beyond
executing the suite.
side_effects
class-attribute
instance-attribute
¶
side_effects: SideEffects = field(default_factory=SideEffects)
unmatched_items
class-attribute
instance-attribute
¶
unmatched_items: Tuple[ContextEvidence, ...] = ()
outside_items
class-attribute
instance-attribute
¶
outside_items: FrozenSet[EvidenceItem] = frozenset()
BenchmarkProgress
dataclass
¶
BenchmarkProgress(kind: str, index: int, total: int, elapsed_ms: float)
One finished repetition: its kind, 1-based index, total and time.
The event precedes classification of the repetition's exit status.
It is progress, not evidence that the repetition passed.
kind is warmup or measured, and elapsed_ms is the
repetition's time in milliseconds.
BenchmarkRequest
dataclass
¶
BenchmarkRequest(suite: Suite, runs: int, timeout_s: float, warmup_runs: int = 1, serialize: bool = True, cpus: Optional[str] = None, runner_options: Any = None, module_under_test: Optional[str] = None, isolation_config: bool = False, remove_created_files: bool = False, progress: Optional[Callable[[BenchmarkProgress], None]] = None)
The timing of suite: warmup_runs repetitions that are
discarded, then runs measured ones, each limited to timeout_s.
The placement: serialize waits for the other benchmarks of this
process, and cpus is the processor set the repetitions run on where
the project runs, as taskset -c takes it (0-3,8). The other
fields are those of a SuiteRunRequest, and a benchmark measures no
coverage. progress receives an event after each warm-up
repetition, and after the first, the last and every
max(1, runs // 4)-th measured repetition. A callback failure never
changes the measured result.
runner_options: the language's own options (for Python,PytestOptions).module_under_test: the module that an instrument carried by a setup file of the suite restores before each test.isolation_config: copies the project's test configuration into the suite root and uses it from there.remove_created_files: before the repetitions, removes the files that earlier runs of this adapter with this option created in the project tree.
progress
class-attribute
instance-attribute
¶
progress: Optional[Callable[[BenchmarkProgress], None]] = field(default=None, repr=False, compare=False)
validate ¶
validate() -> None
Refuse an invalid suite, repetition count, timeout, processor
list or progress callback with InvalidRequest, naming the
field.
Benchmark
dataclass
¶
Benchmark(outcome: Outcome, reason: str, times_ms: Tuple[float, ...], warmup_runs: int, cpus: Optional[str], includes: str, identity: ObservationIdentity, side_effects: SideEffects = SideEffects(), allowed_cpus: Optional[Tuple[int, ...]] = None, returncode: Optional[int] = None)
The measured repetitions' times in milliseconds, and what they cover
(includes). A time is the elapsed time of the processes includes
names, as their supervisor timed them where it does, else as timed
around them. warmup_runs counts the repetitions discarded before
them, cpus is the processor set given, and allowed_cpus the
affinity of a process started like them, read once before them: an
allowed set, not the processors used, and None where the platform
reports none or the reading failed. Only a benchmark whose every
repetition passed has statistics. One that ended early (a repetition
that did not pass, the timeout) keeps the times of the measured
repetitions that passed before it, and its outcome says why.
returncode is the final repetition's exit code, absent when it did
not exit normally (such as a timeout) or the request could not be
executed. times_ms holds the measured times in order, and
side_effects states what the repetitions did beyond executing the
suite.
side_effects
class-attribute
instance-attribute
¶
side_effects: SideEffects = field(default_factory=SideEffects)
median_ms
property
¶
median_ms: Optional[float]
The median of times_ms, None unless the outcome is
completed with times.
mean_ms
property
¶
mean_ms: Optional[float]
The mean of times_ms, None unless the outcome is
completed with times.
min_ms
property
¶
min_ms: Optional[float]
The shortest of times_ms, None unless the outcome is
completed with times.
max_ms
property
¶
max_ms: Optional[float]
The longest of times_ms, None unless the outcome is
completed with times.
SuiteProfileRequest
dataclass
¶
SuiteProfileRequest(suite: Suite, timeout_s: float, runner_options: Any = None)
One profiled run of suite, limited to timeout_s seconds, with
the language's runner_options (for Python, PytestOptions).
ProfiledFunction
dataclass
¶
ProfiledFunction(path: str, name: str, start_line: int, calls: int, own_time_s: float, inclusive_time_s: float, share: float)
One function a profiled run entered: its file (path, relative to
the project root where the file lies inside it, else as the profiler
names it), name and start_line, its calls, own_time_s
(its own code), inclusive_time_s (with its callees) and share,
its own time divided by the run's total own time. Times are seconds of
the profiled run, the profiler's overhead included.
SuiteProfile
dataclass
¶
SuiteProfile(outcome: Outcome, reason: str, profiler: Optional[str], profiler_version: Optional[str], kind: Optional[str], total_own_time_s: Optional[float], functions: Tuple[ProfiledFunction, ...] = ())
Where one profiled run of a suite spent its time. A suite profile
attributes time but does not time the suite: its times include the
profiler's overhead, and profiler, profiler_version and kind
say how they arose. A deterministic profiler adds a cost at every
call and return, so often-called functions appear more expensive.
total_own_time_s is the profiled run's total own time, not the
suite's time, which a benchmark measures. Only a run whose every test
passed has functions, most own time first. Otherwise outcome
and reason say why.
with_detail ¶
with_detail(reason: str, detail: Tuple[str, ...], lines: Optional[int] = 3) -> str
reason followed by the last lines lines of detail (all of
them for None), as "reason: line | line".
items_outside_tests ¶
items_outside_tests(coverage: Optional[CoverageObservation], tests: Sequence[TestEvidence], file: str) -> FrozenSet[EvidenceItem]
The target's lines (file names it as the tests' items do) and
branches that the suite covered and no test's items hold: code that ran
outside every test, at import, in a fixture, setUp or
tearDown, in another thread, in a class's static initialization or
a @BeforeAll. A reduction that keeps every test's items keeps these
only through the code that runs them. Empty without the suite's
coverage.
summarize ¶
summarize(result: WarmedSuiteBenchmark) -> WarmedSuiteBenchmark
Retain raw evidence and summarize only a globally eligible observation.