Mutation¶
langadapter.mutation ¶
Mutation measurement shared by every engine: requests and settings, mutant classes, the kill-matrix schema and scores, the evidence behind a recorded verdict, per-suite verdicts, run results and the mutation session.
A request's settings default to the engine's. A run records each value and
its source (EffectiveSetting), the per-mutant timeout also its rule and
inputs. Every mutant keeps its engine's native outcome and carries one
common class (MutationClass) with a subcategory. A score counts detected
mutants as killed, undetected and untested ones as not killed, and leaves
skipped ones out. Without untested and skipped mutants it is 100 minus
Cosmic Ray's cr-rate, a survival rate that counts a skipped mutant as
killed. Cosmic Ray's INCOMPETENT, where running the tests or applying the
mutation raised, measured nothing: it is untested, where cr-rate counts it
as killed.
Five methods change or explain what a run records in a mutant's kill-matrix
row (evidence_of). None equals a fresh measurement beyond what it states:
Verification (row key verification)
A kill without a named killing test, or by the per-mutant clock, runs
again, at most twice. A completed run that does not kill replaces the
record (corrected), as does a kill naming its killing tests
(recovered). A repeated timeout (reproduced_timeout) or an
unnamed kill the runner reported (reproduced_with_report) is kept.
A kill without evidence in either attempt is kept and flagged
unresolved. A nondeterministic test can give either verdict.
File attribution (row key adjudication)
An unresolved kill runs against each test file alone, and each file
whose run kills is recorded, never a test. A kill that needs several
files together stays unresolved (pool_context).
Timeout evidence (row keys timeout_evidence, pre_attribution_status)
A run the clock ended is read from its per-test journal: tests that
failed before the clock fired are measured failures (named_killer,
status killed). Otherwise the test running then is recorded
(hang, the budget covered the whole run, so it need not hang alone).
With every started test finished, the run is unattributed
(cumulative).
Crash observation (row key active_at_end)
A kill without a named killer whose run ended during a test records the
test the journal shows running: an observation, not a proven killer.
The kill's status and killers stay as recorded.
Derivation (row and matrix key derived)
A subset's result is derived from a measured run without running again.
DERIVATION_ASSUMPTION states when it equals a fresh measurement. A
kill whose run did not record every killing test (a timeout the journal
attributed, or a row with killers_incomplete) is derived as a
timeout is, by its line.
MutationSession.per_suite measures every mutant of one target once per
suite. A suite's verdict is the one its standalone run gives
(PerSuiteRequest.standalone). A faster implementation must give the same
verdicts and killers (conformance.check_per_suite). The verdicts:
killed: a named test failed under the mutant.collection_error: the run failed without a named test: a collection error, a crash, or a result without a clean test outcome.non_viable: the mutant is not a valid program.timeout: the per-mutant time limit ended the run.survived: the suite passed under the mutant.untested: no result (the run failed, the wall clock ended it first, or a filter skipped the mutant). Never counted as survived.
Each verdict but collection_error is the kill-matrix status of the same
word.
CORRECTED
module-attribute
¶
CORRECTED = 'corrected'
A verification's verdict: a completed run that does not kill replaced the recorded kill.
RECOVERED
module-attribute
¶
RECOVERED = 'recovered'
A verification's verdict: a run named the kill's killing tests, which replaced the record.
REPRODUCED_TIMEOUT
module-attribute
¶
REPRODUCED_TIMEOUT = 'reproduced_timeout'
A verification's verdict: the timeout repeated, and the record is kept.
REPRODUCED_WITH_REPORT
module-attribute
¶
REPRODUCED_WITH_REPORT = 'reproduced_with_report'
A verification's verdict: the runner reported the unnamed kill again, and the record is kept.
UNRESOLVED
module-attribute
¶
UNRESOLVED = 'unresolved'
A verification's verdict: neither attempt gave evidence, so the kill is kept and flagged unresolved.
NOT_ATTEMPTED
module-attribute
¶
NOT_ATTEMPTED = 'not_attempted'
A verification's verdict: unresolved without a run, since an earlier batch produced no evidence at all.
CACHED_GENUINE
module-attribute
¶
CACHED_GENUINE = 'cached_genuine'
A verification's verdict: not run again, since the session already found the same mutant's kill genuine.
NOT_VERIFIED
module-attribute
¶
NOT_VERIFIED = 'not_verified'
A verification's verdict: the verification could not run, and the sweep's record is kept.
ATTRIBUTED
module-attribute
¶
ATTRIBUTED = 'attributed'
A file attribution's outcome: each test file whose run alone kills the mutant is recorded.
ATTRIBUTION_INCOMPLETE
module-attribute
¶
ATTRIBUTION_INCOMPLETE = 'incomplete'
A file attribution's outcome: a test file gave no result, so the kill stays unresolved.
POOL_CONTEXT
module-attribute
¶
POOL_CONTEXT = 'pool_context'
A file attribution's outcome: no test file kills alone, so the kill needs several together and stays unresolved.
ATTRIBUTION_FAILED
module-attribute
¶
ATTRIBUTION_FAILED = 'failed'
A file attribution's outcome: the attribution's pass failed before it recorded its outcomes, and the verdicts measured so far are kept.
NAMED_KILLER
module-attribute
¶
NAMED_KILLER = 'named_killer'
The journal's reading of a timeout: tests failed before the clock fired, measured failures that kill the mutant.
HANG
module-attribute
¶
HANG = 'hang'
The journal's reading of a timeout: the test running when the clock fired is recorded.
CUMULATIVE
module-attribute
¶
CUMULATIVE = 'cumulative'
The journal's reading of a timeout: every started test finished, so the run is unattributed.
DERIVATION_ASSUMPTION
module-attribute
¶
DERIVATION_ASSUMPTION = "Derived without running the kept tests: a mutant stays killed when a kept test is one of its recorded killing tests, when it is invalid, when it was killed without a named test, or, for a timeout or a kill whose run did not record every killing test, when the kept tests cover its line or its line is unknown. Otherwise it counts as survived. This equals a fresh measurement of the kept tests only when each kept test's verdict does not depend on the removed tests and every killing test was recorded (attribution 'all'). Kills that depend on a removed test's side effects, kills by a timeout and kills without a named test can differ from a fresh measurement of the kept tests."
What a derived result assumes. A derived kill matrix states it.
Evidence
module-attribute
¶
Evidence = Union[Verification, FileAttribution, TimeoutEvidence, CrashObservation, DerivedVerdict]
One part of a kill-matrix row's evidence, as evidence_of reads
it.
BY_FAILING_TEST
module-attribute
¶
BY_FAILING_TEST = 'failing_test'
A detected mutant's subcategory: a named test failed under it.
WITHOUT_NAMED_TEST
module-attribute
¶
WITHOUT_NAMED_TEST = 'no_named_test'
A detected mutant's subcategory: killed without a named test, by a crash or an error at collection alike.
BY_TIMEOUT
module-attribute
¶
BY_TIMEOUT = 'timeout'
A detected mutant's subcategory: the per-mutant time limit ended the run.
NON_VIABLE
module-attribute
¶
NON_VIABLE = 'non_viable'
A detected mutant's subcategory: it does not compile or load, a property of the source.
SURVIVED
module-attribute
¶
SURVIVED = 'survived'
An undetected mutant's subcategory: the suite passed under it.
INCOMPETENT
module-attribute
¶
INCOMPETENT = 'incompetent'
The subcategory of a mutant not run: the tool could not run the tests for it (Cosmic Ray's INCOMPETENT).
SKIPPED
module-attribute
¶
SKIPPED = 'skipped'
The subcategory of a mutant not run: a filter excluded its operator.
UNTESTED
module-attribute
¶
UNTESTED = 'untested'
The subcategory of a mutant not run: the run ended before it, or no test outcome was recorded.
SUBCATEGORIES
module-attribute
¶
SUBCATEGORIES = {MutationClass.DETECTED: (BY_FAILING_TEST, WITHOUT_NAMED_TEST, BY_TIMEOUT, NON_VIABLE), MutationClass.UNDETECTED: (SURVIVED,), MutationClass.NOT_RUN: (SKIPPED, UNTESTED, INCOMPETENT)}
The subcategories each class admits (Classification).
ATTRIBUTIONS
module-attribute
¶
ATTRIBUTIONS = ('first', 'all')
Killing tests named per mutant: the first (its run stops there) or all.
FROM_REQUEST
module-attribute
¶
FROM_REQUEST = 'request'
Where an effective setting's value came from: the request named it.
FROM_DEFAULT
module-attribute
¶
FROM_DEFAULT = 'default'
Where an effective setting's value came from: the engine's default.
KILLED_STATUS
module-attribute
¶
KILLED_STATUS = 'killed'
The status of a mutant a test failed under, named or not, or whose timeout the per-test journal traced to a failing test.
TIMEOUT_STATUS
module-attribute
¶
TIMEOUT_STATUS = 'timeout'
The status of a mutant whose run reached the per-mutant time limit.
NON_VIABLE_STATUS
module-attribute
¶
NON_VIABLE_STATUS = 'non_viable'
The status of a mutant that does not compile or load.
SURVIVED_STATUS
module-attribute
¶
SURVIVED_STATUS = 'survived'
The status of a mutant the suite let live.
SKIPPED_STATUS
module-attribute
¶
SKIPPED_STATUS = 'skipped'
The status of a mutant a filter excluded. A kill matrix leaves it out.
UNTESTED_STATUS
module-attribute
¶
UNTESTED_STATUS = 'untested'
The status of a mutant not measured: the run ended before it, or no test outcome was recorded.
DETECTED_STATUSES
module-attribute
¶
DETECTED_STATUSES = frozenset({KILLED_STATUS, TIMEOUT_STATUS, NON_VIABLE_STATUS})
The statuses of a detected mutant, which a score counts as killed: the
class MutationClass.DETECTED.
INCOMPLETE_KEY
module-attribute
¶
INCOMPLETE_KEY = 'incomplete'
The stamp on a matrix whose run the wall clock ended with mutants left. Its verdicts are kept, but the mutants left are no random sample: the slow ones are those a timeout would have counted as detected.
UNTESTED_MUTANTS_IN_CATALOGUE
module-attribute
¶
UNTESTED_MUTANTS_IN_CATALOGUE = 'untested_mutants_in_catalogue'
The reason of the INCOMPLETE_KEY stamp on a matrix whose catalogue
holds untested mutants.
RULE_NOT_KILLED
module-attribute
¶
RULE_NOT_KILLED = 'not_killed'
A subset rule (derivation_rule): the measured run did not kill
the mutant.
RULE_NON_VIABLE
module-attribute
¶
RULE_NON_VIABLE = 'non_viable'
A subset rule: the mutant stays killed since it is not viable, a property of the source.
RULE_WITHOUT_NAMED_TEST
module-attribute
¶
RULE_WITHOUT_NAMED_TEST = 'without_named_test'
A subset rule: the mutant stays killed since it was killed without a named test, which every test triggers.
RULE_KEPT_KILLER
module-attribute
¶
RULE_KEPT_KILLER = 'kept_killer'
A subset rule: the mutant stays killed since a kept test is one of its recorded killers.
RULE_COVERED_LINE
module-attribute
¶
RULE_COVERED_LINE = 'covered_line'
A subset rule: the mutant stays killed since it is a timeout, or a kill whose run did not record every killing test, on a line the subset covers.
RULE_TIMEOUT_WITHOUT_LINE
module-attribute
¶
RULE_TIMEOUT_WITHOUT_LINE = 'timeout_without_line'
A subset rule: the mutant stays killed since it is such a kill and its line is unknown.
RULE_NO_KEPT_KILLER
module-attribute
¶
RULE_NO_KEPT_KILLER = 'no_kept_killer'
A subset rule: the mutant survives, with no kept killer and, for such a kill, no covered line.
FAILED_SESSION
module-attribute
¶
FAILED_SESSION = 'session'
What failed in a run that measured nothing (MutationRun.failure):
the session failed or was closed.
FAILED_INITIALIZATION
module-attribute
¶
FAILED_INITIALIZATION = 'initialization'
What failed in a run that measured nothing: the tool did not initialize.
FAILED_INSTRUMENT_CHECK
module-attribute
¶
FAILED_INSTRUMENT_CHECK = 'instrument_check'
What failed in a run that measured nothing: the instrument gave no verdicts.
FAILED_SESSION_READ
module-attribute
¶
FAILED_SESSION_READ = 'session_read'
What failed in a run that measured nothing: the tool's session was unreadable.
FAILED_WORKERS
module-attribute
¶
FAILED_WORKERS = 'workers'
What failed in a run that measured nothing: the worker raised for every mutant, or no mutant has a test outcome.
FAILURES
module-attribute
¶
FAILURES = (FAILED_SESSION, FAILED_INITIALIZATION, FAILED_INSTRUMENT_CHECK, FAILED_SESSION_READ, FAILED_WORKERS)
Every kind of failure of a run that measured nothing.
KILL_MATRIX_SCHEMA_VERSION
module-attribute
¶
KILL_MATRIX_SCHEMA_VERSION = 3
The kill-matrix schema a run writes. A matrix of an earlier version
reads as this one through upgrade_kill_matrix.
COLLECTION_ERROR
module-attribute
¶
COLLECTION_ERROR = 'collection_error'
A per-suite verdict: the run failed without a named test, by a collection error, a crash or a result without a clean test outcome.
VERDICTS
module-attribute
¶
VERDICTS = (KILLED_STATUS, COLLECTION_ERROR, NON_VIABLE_STATUS, TIMEOUT_STATUS, SURVIVED_STATUS, UNTESTED_STATUS)
Every per-suite verdict, in the order of the module's table: the
statuses of those words and COLLECTION_ERROR. UNTESTED_STATUS is a
suite without a result for the mutant, never counted as survived.
DEFAULT_IMPLEMENTATION
module-attribute
¶
DEFAULT_IMPLEMENTATION = 'per_suite_runs'
The implementation that runs one standalone measurement per suite.
NativeRecord
dataclass
¶
NativeRecord(test_outcome: Optional[str], worker_outcome: Optional[str], killers: Tuple[str, ...] = (), reported: bool = False, output: Optional[str] = None)
A per-mutant run as the engine recorded it: test and worker outcomes, named killing tests, whether the runner reported, and any kept output.
from_dict
classmethod
¶
from_dict(data: Mapping[str, Any]) -> 'NativeRecord'
The record that to_dict data holds, an absent key read as
empty.
Verification
dataclass
¶
Verification(verdict: str, clock: bool, recorded: NativeRecord, attempts: Tuple[Optional[NativeRecord], ...] = (), replaced: bool = False)
One kill's verification: verdict, whether by the clock, the sweep's and
each attempt's record (None: no result), and whether the sweep's was replaced.
attempts holds each attempt's record.
FileAttribution
dataclass
¶
FileAttribution(outcome: str, verdicts: Tuple[Tuple[str, Optional[bool]], ...])
The file attribution of an unresolved kill: its outcome and each test file's verdict alone (True kills, False does not, None no result).
TimeoutEvidence
dataclass
¶
TimeoutEvidence(label: str, tests: Tuple[str, ...], status_before: str)
The journal's reading of a run the clock ended: label, the
failing or running tests, and the status before the reading.
status_before is that status.
CrashObservation
dataclass
¶
CrashObservation(active_test: str)
DerivedVerdict
dataclass
¶
DerivedVerdict(measured_status: str, rule: str)
A mutant in a derived result: its measured status and deciding rule.
measured_status is the mutant's status in the measured run.
Derivation
dataclass
¶
Derivation(kept_tests: Tuple[str, ...], covered_lines: Tuple[int, ...], attribution: Optional[str], assumption: str = DERIVATION_ASSUMPTION)
How a result was derived: the kept tests, the target lines they
cover, the measured run's attribution and the assumption. kept_tests
names the kept tests, sorted.
MutationClass ¶
Bases: str, Enum
What a measurement established about one mutant.
DETECTED
class-attribute
instance-attribute
¶
DETECTED = 'detected'
A failing test, a crash or a timeout detected the mutant, or it is not a valid program.
UNDETECTED
class-attribute
instance-attribute
¶
UNDETECTED = 'undetected'
The suite let the mutant live.
NOT_RUN
class-attribute
instance-attribute
¶
NOT_RUN = 'not_run'
The mutant was not measured: a filter skipped it, or no result exists.
Classification
dataclass
¶
Classification(status: str, mutation_class: MutationClass, subcategory: str)
TimeoutRule
dataclass
¶
TimeoutRule(factor: float = 1.5, minimum_s: int = 10, cap_s: Optional[int] = None)
A per-mutant timeout of factor times the passing suite's time,
rounded up, at least minimum_s and at most cap_s seconds (a cap
below the minimum wins). A missing suite time gives the minimum. The
default is TestGenEval's rule: 1.5 times the suite time, at least 10 s.
Catalogue
dataclass
¶
Catalogue(name: Optional[str] = None, operators: Optional[Tuple[str, ...]] = None)
The mutants a run generates: the engine's full catalogue (default), a named operator set of the engine, or an explicit list of operators.
OperatorCatalogue
dataclass
¶
OperatorCatalogue(engine: str, operators: Optional[Tuple[str, ...]], outcome: Outcome = Outcome.COMPLETED, reason: str = '', tool_version: Optional[str] = None)
The operators an engine registers in the project environment, by its
names, else None with the outcome and reason (missing_setup: not
installed, infrastructure_failure: registry unreadable), and the
engine's tool_version when readable.
MutationProgress
dataclass
¶
MutationProgress(kind: str, finished: int, total: Optional[int], elapsed_s: float)
finished of total items done (total None when unknown) after
elapsed_s, the items named by kind: mutants measured, or
kill verifications. Progress, not evidence: a finished mutant's
verdict can still change when its kill is verified or the run fails.
MutationRequest
dataclass
¶
MutationRequest(target: str, suite: Suite, catalogue: Catalogue = Catalogue(), per_mutant_timeout: Union[int, TimeoutRule, None] = None, passing_suite_s: Optional[float] = None, wall_clock_limit_s: Optional[int] = None, attribution: Optional[str] = None, workers: Optional[int] = None, fail_on_unresolved: Optional[bool] = None, timeout_journal: Optional[bool] = None, progress: Optional[Callable[[MutationProgress], None]] = None)
One mutation run: mutate target (a project source file, relative
to its root or absolute under it) and measure the mutants with
suite. Engine defaults for the other fields: the full catalogue, its
per-mutant timeout rule applied to passing_suite_s, a one-hour wall
clock, the first killing test per mutant, the session's workers, and
unresolved kills kept and flagged.
workers=1 runs the mutants one at a time, and fail_on_unresolved
fails a run with an unresolved kill (both suit a retry after running
out of memory). timeout_journal journals each per-mutant run's tests
for the timeout evidence and crash observations. Its default is the
engine's: on for Cosmic Ray, off for an engine that keeps no such journal
(PIT, and Mull for C tests), which refuses it. progress receives
MutationProgress events from a separate thread where the engine
observes them: a local Cosmic Ray run's mutants and every Cosmic Ray
run's kill verifications, none of PIT's or Mull's. A callback failure
never changes the run.
catalogue: the mutants the run generates.per_mutant_timeout: the time limit of each mutant's run, in whole seconds or as aTimeoutRule.passing_suite_s: the passing suite's time a rule applies to. The C engine (Mull) times the suite in its own check, each test as it runs under a mutant, applies a rule to that time and refuses this field.wall_clock_limit_s: the time limit of the whole run in whole seconds.attribution: the killing tests each mutant records:first, where its run stops, orall.
EffectiveSetting
dataclass
¶
EffectiveSetting(value: Any, source: str, inputs: Mapping[str, Any] = dict())
A setting's value, its source (request or default), a rule's inputs.
MutationResolution
dataclass
¶
MutationResolution(requested: str, method: str, reason: str = '')
The requested operation and the mechanism used: a run measures. It
never changes verdicts, completeness or score. Its record holds
prior_identity as null and conditions as an empty list, the
record's fixed form.
to_dict ¶
to_dict() -> Dict[str, Any]
The resolution's record, with prior_identity null and
conditions empty.
from_dict
classmethod
¶
from_dict(value: Mapping[str, Any]) -> 'MutationResolution'
The resolution a record holds. A record with other keys, a
prior_identity or conditions raises InvalidRequest.
MutantKey
dataclass
¶
MutantKey(file: Optional[str], operator: Optional[str], occurrence: Optional[int], line: Optional[int], column: Optional[int], diff_sha256: Optional[str])
A mutant's identity across runs over one source and catalogue: file,
operator, occurrence, location, change SHA-256 (engine ids hold per run).
diff_sha256 is the SHA-256 of the mutant's diff, None without a
diff.
Mutant
dataclass
¶
Mutant(key: MutantKey, engine_id: Any, test_outcome: Optional[str], worker_outcome: Optional[str], classification: Classification, killers: Tuple[str, ...], diff: Optional[str], row: Mapping[str, Any], killing_tests: Tuple[TestId, ...] = (), unmatched_killers: Tuple[str, ...] = ())
One mutant of a run: its kill-matrix row and the row's typed
reading. killers are as the test runner printed them,
killing_tests the same tests within the suite when the run knows
it, and unmatched_killers those naming no test of the suite. A kill
attributed to a file is in neither: its evidence names the file.
MutationIdentity
dataclass
¶
MutationIdentity(observation: ObservationIdentity, catalogue: str, operators: Tuple[str, ...], per_mutant_timeout_s: int, attribution: str, workers: int)
What a mutation run observed: the observation's identity, catalogue,
requested operators, per-mutant timeout, attribution and workers.
per_mutant_timeout_s is the value of the run's per_mutant_timeout
setting in seconds: the timeout, or for PIT the constant its rule adds
to the factor times each test's time.
to_dict ¶
to_dict() -> Dict[str, Any]
The identity as plain data: the observation's to_dict with
the run's settings added.
MutationRun
dataclass
¶
MutationRun(target_file: str, mutants: Tuple[Mutant, ...], summary: Mapping[str, Any], requested_catalogue: str, requested_operators: Tuple[str, ...], incomplete: Optional[Mapping[str, Any]] = None, outcome: Outcome = Outcome.COMPLETED, reason: str = '', identity: Optional[MutationIdentity] = None, settings: Mapping[str, EffectiveSetting] = dict(), derivation: Optional[Derivation] = None, failure: Optional[str] = None, resolution: Optional[MutationResolution] = None, origin_run_s: Optional[float] = None, stopping_tests: Tuple[TestId, ...] = ())
The result of one mutation run over one target file, complete after
its session is gone (to_kill_matrix serializes it). outcome is
completed, timed_out (the wall clock cut it, incomplete), or a
failure without mutants, counts or score, where failure is one of
FAILURES. derivation is set on a result derived for a test
subset. resolution is None when not recorded. origin_run_s is
the time of the unmutated run that checked the suite passes and where
the tests load the target from: operation time, not mutant time (None
without such a run). stopping_tests are, as for
PerTestEvidence, the tests the failure of that run is pinned on: a
test that did not pass in it, or whose own coverage counts in it cannot
be read or are missing. The run then measured nothing. Empty
otherwise.
settings
class-attribute
instance-attribute
¶
settings: Mapping[str, EffectiveSetting] = field(default_factory=dict)
complete
property
¶
complete: bool
Whether the run was measured and every mutant of its catalogue has a result. A run that measured nothing, because a step failed or the suite does not pass, is not complete.
failed
classmethod
¶
failed(target_file: str, outcome: Outcome, reason: str, *, failure: Optional[str] = None, identity: Optional[MutationIdentity] = None, settings: Optional[Mapping[str, EffectiveSetting]] = None, stopping_tests: Iterable[TestId] = ()) -> 'MutationRun'
A run that measured nothing, with the reason, any known
failure and the tests that stopped it.
counts ¶
counts() -> Optional[Dict[MutationClass, int]]
The mutants per common class, or None if nothing was measured.
score ¶
score() -> Optional[float]
The mutation score over the common classes, in percent. None when incomplete, nothing was measured, or no mutant is valid.
progress_score ¶
progress_score() -> Optional[float]
score for any measured run, a lower bound while incomplete (untested
not killed). None when nothing was measured or no mutant is valid.
unmatched_killers ¶
unmatched_killers() -> Tuple[str, ...]
Every killer of the run that names no test of its suite.
derive_for_tests ¶
derive_for_tests(kept: Iterable[TestId], *, covered_lines: Iterable[int]) -> 'MutationRun'
The result for the kept tests of this run's suite, derived by
derivation_rule without running them. covered_lines are the
target lines they cover. Each mutant records its measured status and
rule, the result its Derivation. Raises InvalidRequest for a
derived result, a kept test outside the suite, and a kill whose
killer the run did not place in its suite (attributed to a file, or
a run without a suite).
to_kill_matrix ¶
to_kill_matrix(fields: Optional[Mapping[str, Any]] = None) -> Dict[str, Any]
The run as a kill matrix: schema, target, a row per mutant with
its key and, for a kill with killers, its killing_tests and
any unmatched_killers, counts, catalogue provenance, with a known
identity the suite's files and hashes, the execution and the
attribution (first or all, as the run recorded killing
tests), any derivation and resolution, and the incomplete stamp.
fields are added but cannot replace the resolution.
BudgetedSuite
dataclass
¶
BudgetedSuite(name: str, suite: Suite, per_mutant_timeout: Union[int, TimeoutRule, None] = None, passing_suite_s: Optional[float] = None)
One suite of a per-suite request: unique name, the suite and its
per-mutant budget as in MutationRequest (seconds, a rule applied to
its own passing time, or None for the engine's rule).
PerSuiteRequest
dataclass
¶
PerSuiteRequest(target: str, suites: Tuple[BudgetedSuite, ...], catalogue: Catalogue = Catalogue(), attribution: Optional[str] = None, wall_clock_limit_s: Optional[int] = None, workers: Optional[int] = None, fail_on_unresolved: Optional[bool] = None, timeout_journal: Optional[bool] = None, progress: Optional[Callable[[MutationProgress], None]] = None)
Measure target's mutants with each of suites. The other fields
are MutationRequest settings of every standalone run (each run is
bounded by wall_clock_limit_s).
catalogue: the mutants each run generates.attribution: the killing tests each mutant records,firstorall.workers: the workers of each run.fail_on_unresolved: fails a run with an unresolved kill.timeout_journal: journals each per-mutant run's tests for the timeout evidence and crash observations.progress: receivesMutationProgressevents where the engine observes them.
SuiteVerdict
dataclass
¶
SuiteVerdict(verdict: str, killers: Tuple[str, ...] = (), killing_tests: Tuple[TestId, ...] = ())
One suite's verdict for one mutant, with the killers its run recorded (all or the first, by attribution) and the tests it placed them in.
to_dict ¶
to_dict() -> Dict[str, Any]
The verdict as plain data, with killing_tests only when it
has killers.
MutantVerdicts
dataclass
¶
MutantVerdicts(key: MutantKey, diff: Optional[str], verdicts: Mapping[str, SuiteVerdict])
PerSuiteRun
dataclass
¶
PerSuiteRun(target: str, outcome: Outcome, reason: str, implementation: str, runs: Mapping[str, MutationRun], mutants: Tuple[MutantVerdicts, ...])
The result of MutationSession.per_suite. runs holds each
suite's run, equal to its standalone run, with its outcome, identity,
settings and unresolved kills. mutants aligns each mutant's verdicts
across the suites. outcome is completed when every run completed,
otherwise the first other outcome, with the suites in reason.
implementation names what measured the suites.
verdicts ¶
verdicts(name: str) -> Dict[MutantKey, SuiteVerdict]
Suite name's verdict for every mutant.
kill_matrix ¶
kill_matrix(name: Optional[str] = None, *, fields: Optional[Mapping[str, Any]] = None) -> Dict[str, Any]
Suite name's own kill matrix, or without a name the merge of
all suites, every row with the mutant's key. A merged row is
killed when any suite killed or hit a collection error, else
non_viable when a suite's run found that the mutant did not load
(the C engine's load_error, which belongs to the mutant, not to a
suite), else untested when any has no result, then timeout,
non_viable, survived. Its killers are every suite's in
order, suite_verdicts each suite's verdict. A merge with untested
rows is stamped incomplete.
MutationCleanupError ¶
MutationCleanupError(cleanup_handle: 'MutationSession', reason: str)
Bases: LangAdapterError
Cleanup failed. cleanup_handle.close() retries it, and the handle
keeps its native resources. Never treat it as a successful close or a
scored observation. An interrupt or exit propagates as itself instead,
carrying the unclosed session as cleanup_handle.
MutationSession ¶
MutationSession(*, workers_requested: int = 1, workers_started: int = 0, outcome: Outcome = Outcome.COMPLETED, reason: str = '')
Bases: ABC
A mutation session: one engine measuring one request after another.
An engine constructs it once its opening completed or a failed opening
stopped what it started. An opening that fails after the construction,
such as a worker fleet that does not start, makes each run an
infrastructure failure with its reason once close stopped what it
started. It owns its
resources in measure and release and keeps cleanup handles until
release succeeds. A
concurrent operation is refused, not queued. After an exception or a
cancel the session closes what it owns. If that fails, the error raised
carries it as cleanup_handle. Results stay valid after later runs
and close.
outcome
property
¶
outcome: Outcome
completed for a session that opened, else the outcome each of
its runs reports, such as missing_setup.
workers_requested
property
¶
workers_requested: int
The number of workers the session was asked for.
workers_started
property
¶
workers_started: int
The number of workers it started, from 0 to
workers_requested.
is_open
property
¶
is_open: bool
Whether the session measures runs: it is not closed, and its
outcome is completed.
run ¶
run(request: MutationRequest) -> MutationRun
Measure the request and record its attempted mechanism.
cancel ¶
cancel() -> None
Stop the measurement in flight, from any thread or a signal
handler: its processes are asked to end, and the operation raises
MeasurementCancelled after its cleanup has closed the session,
which then runs nothing more. Harmless when repeated or idle.
per_suite ¶
per_suite(request: PerSuiteRequest) -> PerSuiteRun
Each suite's verdict for every mutant of the target, as its standalone run gives it. Without a faster implementation each suite is measured alone, in request order, within this operation. A failed suite's run does not affect the others.
per_suite_fast ¶
per_suite_fast(request: PerSuiteRequest)
A faster per-suite measurement as (runs by suite name,
implementation name), equal to the standalone runs, or None.
register_owned_commands ¶
register_owned_commands(resources) -> None
Declare acquired persistent commands owned and closed by this session.
measure
abstractmethod
¶
measure(request: MutationRequest) -> MutationRun
The engine's measurement of request: the hook that run and
per_suite call once they validated the request, took the session's
lock and found it open. A caller uses run, since measure alone
does none of that and records no resolution. A returned result means
this operation is quiescent.
release
abstractmethod
¶
release() -> None
The engine's release of what the session holds: the hook that
close calls once no operation is active. It keeps every handle when
cleanup fails. A caller uses close.
evidence_of ¶
evidence_of(row: Mapping[str, Any]) -> Tuple[Evidence, ...]
The evidence a kill-matrix row records, in the methods' order.
resolution_of ¶
resolution_of(matrix: Optional[Mapping[str, Any]]) -> Optional[MutationResolution]
The matrix's resolution, or None (unknown provenance). Refuses an invalid record and a resolution on a matrix derived for a test subset.
has_operators ¶
has_operators(matrix: Optional[Dict[str, Any]]) -> bool
Whether any mutant carries an operator, so selective scores derive.
killed_ids ¶
killed_ids(matrix: Optional[Dict[str, Any]], *, operators: Optional[Set[str]] = None) -> Set[Any]
The ids of killed mutants, optionally of a selective operator set.
valid_ids ¶
valid_ids(matrix: Optional[Dict[str, Any]], *, operators: Optional[Set[str]] = None) -> Set[Any]
The ids that count toward the score: killed, survived and untested
mutants, optionally of a selective operator set. Without untested and
skipped mutants this is Cosmic Ray's denominator (num_results), which
counts a skipped mutant too.
score_percent ¶
score_percent(killed: Set[Any], valid: Set[Any]) -> Optional[float]
The percentage of valid mutants killed, or None without valid ones.
matrix_score_percent ¶
matrix_score_percent(matrix: Optional[Dict[str, Any]], *, operators: Optional[Set[str]] = None) -> Optional[float]
The mutation score of a kill matrix, optionally of a selective
operator set, or None when the matrix is stamped incomplete: the ratio
of its resolved part would read high, since timeouts count as detected.
matrix_lower_bound_percent gives such a matrix's lower bound.
matrix_lower_bound_percent ¶
matrix_lower_bound_percent(matrix: Optional[Dict[str, Any]], *, operators: Optional[Set[str]] = None) -> Optional[float]
The lowest score a kill matrix admits (untested not killed), also when
incomplete, else matrix_score_percent. None without a valid mutant.
unattributed_kill_fraction ¶
unattributed_kill_fraction(matrix: Optional[Dict[str, Any]]) -> Optional[float]
The fraction of killed mutants that are killed without a named
killer (every test triggers them, as a collection error does), or None
without kills. Timeouts and non_viable mutants, killed by nature,
count only in the denominator.
timeout_kill_count ¶
timeout_kill_count(matrix: Optional[Dict[str, Any]]) -> int
How many mutants the per-mutant clock killed (timeout).
unresolved_kill_count ¶
unresolved_kill_count(matrix: Optional[Dict[str, Any]]) -> int
How many mutants carry a kill flagged unresolved.
derivation_rule ¶
derivation_rule(m: Dict[str, Any], kept_killed_ids: Set[str], covers_line: Callable[[Optional[int]], bool]) -> str
The rule (RULE_*) that decides mutant m's status under a
test subset: kept_killed_ids are the mutants its tests kill with a
named failing test (the K:<mutant_id> items of their per-test
evidence), and covers_line(line) whether it reaches a line. A
killed mutant stays killed when it is not viable, killed without a named
test, killed by a kept test, or, on a covered or unknown line, a timeout
or a kill whose run did not record every killing test: a timeout the
journal attributed to the tests that failed before the clock fired
(pre_attribution_status), or a kill a run of every test did not
record (killers_incomplete). Reads only opaque mutant ids, line
numbers and these two row keys.
killed_under ¶
killed_under(m: Dict[str, Any], kept_killed_ids: Set[str], covers_line: Callable[[Optional[int]], bool]) -> bool
Whether mutant m stays killed under a test subset.
refresh_summary ¶
refresh_summary(matrix: Dict[str, Any]) -> Dict[str, Any]
Refresh a matrix's summary after in-place mutant status edits. A
matrix of an earlier schema version first takes the keys of its upgrade
(upgrade_kill_matrix), so its rows are new copies: a row taken from
it before that first refresh no longer belongs to it.
restrict_kill_matrix ¶
restrict_kill_matrix(matrix: Optional[Dict[str, Any]], *, kept_killed_ids: Set[str], kept_lines: Optional[Set[int]] = None) -> Dict[str, Any]
The kill matrix derived for a test subset from the mutants it kills
(kept_killed_ids) and the lines it covers (kept_lines), by
derivation_rule, as MutationRun.derive_for_tests does: a kill
not reproduced becomes survived without killers, all else is kept, and
a smaller subset can only remove kills. Its derived names the kept
kills, lines and assumption, with of_derived for a derived input,
and each row's derived its measured status and rule. Any resolution
is removed.
survivors_from_matrix ¶
survivors_from_matrix(matrix: Optional[Dict[str, Any]], *, operators: Optional[Set[str]] = None) -> List[Dict[str, Any]]
The surviving mutants of a stored kill matrix, each {id, line,
diff} in matrix order, optionally of a selective operator set.
upgrade_kill_matrix ¶
upgrade_kill_matrix(matrix: Mapping[str, Any]) -> Dict[str, Any]
A stored kill matrix of an earlier schema version, or of none, as a
new matrix of KILL_MATRIX_SCHEMA_VERSION: each row's status,
pre_attribution_status and derived status and rule, its per-suite
verdicts and the summary's keys in this version's words, every other
value unchanged but one. A C mutant under which the Python extension did
not load (native status load_error), which version 2 recorded as
killed, becomes non_viable as the C engine records it, still
detected, and the summary's counts follow it. A matrix of this version
comes back as an equal copy. Raises InvalidRequest for a later or
unreadable version.
mutant_key ¶
mutant_key(file: Optional[str], operator: Optional[str], occurrence: Optional[int], line: Optional[int], column: Optional[int], diff: Optional[str]) -> MutantKey
The key of a mutant from its engine record.