Skip to content

Mutation

langadapter.mutation

Mutation measurement shared by every engine: requests and settings, mutant classes, the kill-matrix schema and scores, the evidence behind a recorded verdict, per-suite verdicts, run results and the mutation session.

A request's settings default to the engine's. A run records each value and its source (EffectiveSetting), the per-mutant timeout also its rule and inputs. Every mutant keeps its engine's native outcome and carries one common class (MutationClass) with a subcategory. A score counts detected mutants as killed, undetected and untested ones as not killed, and leaves skipped ones out. Without untested and skipped mutants it is 100 minus Cosmic Ray's cr-rate, a survival rate that counts a skipped mutant as killed. Cosmic Ray's INCOMPETENT, where running the tests or applying the mutation raised, measured nothing: it is untested, where cr-rate counts it as killed.

Five methods change or explain what a run records in a mutant's kill-matrix row (evidence_of). None equals a fresh measurement beyond what it states:

Verification (row key verification) A kill without a named killing test, or by the per-mutant clock, runs again, at most twice. A completed run that does not kill replaces the record (corrected), as does a kill naming its killing tests (recovered). A repeated timeout (reproduced_timeout) or an unnamed kill the runner reported (reproduced_with_report) is kept. A kill without evidence in either attempt is kept and flagged unresolved. A nondeterministic test can give either verdict. File attribution (row key adjudication) An unresolved kill runs against each test file alone, and each file whose run kills is recorded, never a test. A kill that needs several files together stays unresolved (pool_context). Timeout evidence (row keys timeout_evidence, pre_attribution_status) A run the clock ended is read from its per-test journal: tests that failed before the clock fired are measured failures (named_killer, status killed). Otherwise the test running then is recorded (hang, the budget covered the whole run, so it need not hang alone). With every started test finished, the run is unattributed (cumulative). Crash observation (row key active_at_end) A kill without a named killer whose run ended during a test records the test the journal shows running: an observation, not a proven killer. The kill's status and killers stay as recorded. Derivation (row and matrix key derived) A subset's result is derived from a measured run without running again. DERIVATION_ASSUMPTION states when it equals a fresh measurement. A kill whose run did not record every killing test (a timeout the journal attributed, or a row with killers_incomplete) is derived as a timeout is, by its line.

MutationSession.per_suite measures every mutant of one target once per suite. A suite's verdict is the one its standalone run gives (PerSuiteRequest.standalone). A faster implementation must give the same verdicts and killers (conformance.check_per_suite). The verdicts:

  • killed: a named test failed under the mutant.
  • collection_error: the run failed without a named test: a collection error, a crash, or a result without a clean test outcome.
  • non_viable: the mutant is not a valid program.
  • timeout: the per-mutant time limit ended the run.
  • survived: the suite passed under the mutant.
  • untested: no result (the run failed, the wall clock ended it first, or a filter skipped the mutant). Never counted as survived.

Each verdict but collection_error is the kill-matrix status of the same word.

CORRECTED module-attribute

CORRECTED = 'corrected'

A verification's verdict: a completed run that does not kill replaced the recorded kill.

RECOVERED module-attribute

RECOVERED = 'recovered'

A verification's verdict: a run named the kill's killing tests, which replaced the record.

REPRODUCED_TIMEOUT module-attribute

REPRODUCED_TIMEOUT = 'reproduced_timeout'

A verification's verdict: the timeout repeated, and the record is kept.

REPRODUCED_WITH_REPORT module-attribute

REPRODUCED_WITH_REPORT = 'reproduced_with_report'

A verification's verdict: the runner reported the unnamed kill again, and the record is kept.

UNRESOLVED module-attribute

UNRESOLVED = 'unresolved'

A verification's verdict: neither attempt gave evidence, so the kill is kept and flagged unresolved.

NOT_ATTEMPTED module-attribute

NOT_ATTEMPTED = 'not_attempted'

A verification's verdict: unresolved without a run, since an earlier batch produced no evidence at all.

CACHED_GENUINE module-attribute

CACHED_GENUINE = 'cached_genuine'

A verification's verdict: not run again, since the session already found the same mutant's kill genuine.

NOT_VERIFIED module-attribute

NOT_VERIFIED = 'not_verified'

A verification's verdict: the verification could not run, and the sweep's record is kept.

ATTRIBUTED module-attribute

ATTRIBUTED = 'attributed'

A file attribution's outcome: each test file whose run alone kills the mutant is recorded.

ATTRIBUTION_INCOMPLETE module-attribute

ATTRIBUTION_INCOMPLETE = 'incomplete'

A file attribution's outcome: a test file gave no result, so the kill stays unresolved.

POOL_CONTEXT module-attribute

POOL_CONTEXT = 'pool_context'

A file attribution's outcome: no test file kills alone, so the kill needs several together and stays unresolved.

ATTRIBUTION_FAILED module-attribute

ATTRIBUTION_FAILED = 'failed'

A file attribution's outcome: the attribution's pass failed before it recorded its outcomes, and the verdicts measured so far are kept.

NAMED_KILLER module-attribute

NAMED_KILLER = 'named_killer'

The journal's reading of a timeout: tests failed before the clock fired, measured failures that kill the mutant.

HANG module-attribute

HANG = 'hang'

The journal's reading of a timeout: the test running when the clock fired is recorded.

CUMULATIVE module-attribute

CUMULATIVE = 'cumulative'

The journal's reading of a timeout: every started test finished, so the run is unattributed.

DERIVATION_ASSUMPTION module-attribute

DERIVATION_ASSUMPTION = "Derived without running the kept tests: a mutant stays killed when a kept test is one of its recorded killing tests, when it is invalid, when it was killed without a named test, or, for a timeout or a kill whose run did not record every killing test, when the kept tests cover its line or its line is unknown. Otherwise it counts as survived. This equals a fresh measurement of the kept tests only when each kept test's verdict does not depend on the removed tests and every killing test was recorded (attribution 'all'). Kills that depend on a removed test's side effects, kills by a timeout and kills without a named test can differ from a fresh measurement of the kept tests."

What a derived result assumes. A derived kill matrix states it.

Evidence module-attribute

Evidence = Union[Verification, FileAttribution, TimeoutEvidence, CrashObservation, DerivedVerdict]

One part of a kill-matrix row's evidence, as evidence_of reads it.

BY_FAILING_TEST module-attribute

BY_FAILING_TEST = 'failing_test'

A detected mutant's subcategory: a named test failed under it.

WITHOUT_NAMED_TEST module-attribute

WITHOUT_NAMED_TEST = 'no_named_test'

A detected mutant's subcategory: killed without a named test, by a crash or an error at collection alike.

BY_TIMEOUT module-attribute

BY_TIMEOUT = 'timeout'

A detected mutant's subcategory: the per-mutant time limit ended the run.

NON_VIABLE module-attribute

NON_VIABLE = 'non_viable'

A detected mutant's subcategory: it does not compile or load, a property of the source.

SURVIVED module-attribute

SURVIVED = 'survived'

An undetected mutant's subcategory: the suite passed under it.

INCOMPETENT module-attribute

INCOMPETENT = 'incompetent'

The subcategory of a mutant not run: the tool could not run the tests for it (Cosmic Ray's INCOMPETENT).

SKIPPED module-attribute

SKIPPED = 'skipped'

The subcategory of a mutant not run: a filter excluded its operator.

UNTESTED module-attribute

UNTESTED = 'untested'

The subcategory of a mutant not run: the run ended before it, or no test outcome was recorded.

SUBCATEGORIES module-attribute

SUBCATEGORIES = {MutationClass.DETECTED: (BY_FAILING_TEST, WITHOUT_NAMED_TEST, BY_TIMEOUT, NON_VIABLE), MutationClass.UNDETECTED: (SURVIVED,), MutationClass.NOT_RUN: (SKIPPED, UNTESTED, INCOMPETENT)}

The subcategories each class admits (Classification).

ATTRIBUTIONS module-attribute

ATTRIBUTIONS = ('first', 'all')

Killing tests named per mutant: the first (its run stops there) or all.

FROM_REQUEST module-attribute

FROM_REQUEST = 'request'

Where an effective setting's value came from: the request named it.

FROM_DEFAULT module-attribute

FROM_DEFAULT = 'default'

Where an effective setting's value came from: the engine's default.

KILLED_STATUS module-attribute

KILLED_STATUS = 'killed'

The status of a mutant a test failed under, named or not, or whose timeout the per-test journal traced to a failing test.

TIMEOUT_STATUS module-attribute

TIMEOUT_STATUS = 'timeout'

The status of a mutant whose run reached the per-mutant time limit.

NON_VIABLE_STATUS module-attribute

NON_VIABLE_STATUS = 'non_viable'

The status of a mutant that does not compile or load.

SURVIVED_STATUS module-attribute

SURVIVED_STATUS = 'survived'

The status of a mutant the suite let live.

SKIPPED_STATUS module-attribute

SKIPPED_STATUS = 'skipped'

The status of a mutant a filter excluded. A kill matrix leaves it out.

UNTESTED_STATUS module-attribute

UNTESTED_STATUS = 'untested'

The status of a mutant not measured: the run ended before it, or no test outcome was recorded.

DETECTED_STATUSES module-attribute

DETECTED_STATUSES = frozenset({KILLED_STATUS, TIMEOUT_STATUS, NON_VIABLE_STATUS})

The statuses of a detected mutant, which a score counts as killed: the class MutationClass.DETECTED.

INCOMPLETE_KEY module-attribute

INCOMPLETE_KEY = 'incomplete'

The stamp on a matrix whose run the wall clock ended with mutants left. Its verdicts are kept, but the mutants left are no random sample: the slow ones are those a timeout would have counted as detected.

UNTESTED_MUTANTS_IN_CATALOGUE module-attribute

UNTESTED_MUTANTS_IN_CATALOGUE = 'untested_mutants_in_catalogue'

The reason of the INCOMPLETE_KEY stamp on a matrix whose catalogue holds untested mutants.

RULE_NOT_KILLED module-attribute

RULE_NOT_KILLED = 'not_killed'

A subset rule (derivation_rule): the measured run did not kill the mutant.

RULE_NON_VIABLE module-attribute

RULE_NON_VIABLE = 'non_viable'

A subset rule: the mutant stays killed since it is not viable, a property of the source.

RULE_WITHOUT_NAMED_TEST module-attribute

RULE_WITHOUT_NAMED_TEST = 'without_named_test'

A subset rule: the mutant stays killed since it was killed without a named test, which every test triggers.

RULE_KEPT_KILLER module-attribute

RULE_KEPT_KILLER = 'kept_killer'

A subset rule: the mutant stays killed since a kept test is one of its recorded killers.

RULE_COVERED_LINE module-attribute

RULE_COVERED_LINE = 'covered_line'

A subset rule: the mutant stays killed since it is a timeout, or a kill whose run did not record every killing test, on a line the subset covers.

RULE_TIMEOUT_WITHOUT_LINE module-attribute

RULE_TIMEOUT_WITHOUT_LINE = 'timeout_without_line'

A subset rule: the mutant stays killed since it is such a kill and its line is unknown.

RULE_NO_KEPT_KILLER module-attribute

RULE_NO_KEPT_KILLER = 'no_kept_killer'

A subset rule: the mutant survives, with no kept killer and, for such a kill, no covered line.

FAILED_SESSION module-attribute

FAILED_SESSION = 'session'

What failed in a run that measured nothing (MutationRun.failure): the session failed or was closed.

FAILED_INITIALIZATION module-attribute

FAILED_INITIALIZATION = 'initialization'

What failed in a run that measured nothing: the tool did not initialize.

FAILED_INSTRUMENT_CHECK module-attribute

FAILED_INSTRUMENT_CHECK = 'instrument_check'

What failed in a run that measured nothing: the instrument gave no verdicts.

FAILED_SESSION_READ module-attribute

FAILED_SESSION_READ = 'session_read'

What failed in a run that measured nothing: the tool's session was unreadable.

FAILED_WORKERS module-attribute

FAILED_WORKERS = 'workers'

What failed in a run that measured nothing: the worker raised for every mutant, or no mutant has a test outcome.

FAILURES module-attribute

FAILURES = (FAILED_SESSION, FAILED_INITIALIZATION, FAILED_INSTRUMENT_CHECK, FAILED_SESSION_READ, FAILED_WORKERS)

Every kind of failure of a run that measured nothing.

KILL_MATRIX_SCHEMA_VERSION module-attribute

KILL_MATRIX_SCHEMA_VERSION = 3

The kill-matrix schema a run writes. A matrix of an earlier version reads as this one through upgrade_kill_matrix.

COLLECTION_ERROR module-attribute

COLLECTION_ERROR = 'collection_error'

A per-suite verdict: the run failed without a named test, by a collection error, a crash or a result without a clean test outcome.

VERDICTS module-attribute

VERDICTS = (KILLED_STATUS, COLLECTION_ERROR, NON_VIABLE_STATUS, TIMEOUT_STATUS, SURVIVED_STATUS, UNTESTED_STATUS)

Every per-suite verdict, in the order of the module's table: the statuses of those words and COLLECTION_ERROR. UNTESTED_STATUS is a suite without a result for the mutant, never counted as survived.

DEFAULT_IMPLEMENTATION module-attribute

DEFAULT_IMPLEMENTATION = 'per_suite_runs'

The implementation that runs one standalone measurement per suite.

NativeRecord dataclass

NativeRecord(test_outcome: Optional[str], worker_outcome: Optional[str], killers: Tuple[str, ...] = (), reported: bool = False, output: Optional[str] = None)

A per-mutant run as the engine recorded it: test and worker outcomes, named killing tests, whether the runner reported, and any kept output.

test_outcome instance-attribute

test_outcome: Optional[str]

worker_outcome instance-attribute

worker_outcome: Optional[str]

killers class-attribute instance-attribute

killers: Tuple[str, ...] = ()

reported class-attribute instance-attribute

reported: bool = False

output class-attribute instance-attribute

output: Optional[str] = None

to_dict

to_dict() -> Dict[str, Any]

The record as plain data, with output only when it is kept.

from_dict classmethod

from_dict(data: Mapping[str, Any]) -> 'NativeRecord'

The record that to_dict data holds, an absent key read as empty.

Verification dataclass

Verification(verdict: str, clock: bool, recorded: NativeRecord, attempts: Tuple[Optional[NativeRecord], ...] = (), replaced: bool = False)

One kill's verification: verdict, whether by the clock, the sweep's and each attempt's record (None: no result), and whether the sweep's was replaced. attempts holds each attempt's record.

kind class-attribute

kind: str = 'verification'

verdict instance-attribute

verdict: str

clock instance-attribute

clock: bool

recorded instance-attribute

recorded: NativeRecord

attempts class-attribute instance-attribute

attempts: Tuple[Optional[NativeRecord], ...] = ()

replaced class-attribute instance-attribute

replaced: bool = False

to_dict

to_dict() -> Dict[str, Any]

The verification as plain data, each record in its to_dict form.

from_dict classmethod

from_dict(data: Mapping[str, Any]) -> 'Verification'

The verification that to_dict data holds.

FileAttribution dataclass

FileAttribution(outcome: str, verdicts: Tuple[Tuple[str, Optional[bool]], ...])

The file attribution of an unresolved kill: its outcome and each test file's verdict alone (True kills, False does not, None no result).

kind class-attribute

kind: str = 'file_attribution'

outcome instance-attribute

outcome: str

verdicts instance-attribute

verdicts: Tuple[Tuple[str, Optional[bool]], ...]

files property

files: Tuple[str, ...]

The files that kill the mutant alone.

to_dict

to_dict() -> Dict[str, Any]

The attribution as plain data: outcome, and files with each file's verdict.

from_dict classmethod

from_dict(data: Mapping[str, Any]) -> 'FileAttribution'

The attribution that to_dict data holds.

TimeoutEvidence dataclass

TimeoutEvidence(label: str, tests: Tuple[str, ...], status_before: str)

The journal's reading of a run the clock ended: label, the failing or running tests, and the status before the reading. status_before is that status.

kind class-attribute

kind: str = 'timeout'

label instance-attribute

label: str

tests instance-attribute

tests: Tuple[str, ...]

status_before instance-attribute

status_before: str

CrashObservation dataclass

CrashObservation(active_test: str)

The test running when a mutant ended the test process, on a kill without a named killer. An observation, not a proven killer. active_test names that test.

kind class-attribute

kind: str = 'crash_observation'

active_test instance-attribute

active_test: str

DerivedVerdict dataclass

DerivedVerdict(measured_status: str, rule: str)

A mutant in a derived result: its measured status and deciding rule. measured_status is the mutant's status in the measured run.

kind class-attribute

kind: str = 'derived'

measured_status instance-attribute

measured_status: str

rule instance-attribute

rule: str

to_dict

to_dict() -> Dict[str, Any]

The verdict as plain data: measured_status and rule.

Derivation dataclass

Derivation(kept_tests: Tuple[str, ...], covered_lines: Tuple[int, ...], attribution: Optional[str], assumption: str = DERIVATION_ASSUMPTION)

How a result was derived: the kept tests, the target lines they cover, the measured run's attribution and the assumption. kept_tests names the kept tests, sorted.

kept_tests instance-attribute

kept_tests: Tuple[str, ...]

covered_lines instance-attribute

covered_lines: Tuple[int, ...]

attribution instance-attribute

attribution: Optional[str]

assumption class-attribute instance-attribute

assumption: str = DERIVATION_ASSUMPTION

to_dict

to_dict() -> Dict[str, Any]

The derivation as plain data, its kind test_subset.

MutationClass

Bases: str, Enum

What a measurement established about one mutant.

DETECTED class-attribute instance-attribute

DETECTED = 'detected'

A failing test, a crash or a timeout detected the mutant, or it is not a valid program.

UNDETECTED class-attribute instance-attribute

UNDETECTED = 'undetected'

The suite let the mutant live.

NOT_RUN class-attribute instance-attribute

NOT_RUN = 'not_run'

The mutant was not measured: a filter skipped it, or no result exists.

Classification dataclass

Classification(status: str, mutation_class: MutationClass, subcategory: str)

One mutant's kill-matrix status, common class and subcategory. A skipped mutant's status is skipped. A kill matrix leaves it out.

status instance-attribute

status: str

mutation_class instance-attribute

mutation_class: MutationClass

subcategory instance-attribute

subcategory: str

TimeoutRule dataclass

TimeoutRule(factor: float = 1.5, minimum_s: int = 10, cap_s: Optional[int] = None)

A per-mutant timeout of factor times the passing suite's time, rounded up, at least minimum_s and at most cap_s seconds (a cap below the minimum wins). A missing suite time gives the minimum. The default is TestGenEval's rule: 1.5 times the suite time, at least 10 s.

factor class-attribute instance-attribute

factor: float = 1.5

minimum_s class-attribute instance-attribute

minimum_s: int = 10

cap_s class-attribute instance-attribute

cap_s: Optional[int] = None

seconds

seconds(passing_suite_s: Optional[float]) -> int

The per-mutant timeout for a suite that passes in passing_suite_s seconds.

inputs

inputs(passing_suite_s: Optional[float]) -> Dict[str, Any]

The rule and the suite time it is applied to.

Catalogue dataclass

Catalogue(name: Optional[str] = None, operators: Optional[Tuple[str, ...]] = None)

The mutants a run generates: the engine's full catalogue (default), a named operator set of the engine, or an explicit list of operators.

name class-attribute instance-attribute

name: Optional[str] = None

operators class-attribute instance-attribute

operators: Optional[Tuple[str, ...]] = None

kind property

kind: str

full, named or operators.

OperatorCatalogue dataclass

OperatorCatalogue(engine: str, operators: Optional[Tuple[str, ...]], outcome: Outcome = Outcome.COMPLETED, reason: str = '', tool_version: Optional[str] = None)

The operators an engine registers in the project environment, by its names, else None with the outcome and reason (missing_setup: not installed, infrastructure_failure: registry unreadable), and the engine's tool_version when readable.

engine instance-attribute

engine: str

operators instance-attribute

operators: Optional[Tuple[str, ...]]

outcome class-attribute instance-attribute

outcome: Outcome = Outcome.COMPLETED

reason class-attribute instance-attribute

reason: str = ''

tool_version class-attribute instance-attribute

tool_version: Optional[str] = None

MutationProgress dataclass

MutationProgress(kind: str, finished: int, total: Optional[int], elapsed_s: float)

finished of total items done (total None when unknown) after elapsed_s, the items named by kind: mutants measured, or kill verifications. Progress, not evidence: a finished mutant's verdict can still change when its kill is verified or the run fails.

kind instance-attribute

kind: str

finished instance-attribute

finished: int

total instance-attribute

total: Optional[int]

elapsed_s instance-attribute

elapsed_s: float

MutationRequest dataclass

MutationRequest(target: str, suite: Suite, catalogue: Catalogue = Catalogue(), per_mutant_timeout: Union[int, TimeoutRule, None] = None, passing_suite_s: Optional[float] = None, wall_clock_limit_s: Optional[int] = None, attribution: Optional[str] = None, workers: Optional[int] = None, fail_on_unresolved: Optional[bool] = None, timeout_journal: Optional[bool] = None, progress: Optional[Callable[[MutationProgress], None]] = None)

One mutation run: mutate target (a project source file, relative to its root or absolute under it) and measure the mutants with suite. Engine defaults for the other fields: the full catalogue, its per-mutant timeout rule applied to passing_suite_s, a one-hour wall clock, the first killing test per mutant, the session's workers, and unresolved kills kept and flagged.

workers=1 runs the mutants one at a time, and fail_on_unresolved fails a run with an unresolved kill (both suit a retry after running out of memory). timeout_journal journals each per-mutant run's tests for the timeout evidence and crash observations. Its default is the engine's: on for Cosmic Ray, off for an engine that keeps no such journal (PIT, and Mull for C tests), which refuses it. progress receives MutationProgress events from a separate thread where the engine observes them: a local Cosmic Ray run's mutants and every Cosmic Ray run's kill verifications, none of PIT's or Mull's. A callback failure never changes the run.

  • catalogue: the mutants the run generates.
  • per_mutant_timeout: the time limit of each mutant's run, in whole seconds or as a TimeoutRule.
  • passing_suite_s: the passing suite's time a rule applies to. The C engine (Mull) times the suite in its own check, each test as it runs under a mutant, applies a rule to that time and refuses this field.
  • wall_clock_limit_s: the time limit of the whole run in whole seconds.
  • attribution: the killing tests each mutant records: first, where its run stops, or all.

target instance-attribute

target: str

suite instance-attribute

suite: Suite

catalogue class-attribute instance-attribute

catalogue: Catalogue = Catalogue()

per_mutant_timeout class-attribute instance-attribute

per_mutant_timeout: Union[int, TimeoutRule, None] = None

passing_suite_s class-attribute instance-attribute

passing_suite_s: Optional[float] = None

wall_clock_limit_s class-attribute instance-attribute

wall_clock_limit_s: Optional[int] = None

attribution class-attribute instance-attribute

attribution: Optional[str] = None

workers class-attribute instance-attribute

workers: Optional[int] = None

fail_on_unresolved class-attribute instance-attribute

fail_on_unresolved: Optional[bool] = None

timeout_journal class-attribute instance-attribute

timeout_journal: Optional[bool] = None

progress class-attribute instance-attribute

progress: Optional[Callable[[MutationProgress], None]] = field(default=None, repr=False, compare=False)

validate

validate() -> None

Refuse a request the engine cannot interpret, naming the field.

EffectiveSetting dataclass

EffectiveSetting(value: Any, source: str, inputs: Mapping[str, Any] = dict())

A setting's value, its source (request or default), a rule's inputs.

value instance-attribute

value: Any

source instance-attribute

source: str

inputs class-attribute instance-attribute

inputs: Mapping[str, Any] = field(default_factory=dict)

to_dict

to_dict() -> Dict[str, Any]

The setting as plain data: value, source and inputs.

MutationResolution dataclass

MutationResolution(requested: str, method: str, reason: str = '')

The requested operation and the mechanism used: a run measures. It never changes verdicts, completeness or score. Its record holds prior_identity as null and conditions as an empty list, the record's fixed form.

requested instance-attribute

requested: str

method instance-attribute

method: str

reason class-attribute instance-attribute

reason: str = ''

to_dict

to_dict() -> Dict[str, Any]

The resolution's record, with prior_identity null and conditions empty.

from_dict classmethod

from_dict(value: Mapping[str, Any]) -> 'MutationResolution'

The resolution a record holds. A record with other keys, a prior_identity or conditions raises InvalidRequest.

MutantKey dataclass

MutantKey(file: Optional[str], operator: Optional[str], occurrence: Optional[int], line: Optional[int], column: Optional[int], diff_sha256: Optional[str])

A mutant's identity across runs over one source and catalogue: file, operator, occurrence, location, change SHA-256 (engine ids hold per run). diff_sha256 is the SHA-256 of the mutant's diff, None without a diff.

file instance-attribute

file: Optional[str]

operator instance-attribute

operator: Optional[str]

occurrence instance-attribute

occurrence: Optional[int]

line instance-attribute

line: Optional[int]

column instance-attribute

column: Optional[int]

diff_sha256 instance-attribute

diff_sha256: Optional[str]

Mutant dataclass

Mutant(key: MutantKey, engine_id: Any, test_outcome: Optional[str], worker_outcome: Optional[str], classification: Classification, killers: Tuple[str, ...], diff: Optional[str], row: Mapping[str, Any], killing_tests: Tuple[TestId, ...] = (), unmatched_killers: Tuple[str, ...] = ())

One mutant of a run: its kill-matrix row and the row's typed reading. killers are as the test runner printed them, killing_tests the same tests within the suite when the run knows it, and unmatched_killers those naming no test of the suite. A kill attributed to a file is in neither: its evidence names the file.

key instance-attribute

key: MutantKey

engine_id instance-attribute

engine_id: Any

test_outcome instance-attribute

test_outcome: Optional[str]

worker_outcome instance-attribute

worker_outcome: Optional[str]

classification instance-attribute

classification: Classification

killers instance-attribute

killers: Tuple[str, ...]

diff instance-attribute

diff: Optional[str]

row instance-attribute

row: Mapping[str, Any]

killing_tests class-attribute instance-attribute

killing_tests: Tuple[TestId, ...] = ()

unmatched_killers class-attribute instance-attribute

unmatched_killers: Tuple[str, ...] = ()

status property

status: str

The kill-matrix status.

mutant_id property

mutant_id: Any

The row's mutant_id, stable across runs over the same source and catalogue, else the engine id (Cosmic Ray's changes per session).

evidence property

evidence: Tuple[Evidence, ...]

The evidence recorded for this mutant (evidence_of).

MutationIdentity dataclass

MutationIdentity(observation: ObservationIdentity, catalogue: str, operators: Tuple[str, ...], per_mutant_timeout_s: int, attribution: str, workers: int)

What a mutation run observed: the observation's identity, catalogue, requested operators, per-mutant timeout, attribution and workers. per_mutant_timeout_s is the value of the run's per_mutant_timeout setting in seconds: the timeout, or for PIT the constant its rule adds to the factor times each test's time.

observation instance-attribute

observation: ObservationIdentity

catalogue instance-attribute

catalogue: str

operators instance-attribute

operators: Tuple[str, ...]

per_mutant_timeout_s instance-attribute

per_mutant_timeout_s: int

attribution instance-attribute

attribution: str

workers instance-attribute

workers: int

to_dict

to_dict() -> Dict[str, Any]

The identity as plain data: the observation's to_dict with the run's settings added.

MutationRun dataclass

MutationRun(target_file: str, mutants: Tuple[Mutant, ...], summary: Mapping[str, Any], requested_catalogue: str, requested_operators: Tuple[str, ...], incomplete: Optional[Mapping[str, Any]] = None, outcome: Outcome = Outcome.COMPLETED, reason: str = '', identity: Optional[MutationIdentity] = None, settings: Mapping[str, EffectiveSetting] = dict(), derivation: Optional[Derivation] = None, failure: Optional[str] = None, resolution: Optional[MutationResolution] = None, origin_run_s: Optional[float] = None, stopping_tests: Tuple[TestId, ...] = ())

The result of one mutation run over one target file, complete after its session is gone (to_kill_matrix serializes it). outcome is completed, timed_out (the wall clock cut it, incomplete), or a failure without mutants, counts or score, where failure is one of FAILURES. derivation is set on a result derived for a test subset. resolution is None when not recorded. origin_run_s is the time of the unmutated run that checked the suite passes and where the tests load the target from: operation time, not mutant time (None without such a run). stopping_tests are, as for PerTestEvidence, the tests the failure of that run is pinned on: a test that did not pass in it, or whose own coverage counts in it cannot be read or are missing. The run then measured nothing. Empty otherwise.

target_file instance-attribute

target_file: str

mutants instance-attribute

mutants: Tuple[Mutant, ...]

summary instance-attribute

summary: Mapping[str, Any]

requested_catalogue instance-attribute

requested_catalogue: str

requested_operators instance-attribute

requested_operators: Tuple[str, ...]

incomplete class-attribute instance-attribute

incomplete: Optional[Mapping[str, Any]] = None

outcome class-attribute instance-attribute

outcome: Outcome = Outcome.COMPLETED

reason class-attribute instance-attribute

reason: str = ''

identity class-attribute instance-attribute

identity: Optional[MutationIdentity] = None

settings class-attribute instance-attribute

settings: Mapping[str, EffectiveSetting] = field(default_factory=dict)

derivation class-attribute instance-attribute

derivation: Optional[Derivation] = None

failure class-attribute instance-attribute

failure: Optional[str] = None

resolution class-attribute instance-attribute

resolution: Optional[MutationResolution] = None

origin_run_s class-attribute instance-attribute

origin_run_s: Optional[float] = None

stopping_tests class-attribute instance-attribute

stopping_tests: Tuple[TestId, ...] = ()

measured property

measured: bool

Whether the run completed or was cut by the wall clock.

complete property

complete: bool

Whether the run was measured and every mutant of its catalogue has a result. A run that measured nothing, because a step failed or the suite does not pass, is not complete.

failed classmethod

failed(target_file: str, outcome: Outcome, reason: str, *, failure: Optional[str] = None, identity: Optional[MutationIdentity] = None, settings: Optional[Mapping[str, EffectiveSetting]] = None, stopping_tests: Iterable[TestId] = ()) -> 'MutationRun'

A run that measured nothing, with the reason, any known failure and the tests that stopped it.

counts

counts() -> Optional[Dict[MutationClass, int]]

The mutants per common class, or None if nothing was measured.

score

score() -> Optional[float]

The mutation score over the common classes, in percent. None when incomplete, nothing was measured, or no mutant is valid.

progress_score

progress_score() -> Optional[float]

score for any measured run, a lower bound while incomplete (untested not killed). None when nothing was measured or no mutant is valid.

survivors

survivors() -> Tuple[Mutant, ...]

The mutants the suite let live, with their changes.

unmatched_killers

unmatched_killers() -> Tuple[str, ...]

Every killer of the run that names no test of its suite.

derive_for_tests

derive_for_tests(kept: Iterable[TestId], *, covered_lines: Iterable[int]) -> 'MutationRun'

The result for the kept tests of this run's suite, derived by derivation_rule without running them. covered_lines are the target lines they cover. Each mutant records its measured status and rule, the result its Derivation. Raises InvalidRequest for a derived result, a kept test outside the suite, and a kill whose killer the run did not place in its suite (attributed to a file, or a run without a suite).

to_kill_matrix

to_kill_matrix(fields: Optional[Mapping[str, Any]] = None) -> Dict[str, Any]

The run as a kill matrix: schema, target, a row per mutant with its key and, for a kill with killers, its killing_tests and any unmatched_killers, counts, catalogue provenance, with a known identity the suite's files and hashes, the execution and the attribution (first or all, as the run recorded killing tests), any derivation and resolution, and the incomplete stamp. fields are added but cannot replace the resolution.

BudgetedSuite dataclass

BudgetedSuite(name: str, suite: Suite, per_mutant_timeout: Union[int, TimeoutRule, None] = None, passing_suite_s: Optional[float] = None)

One suite of a per-suite request: unique name, the suite and its per-mutant budget as in MutationRequest (seconds, a rule applied to its own passing time, or None for the engine's rule).

name instance-attribute

name: str

suite instance-attribute

suite: Suite

per_mutant_timeout class-attribute instance-attribute

per_mutant_timeout: Union[int, TimeoutRule, None] = None

passing_suite_s class-attribute instance-attribute

passing_suite_s: Optional[float] = None

validate

validate() -> None

Refuse an empty name, a suite that is no Suite or an invalid per-mutant timeout with InvalidRequest, naming the field.

PerSuiteRequest dataclass

PerSuiteRequest(target: str, suites: Tuple[BudgetedSuite, ...], catalogue: Catalogue = Catalogue(), attribution: Optional[str] = None, wall_clock_limit_s: Optional[int] = None, workers: Optional[int] = None, fail_on_unresolved: Optional[bool] = None, timeout_journal: Optional[bool] = None, progress: Optional[Callable[[MutationProgress], None]] = None)

Measure target's mutants with each of suites. The other fields are MutationRequest settings of every standalone run (each run is bounded by wall_clock_limit_s).

  • catalogue: the mutants each run generates.
  • attribution: the killing tests each mutant records, first or all.
  • workers: the workers of each run.
  • fail_on_unresolved: fails a run with an unresolved kill.
  • timeout_journal: journals each per-mutant run's tests for the timeout evidence and crash observations.
  • progress: receives MutationProgress events where the engine observes them.

target instance-attribute

target: str

suites instance-attribute

suites: Tuple[BudgetedSuite, ...]

catalogue class-attribute instance-attribute

catalogue: Catalogue = Catalogue()

attribution class-attribute instance-attribute

attribution: Optional[str] = None

wall_clock_limit_s class-attribute instance-attribute

wall_clock_limit_s: Optional[int] = None

workers class-attribute instance-attribute

workers: Optional[int] = None

fail_on_unresolved class-attribute instance-attribute

fail_on_unresolved: Optional[bool] = None

timeout_journal class-attribute instance-attribute

timeout_journal: Optional[bool] = None

progress class-attribute instance-attribute

progress: Optional[Callable[[MutationProgress], None]] = field(default=None, repr=False, compare=False)

names property

names: Tuple[str, ...]

The suites' names, in request order.

validate

validate() -> None

Refuse a request no standalone run could interpret.

standalone

standalone(name: str) -> MutationRequest

The standalone run that defines suite name's verdicts.

SuiteVerdict dataclass

SuiteVerdict(verdict: str, killers: Tuple[str, ...] = (), killing_tests: Tuple[TestId, ...] = ())

One suite's verdict for one mutant, with the killers its run recorded (all or the first, by attribution) and the tests it placed them in.

verdict instance-attribute

verdict: str

killers class-attribute instance-attribute

killers: Tuple[str, ...] = ()

killing_tests class-attribute instance-attribute

killing_tests: Tuple[TestId, ...] = ()

to_dict

to_dict() -> Dict[str, Any]

The verdict as plain data, with killing_tests only when it has killers.

MutantVerdicts dataclass

MutantVerdicts(key: MutantKey, diff: Optional[str], verdicts: Mapping[str, SuiteVerdict])

One mutant's verdict from every suite, by suite name in request order. key matches the mutant across the suites' runs.

key instance-attribute

key: MutantKey

diff instance-attribute

diff: Optional[str]

verdicts instance-attribute

verdicts: Mapping[str, SuiteVerdict]

PerSuiteRun dataclass

PerSuiteRun(target: str, outcome: Outcome, reason: str, implementation: str, runs: Mapping[str, MutationRun], mutants: Tuple[MutantVerdicts, ...])

The result of MutationSession.per_suite. runs holds each suite's run, equal to its standalone run, with its outcome, identity, settings and unresolved kills. mutants aligns each mutant's verdicts across the suites. outcome is completed when every run completed, otherwise the first other outcome, with the suites in reason. implementation names what measured the suites.

target instance-attribute

target: str

outcome instance-attribute

outcome: Outcome

reason instance-attribute

reason: str

implementation instance-attribute

implementation: str

runs instance-attribute

runs: Mapping[str, MutationRun]

mutants instance-attribute

mutants: Tuple[MutantVerdicts, ...]

verdicts

verdicts(name: str) -> Dict[MutantKey, SuiteVerdict]

Suite name's verdict for every mutant.

kill_matrix

kill_matrix(name: Optional[str] = None, *, fields: Optional[Mapping[str, Any]] = None) -> Dict[str, Any]

Suite name's own kill matrix, or without a name the merge of all suites, every row with the mutant's key. A merged row is killed when any suite killed or hit a collection error, else non_viable when a suite's run found that the mutant did not load (the C engine's load_error, which belongs to the mutant, not to a suite), else untested when any has no result, then timeout, non_viable, survived. Its killers are every suite's in order, suite_verdicts each suite's verdict. A merge with untested rows is stamped incomplete.

MutationCleanupError

MutationCleanupError(cleanup_handle: 'MutationSession', reason: str)

Bases: LangAdapterError

Cleanup failed. cleanup_handle.close() retries it, and the handle keeps its native resources. Never treat it as a successful close or a scored observation. An interrupt or exit propagates as itself instead, carrying the unclosed session as cleanup_handle.

cleanup_handle instance-attribute

cleanup_handle = cleanup_handle

reason instance-attribute

reason = reason

MutationSession

MutationSession(*, workers_requested: int = 1, workers_started: int = 0, outcome: Outcome = Outcome.COMPLETED, reason: str = '')

Bases: ABC

A mutation session: one engine measuring one request after another.

An engine constructs it once its opening completed or a failed opening stopped what it started. An opening that fails after the construction, such as a worker fleet that does not start, makes each run an infrastructure failure with its reason once close stopped what it started. It owns its resources in measure and release and keeps cleanup handles until release succeeds. A concurrent operation is refused, not queued. After an exception or a cancel the session closes what it owns. If that fails, the error raised carries it as cleanup_handle. Results stay valid after later runs and close.

outcome property

outcome: Outcome

completed for a session that opened, else the outcome each of its runs reports, such as missing_setup.

reason property

reason: str

Why the session did not open, empty for one that opened.

workers_requested property

workers_requested: int

The number of workers the session was asked for.

workers_started property

workers_started: int

The number of workers it started, from 0 to workers_requested.

is_open property

is_open: bool

Whether the session measures runs: it is not closed, and its outcome is completed.

run

run(request: MutationRequest) -> MutationRun

Measure the request and record its attempted mechanism.

cancel

cancel() -> None

Stop the measurement in flight, from any thread or a signal handler: its processes are asked to end, and the operation raises MeasurementCancelled after its cleanup has closed the session, which then runs nothing more. Harmless when repeated or idle.

per_suite

per_suite(request: PerSuiteRequest) -> PerSuiteRun

Each suite's verdict for every mutant of the target, as its standalone run gives it. Without a faster implementation each suite is measured alone, in request order, within this operation. A failed suite's run does not affect the others.

per_suite_fast

per_suite_fast(request: PerSuiteRequest)

A faster per-suite measurement as (runs by suite name, implementation name), equal to the standalone runs, or None.

close

close() -> None

Close only when no operation is active, and retain handles on failure.

register_owned_commands

register_owned_commands(resources) -> None

Declare acquired persistent commands owned and closed by this session.

measure abstractmethod

measure(request: MutationRequest) -> MutationRun

The engine's measurement of request: the hook that run and per_suite call once they validated the request, took the session's lock and found it open. A caller uses run, since measure alone does none of that and records no resolution. A returned result means this operation is quiescent.

release abstractmethod

release() -> None

The engine's release of what the session holds: the hook that close calls once no operation is active. It keeps every handle when cleanup fails. A caller uses close.

evidence_of

evidence_of(row: Mapping[str, Any]) -> Tuple[Evidence, ...]

The evidence a kill-matrix row records, in the methods' order.

resolution_of

resolution_of(matrix: Optional[Mapping[str, Any]]) -> Optional[MutationResolution]

The matrix's resolution, or None (unknown provenance). Refuses an invalid record and a resolution on a matrix derived for a test subset.

has_operators

has_operators(matrix: Optional[Dict[str, Any]]) -> bool

Whether any mutant carries an operator, so selective scores derive.

killed_ids

killed_ids(matrix: Optional[Dict[str, Any]], *, operators: Optional[Set[str]] = None) -> Set[Any]

The ids of killed mutants, optionally of a selective operator set.

valid_ids

valid_ids(matrix: Optional[Dict[str, Any]], *, operators: Optional[Set[str]] = None) -> Set[Any]

The ids that count toward the score: killed, survived and untested mutants, optionally of a selective operator set. Without untested and skipped mutants this is Cosmic Ray's denominator (num_results), which counts a skipped mutant too.

score_percent

score_percent(killed: Set[Any], valid: Set[Any]) -> Optional[float]

The percentage of valid mutants killed, or None without valid ones.

matrix_score_percent

matrix_score_percent(matrix: Optional[Dict[str, Any]], *, operators: Optional[Set[str]] = None) -> Optional[float]

The mutation score of a kill matrix, optionally of a selective operator set, or None when the matrix is stamped incomplete: the ratio of its resolved part would read high, since timeouts count as detected. matrix_lower_bound_percent gives such a matrix's lower bound.

matrix_lower_bound_percent

matrix_lower_bound_percent(matrix: Optional[Dict[str, Any]], *, operators: Optional[Set[str]] = None) -> Optional[float]

The lowest score a kill matrix admits (untested not killed), also when incomplete, else matrix_score_percent. None without a valid mutant.

unattributed_kill_fraction

unattributed_kill_fraction(matrix: Optional[Dict[str, Any]]) -> Optional[float]

The fraction of killed mutants that are killed without a named killer (every test triggers them, as a collection error does), or None without kills. Timeouts and non_viable mutants, killed by nature, count only in the denominator.

timeout_kill_count

timeout_kill_count(matrix: Optional[Dict[str, Any]]) -> int

How many mutants the per-mutant clock killed (timeout).

unresolved_kill_count

unresolved_kill_count(matrix: Optional[Dict[str, Any]]) -> int

How many mutants carry a kill flagged unresolved.

derivation_rule

derivation_rule(m: Dict[str, Any], kept_killed_ids: Set[str], covers_line: Callable[[Optional[int]], bool]) -> str

The rule (RULE_*) that decides mutant m's status under a test subset: kept_killed_ids are the mutants its tests kill with a named failing test (the K:<mutant_id> items of their per-test evidence), and covers_line(line) whether it reaches a line. A killed mutant stays killed when it is not viable, killed without a named test, killed by a kept test, or, on a covered or unknown line, a timeout or a kill whose run did not record every killing test: a timeout the journal attributed to the tests that failed before the clock fired (pre_attribution_status), or a kill a run of every test did not record (killers_incomplete). Reads only opaque mutant ids, line numbers and these two row keys.

killed_under

killed_under(m: Dict[str, Any], kept_killed_ids: Set[str], covers_line: Callable[[Optional[int]], bool]) -> bool

Whether mutant m stays killed under a test subset.

refresh_summary

refresh_summary(matrix: Dict[str, Any]) -> Dict[str, Any]

Refresh a matrix's summary after in-place mutant status edits. A matrix of an earlier schema version first takes the keys of its upgrade (upgrade_kill_matrix), so its rows are new copies: a row taken from it before that first refresh no longer belongs to it.

restrict_kill_matrix

restrict_kill_matrix(matrix: Optional[Dict[str, Any]], *, kept_killed_ids: Set[str], kept_lines: Optional[Set[int]] = None) -> Dict[str, Any]

The kill matrix derived for a test subset from the mutants it kills (kept_killed_ids) and the lines it covers (kept_lines), by derivation_rule, as MutationRun.derive_for_tests does: a kill not reproduced becomes survived without killers, all else is kept, and a smaller subset can only remove kills. Its derived names the kept kills, lines and assumption, with of_derived for a derived input, and each row's derived its measured status and rule. Any resolution is removed.

survivors_from_matrix

survivors_from_matrix(matrix: Optional[Dict[str, Any]], *, operators: Optional[Set[str]] = None) -> List[Dict[str, Any]]

The surviving mutants of a stored kill matrix, each {id, line, diff} in matrix order, optionally of a selective operator set.

upgrade_kill_matrix

upgrade_kill_matrix(matrix: Mapping[str, Any]) -> Dict[str, Any]

A stored kill matrix of an earlier schema version, or of none, as a new matrix of KILL_MATRIX_SCHEMA_VERSION: each row's status, pre_attribution_status and derived status and rule, its per-suite verdicts and the summary's keys in this version's words, every other value unchanged but one. A C mutant under which the Python extension did not load (native status load_error), which version 2 recorded as killed, becomes non_viable as the C engine records it, still detected, and the summary's counts follow it. A matrix of this version comes back as an equal copy. Raises InvalidRequest for a later or unreadable version.

mutant_key

mutant_key(file: Optional[str], operator: Optional[str], occurrence: Optional[int], line: Optional[int], column: Optional[int], diff: Optional[str]) -> MutantKey

The key of a mutant from its engine record.