The conformance check¶
langadapter.conformance checks an adapter or a metric provider against the
rules every adapter and provider must follow, the shared contract, on a small
example of its language. Each check returns one
sentence per problem it finds, naming the rule broken, so an adapter's own
tests can assert that there are none.
langadapter tutorial --auto --output-dir toy writes the Toy's adapter as
one file, toy/toy.py, and its example project, toy/project. This script
checks them:
import sys
import langadapter as la
from langadapter.conformance import AdapterExample, check_adapter
la.load_extension("toy/toy.py")
context = la.ProjectContext("toy/project", la.Interpreter(path=sys.executable))
suite = la.Suite(context.root, ("test_calc.toy",))
example = AdapterExample("toy", context, suite, coverage_target="calc.toy",
expected={la.TestId("test_calc.toy", "doubles"): "passed"},
collection_breaker="test adds: 1 + 1 == 2\nthis line is not a test\n")
problems = check_adapter(example)
assert not problems
langadapter.load_extension registers the language a file adds in your own
script, as langadapter check --extension does for the command's check. A
language registered in code (register) needs no load.
An AdapterExample names the language, a ProjectContext and a Suite of
that project that runs: its tests pass or fail rather than fail to collect, and
each declared test file holds at least one test. expected optionally maps
some of its tests, as TestId(file, name) keys, to their outcome words, such
as passed. A coverage_target (a source file of the project) adds the
coverage check. An adapter that prunes collection breakers also needs a
collection_breaker: the text of a test source with one unit that keeps its
tests from loading, the language's unit that
Collection breakers
names. For Python that is a module-level statement that fails at
collection:
collection_breaker="BROKEN = undefined_name\n\n\ndef test_ok():\n assert True\n"
What check_adapter checks¶
- the registered file extensions map the example's test files to the language
createbinds an adapter of the language to the context- the adapter declares its support as a
LanguageSupportand its profile is aVersionProfile(langadapter.versions) - each of its
project_settingshas the kindpath,executable,choiceortext, whichlangadapter checknames too - an adapter that prunes collection breakers declares the unit it removes
(the adapter's
collection_unit), removes the example'scollection_breakerand names what it removed, and leaves the result unchanged when it prunes it again - the operation report lists every operation, with a reason for each unsupported one
- the capability report names only known capabilities
- a suite run reports consistent counts, at least one test outcome for every
declared test file and the outcomes
expectednames, or answersunsupportedwhere suite runs are not implemented - where
provides_timeout_journalis on, a run with the journal reports as consistently - coverage is measured where the capability report offers it and refused where it does not
- the quick check (
inspect_project) reports on every operation, given the test files that lie inside the project (none for a suite outside it) - each of the language's own metric providers (
metric_providers) gives every file metric a value for the example's first test file
What compare_runs compares¶
The normal test run decides whether a test passes, and the coverage and
mutation tools only measure (One run decides
says why). compare_runs(example) runs the example suite normally
and, with its coverage_target, through each tool the adapter offers: the
coverage run, the per-test evidence and a mutation run of that file with the
first engine's default catalogue, on a new adapter object, so that a mutation
run which works only after another run of the same object shows. It names
each difference: a test the
coverage run gives another outcome, a green evidence run of a suite the normal
run fails or none of a suite it passes, and a mutation tool that measures a
suite the normal run fails, refuses one it passes, or names other tests than
the normal run's failing ones as the tests that stopped it
(MutationRun.stopping_tests).
Find differences early¶
A long run of a program such as Testuary meets a difference between runs only
when a generated test happens to trigger it, often after hours and many model
requests. compare_runs looks for differences on your example in the time of
a few runs, one of them a mutation run with the first engine's whole default
catalogue. It returns one sentence per difference, which names the run and,
where a test differs, the test. Give the example a coverage_target: the
tools' runs measure that file, and without one compare_runs runs only the
normal run and compares no tool. A tool the adapter or the context does not
offer is skipped, so an empty tuple means that every tool offered agrees
with the normal run. A shared language package's own test can assert that it
finds no difference.
It finds what the example's tests can show. Run it on two examples: one whose tests pass, and one with a failing test, which checks that the mutation tool refuses that suite and names that test. Tests that read their working directory, a file by a relative path, the import or class path, or an environment variable show whether the tools' runs see the normal run's environment. A test that writes a file shows whether a run leaves anything behind for the next one.
What each sentence means, and where to look:
normal run: the adapter runs no suite, so no tool's run compares, ornormal run: the example suite did not run (<outcome>: <reason>): nothing else is compared. Writerun_suitefirst, or give the example a suite whose tests pass or fail rather than fail to load.coverage run: <test> was failed, in the normal run passed: the coverage run starts the tests otherwise than the normal run. Let it reuse the normal run's command and environment. When the tool itself causes the difference, such as a hidden member a test inspects, it stays, and a consumer gives such a suite no coverage number.coverage run: no outcome for <test>, which the normal run ran, orcoverage run: <test> is no test of the normal run: the coverage run selects other tests than the normal run, or names them otherwise. Give it the normal run's selection and test ids.coverage run: the suite did not run (<outcome>: <reason>): the coverage run failed where the normal run ran. Its reason says why.per-test evidence: no evidence of a suite the normal run passed, ora green run of a suite whose normal run failed: the same for the evidence run.per-test evidence: no evidence for <test>, which the normal run ran: the evidence run left out a test that the normal run ran, as a coverage run that selects other tests does.mutation tool's run: stopped by <tests>, which the normal run passed: the tool's check of the unchanged suite runs in another place or environment than the normal run. Run it where the first mutants run, with the normal run's environment.mutation tool's run: measured nothing of a suite the normal run passed, with the tool's outcome and reason: the tool failed or refused a passing suite without naming a test. Its reason says why, and the same causes as above apply.mutation tool's run: measured a suite whose normal run failed: under the tool a failing test passed or did not run. Compare the tool's test selection, order and environment with the normal run's.mutation tool's run: stopped by ..., where the normal run failed ...: the tool refused the suite, but the tests the adapter names as stopping it, orno named test, are not the normal run's failing tests. ReadMutationRun.stopping_testsfrom the tool's report.
What check_metric_provider checks¶
A metric provider shows the call and in which
process to run it. It registers the provider on the language's adapter, as
its users do. A provider without a name, a name defined twice, or a metric
the adapter
already has, such as nloc, is refused there. Then it checks that every
file metric gets a value. When unparsable= gives source
the provider cannot read, it also checks that no metric reports a number for
it.
What check_per_suite checks¶
A mutation session may answer per_suite with its own faster
implementation (per_suite_fast). check_per_suite(session, request) runs it and the
reference, which measures each suite on its own, in the same session. Both
must give every suite the same outcome, the same mutants and, for each
mutant, the same verdict and killers. Choose suites that reach every
verdict the engine can give, and suites that would disturb each other if
they ran in one process. A session without a faster implementation has
nothing to compare.
None of the checks judges whether a language's measurements are right. That is the job of the adapter's own tests against its tools.
Running it on the example¶
examples/toy_adapter.py --check <directory> runs check_adapter on the Toy's
adapter at step 1 of langadapter tutorial and prints each problem it finds. The directory must hold
at least one test_*.toy test file, and the .toy sources its tests use.
The tutorial's example project is one, which
langadapter tutorial --auto --output-dir toy writes to toy/project, at a
pace for watching, or faster with --speed 50. The examples are where
Getting started says. With <prefix>
the environment's directory named there, give the example's full path from
the directory that holds toy:
python <prefix>/share/doc/langadapter/examples/toy_adapter.py --check toy/project
A pass prints the Toy adapter keeps the adapter contract, and each problem
is one line. A missing directory, or one without a test_*.toy file, is
refused in one line.