Python projects¶
The Python adapter runs a project's pytest suite with the project's own interpreter, measures coverage with coverage.py and mutation with Cosmic Ray. Select the interpreter in the context. When none is selected, the setup check reports a candidate, as Environment selection describes.
An environment that provision_runtime makes for a requested Python version
lives in the directory the caller names (store), in a directory of its own
named after the caller's name and a hash of its inputs: the version, the
requirements file's resolved path and content and the tools the library
pins for that version. uv installs the requirements file from its own
directory, so a relative entry in it, such as -e ., names a path beside the
file. Equal inputs reuse it. Changed inputs make another environment
beside it, so a run that uses the earlier one keeps it. Earlier
environments stay, and one that no run uses may be removed.
Test operations¶
adapter.tests is the typed entry point for run_suite, collect_suite,
run_as_delivered, benchmark, benchmark_warmed, profile_suite and
per_test_evidence.
It takes the requests from langadapter.runs and returns what the tools
observed. The adapter's own methods, such as adapter.run_suite, are the
language-specific operations these groups call, and can be used directly.
Python does not implement benchmark_warmed. An operation the adapter does
not provide still validates its request and returns an explicit
unsupported observation, with unknown counts and no score. An implemented operation without a runtime reports missing setup.
adapter.operation_capabilities() lists what is implemented without starting
a tool. Whether an operation is ready stays unknown until it is checked: the
list does not try anything. adapter.probe_capabilities() adds readiness by
checking what the tools need. An implemented operation can still be
unsupported for the checked project, which is different from having no
implementation. A ready test runner does not mean that every optional request
is ready: coverage and kill attribution need their own capabilities. Reading
source works even when the check of the runtime failed.
Suite results¶
Suite results keep the runner's whole output, its return code and its
structured warnings, and the caller decides how much of the output to show.
Only a text beyond four million characters is cut, a bound against memory
exhaustion: it keeps the first third of those four million characters and the
last two thirds, around [... N chars of suite output omitted ...]. A reason that summarizes the
runner's output lines keeps all of them in reason_detail, and
reason_text(lines) shows the reason with as many as the caller chooses.
The SuiteRun entry lists
the result's fields.
Coverage separates the raw traced lines from covered statements, and only the statements count toward line coverage. coverage.py records the lines of each test under a context name. Per-test evidence lists contexts that match no collected test, with their lines, apart from the identified tests and never credited to a test. Missing timings stay missing.
collect_suite(CollectionRequest(...)) collects an explicit suite without
running its test bodies. Imports, setup files and collection hooks still run.
The result tells a collection error, no tests, a timeout and an execution
failure apart. It is not evidence that the tests pass. The exception summary
is kept apart from the output. The pytest configuration a run uses is the one
the context selects (ProjectContext.test_framework, see
Settings), passed to pytest
explicitly.
A file named test_*.py or *_test.py is a test file, as are conftest.py
and tests.py, whatever their case. The test file named for a source file
calc.py is test_calc.py.
adapter.test_files(paths) replaces each directory by the test files pytest
collects in it. adapter.project_test_files() gives the project's own
test files as its configuration bounds them: pytest collects from the
project's root with no file or directory named, so its testpaths apply. A
directory of test files that testpaths leaves out, such as type tests that
do not collect, is in test_files of the root and not in
project_test_files. Both list the files with collected tests first, then
those pytest could not collect.
Python runner options are langadapter.python.project.PytestOptions.
Benchmark requests may receive progress events after every warm-up run and
after some measured repetitions: the first, the last and about every quarter
of the run. The callbacks are outside the timed interval and do not change a
repetition's verdict.
Generated suites¶
PythonAdapter.prepare_suite(suite_root) writes an import shim, a
conftest.py that makes the project importable, into an existing directory of
generated tests. It replaces that directory's conftest.py, so do not point
it at a setup file that people maintain. The configuration anchor, a
pytest.ini that fixes pytest's root and options for the generated suite, is
written only on request:
prepared = adapter.prepare_suite(generated_directory, isolation_config=True)
print(prepared.outcome, prepared.reason, prepared.setup_files)
The import shim follows the project context, including container path
mappings. The variables through which the library talks to its own helpers
all start with LANGADAPTER_. Every run also sets the variables
Environment selection lists. Ordinary
suite runs never prepare a suite implicitly.
langadapter.python.project provides the configuration lookup and the
staged copy for callers that build their own runner commands.
PythonAdapter.run_as_delivered runs each file of a generated suite in
its own session with the request's session_seed. It keeps the setup
file's import paths, leaves out a measurement helper the setup file
carries, and does not modify the suite. The request's
timeout_s bounds the whole run: each file runs in what is left of it,
and a run whose time is spent ends timed_out before the next file.
With record_leaks and the request's
module_under_test it names the tests that add, rebind or remove a name
of that module, or that change numpy's error or print settings or xarray's
options when those libraries are loaded. A read-only observer, loaded on
the command line, takes these before each test and compares them after the
test's teardown. A value changed in place, such as an item appended to a
list the module holds, is not named. The observer imports no library and
restores nothing, so a test that undoes an earlier test's change is named
too. request_options("delivered", ...) passes the caller's traceback and
first-failure stop, pytest's own traceback by default, and record_leaks.
A named target becomes the module under test, whether or not the record
is asked for: in a prepared project it also decides which file the run
imports, the copy's, and it is the module the leak record compares.
Python imports in a copy
says when a run is refused instead, such as under an editable install's
import hook. The request's timeout_journal applies as in run_suite.
The optional
sources replace the text of a listed test file for that run only, and
the observation identifies the replacement bytes. This operation refuses
conftest.py files in directories above the test files. Ordinary suites use
run_suite.
Which test killed which mutant¶
PythonAdapter.kill_tokens reads named kills and crash observations from a
saved kill matrix: each kill's killing tests, as the run recorded them, and
the test a crash ended in. It runs no measurement and derives no score. It
reports named killers it cannot attribute uniquely, and crash observations
never become kill items.