Timing¶
tests.benchmark times new processes: every repetition starts a new process,
so a discarded warm-up warms only what lasts between processes, such as
Python's bytecode cache, which the repetitions share. Benchmark results keep
the final repetition's exit code when it is available, and progress callbacks
run outside the timed interval.
Warmed Java timing¶
tests.benchmark_warmed (Java, locally or in a container) starts a number of forks,
each a fresh JVM process, and runs every warm-up and measured iteration of the
suite in the fork's JVM:
from langadapter.runs import WarmedSuiteBenchmarkRequest
result = adapter.tests.benchmark_warmed(WarmedSuiteBenchmarkRequest(
suite, "native_lifecycle", iteration_timeout_s=60, total_timeout_s=900,
forks=3, warmup_runs=5, runs=10))
The project is built and the suite compiled once, outside the timed
intervals, and each fork runs in its own copy of that build. Only
Launcher.execute of a fresh JUnit test plan is timed, in nanoseconds, by the
JVM. The suite's fixtures and the engine's listeners run inside that interval.
"native_lifecycle" states that each iteration runs the suite's own fixtures
as JUnit runs them, inside the timed interval, and that static state stays in
the JVM: nothing is reset for you.
A warmed benchmark runs on the project's own context. Inside a prepared
copy's use it raises OwnershipRefused: each fork prepares its own copy of
the build, and a prepared copy refuses a further copy while an operation runs
in it.
In a container (see Containers), the copy of the build and
each fork's copy are made inside the container, as a prepared project's copy
is, and the build, the forks and their JVMs run there. The timed interval is
the same Launcher.execute, so the runner's round trip into the container is
no part of a sample. A requested processor set uses the container's own
taskset, and allowed_cpus is read inside the container. Without the
container's helper Python the operation answers missing_setup and runs
nothing.
A suite is repeat-safe when its fixtures restore what its tests change:
class CounterTest {
private Counter counter;
@BeforeEach void fresh() { counter = new Counter(); }
@Test void counts() { counter.add(); assertEquals(1, counter.value()); }
}
A suite that keeps state between iterations is not:
class CounterTest {
static Counter counter = new Counter();
@Test void counts() { counter.add(); assertEquals(1, counter.value()); }
}
Its second iteration fails, the outcome pattern changes, and the result is
non_repeatable.
Validity¶
A result is complete, incomplete, non_repeatable or no_samples. Only a
complete result carries the mean of fork means, the median of fork medians and
the sample range, and with at least two forks the standard deviation between
forks. These describe the samples and are not confidence intervals. Every
iteration's raw record is kept either way. Python and other languages answer
unsupported.
Suite profiles¶
tests.profile_suite(SuiteProfileRequest(suite, timeout_s)) runs a suite
once under the language's profiler and attributes the run's time to the
functions it entered. Each
ProfiledFunction has its file as path (relative to the project root
where it lies inside it), its name and start_line, its calls, its own
time as own_time_s, its time with its callees as inclusive_time_s, and
its share of the run's total own time.
A suite profile attributes time but does not time the suite. Python's
profiler is the standard library's cProfile, around each test's setup, call
and teardown: interpreter start, imports and collection are not profiled,
although a benchmark's repetitions include them. cProfile is deterministic:
it adds a cost at every call and return, so functions with many short calls
take a larger share than in an unprofiled run, and every time is longer.
The result names the profiler, its version and its kind (deterministic),
and reports no suite time: total_own_time_s is the profiled run's. Time a
suite with tests.benchmark. Built-in functions are not listed, and their
time stays in the total. Under pytest-xdist each worker profiles the tests it
runs, and the suite profile sums the workers' times per function.
Only a run whose every test passed lists functions. Failing tests, a
collection failure, a timeout and a missing pytest give the run's outcome
and reason. Java answers unsupported and runs nothing.
In a prepared copy (prepare_project) the profiled run puts the copy's
src, lib and python directories that hold a package first on the import
path, so the tests execute the copy's files where an editable install or an
installed copy of the project would provide the original's. A context that
names its import root (LANGADAPTER_PYTHON_IMPORT_ROOT in
ProjectContext.environment) has that directory first instead.
An import hook that still makes the tests execute a module of the project
from outside the copy is caught by the check suite runs in a prepared copy
share: the outcome is infrastructure_failure, the reason names the module
and where it came from, and the profile lists no functions.