Running on a cluster¶
Testuary runs as an ordinary batch job. On a shared cluster a few things need attention that a workstation hides: how many CPUs the mutation workers get, where the mutation engine keeps its working files, the limit on open files, the route of the model's traffic, how many starter requests run at once, and how to continue after a job is cancelled. Your site's documentation names its scheduler options, storage and network rules. The example uses Slurm with placeholders in angle brackets.
Settings can go into an INI file passed with --config, or into environment
variables of the form TESTUARY_<SECTION>_<KEY>, for example
TESTUARY_MUTATION_ENGINE_WORKERS. The
configuration guide explains the layers. Mistakes are
refused before the run starts: an unknown section or key, in a file or as a
variable, and a TESTUARY_ variable that names no known section, such as a
misspelled TESTUARY_COVERAG_MAX_REQUESTS_PER_PHASE. The variables Testuary
reads for their own purpose are not settings and are not refused:
TESTUARY_<PROVIDER>_API_KEY, a variable a model alias names with
api_key_env, TESTUARY_HOME, TESTUARY_CONFIG_FILE, TESTUARY_EXECUTION_ID,
TESTUARY_CLOCK_DOMAIN_ID, TESTUARY_MAIN_WORK_DIR_NAMESPACE and the
container inputs below. Run a small job first and check the log
lines named below before you start many.
Installing¶
Create a virtual environment on storage every node can read, install Testuary
into it with python -m pip install testuary, which brings its pinned
dependencies, and activate it in the job. Check
that python -c "import testuary; print(testuary.__file__)", run from outside
any source checkout, names the installed package. The activated environment
is Testuary's own. Name the project's interpreter with python_executable in
[project] or --python-executable. The
settings reference says which interpreter
a run takes without one. Keep caches and temporary
directories of the tools out of your home directory if your site limits it.
CPUs and mutation workers¶
Mutation testing dominates a run's CPU use. engine_workers in
[mutation] sets how many mutants run in parallel: Cosmic Ray's worker
processes for Python, PIT's threads for Java. For Python,
cosmic_ray_distributor chooses how
Cosmic Ray spreads the mutants over those workers. Each Python worker holds
its own copy of the project. For Java the distributor does not apply.
Give the workers fewer CPUs than the job has, and leave one or two for
Testuary itself and the model client. A layout that has worked: 18 allocated
CPUs, Testuary and its children restricted to 16 of them, 15 workers. Restrict
the processes with taskset -c on the Testuary command, so every child
inherits the set. Read the allowed CPUs inside the job, with
os.sched_getaffinity(0), instead of assuming that they are numbered from 0.
The worker count and the CPUs behind it are part of a run's execution conditions. Per-mutant timeouts derive from the suite's measured runtime, so fewer CPUs than workers can turn slow runs into timeouts and change a score. Record the allocation with the results, and keep it the same across runs you want to compare. Memory is limited for the whole job: the workers, their project copies and any RAM-backed scratch share it.
Node-local storage for the mutation engine¶
For Python, Cosmic Ray copies the project once per worker and records every
mutant's result in a session database. Both are read and written constantly
during a mutation measurement. On a shared network file system this is slow:
every import in every mutant run touches the file system's metadata, and the
database's locking waits on the network. A mutation measurement can then take
many times longer, run into per-mutant timeouts or its time limit
(measurement_timeout_seconds), and give a different score, which in turn
changes what the mutation phase does next.
Two settings in [mutation] decide where this work goes:
cosmic_ray_clone_dir: where the workers' project copies go.cosmic_ray_local_session_dir: where the session database is staged while the mutation measurement runs.
Left at their defaults, the choice depends on how the project runs. When
Testuary runs the project directly, auto puts the copies in /dev/shm
(on Linux and other POSIX systems), else in the system's temporary directory,
each only when it is writable with at least 1 GiB free, else in a
cr_worker_clones directory in the directory the run started from. An empty
session setting then means the system's temporary directory. When the project
runs in a container (project_exec = container in [main]), auto puts the
copies beside the session in the run's work directory, which can be on shared
storage, and an empty session setting picks /dev/shm when it is writable, else
the system's temporary directory.
On many clusters the system's temporary directory (TMPDIR) and the work
directory are on shared storage, so set both settings explicitly to a directory
on the node, unique to the job, for example below /dev/shm. Each session then
takes its own subdirectory there. A configured directory that cannot be created
or written stops the run, with no fallback to another place. For a Python
project run directly, a session directory that cannot be used stops the run when
it starts, before any model request, and testuary check --verify tests it
beforehand. Otherwise the directory is checked before the mutation measurement.
A RAM-backed directory counts against the job's memory. The engine removes a
session's copies when the session ends, and cosmic_ray_clone_sweep_hours
reclaims copies that a killed job left behind (0 switches that off). When the
project runs in a container, the directory must be visible inside the container
too.
The run's log names the choice for each mutation run, on a line from
[CosmicRayEngine]. Look for session SQLite at <directory> with the reason
for the place, and 15/15 workers ready (with your worker count), at every
phase that measures mutation, not only the first.
Running the project's tests in a container¶
Where a site runs software in containers, the project's tests can run in a running container while Testuary runs outside it. Six variables and the source directory in the container select this:
TESTUARY_MAIN_PROJECT_EXEC:localorcontainer. It sets[main] project_exec.TESTUARY_PROJECT_CONTAINER: the name of the running container the tests run in. Required withcontainer.TESTUARY_PROJECT_CONTAINER_RUNNER: the container command,dockerby default.TESTUARY_PROJECT_PYTHON_EXECUTABLE: the project's interpreter, a path inside the container withcontainer. It sets[project] python_executable.LANGADAPTER_HELPER_PYTHON: the interpreter in the container that supervises the project's commands, the project's interpreter by default.TESTUARY_PROJECT_WORK_DIR: the directory on the host for the run's work copies, which the container must also mount. Required withcontainer.--source-dir, orTESTUARY_PROJECT_SOURCE_DIR: the source directory inside the container, an absolute POSIX path. Required withcontainer.
A container run continues with --resume when the container command
(TESTUARY_PROJECT_CONTAINER_RUNNER), the project's interpreter in the
container (TESTUARY_PROJECT_PYTHON_EXECUTABLE) and, for a language such as
Java, its runtime there are the same as when the work was done.
The container's name may differ, so a new container of the same setup in a
later job continues the run. The target and every saved file are still
checked by their hashes. When the command, the interpreter or whether the tests
run in a container changed, the resume does that work again, and the log says
why, for example "the container command or the project's runtime changed".
Open files¶
A mutant run starts pytest, which imports the project's libraries and opens
many files for a moment. With many workers the default limit on open files is
reached, test collection fails with "Too many open files", and the failure
reads as a killed mutant. Each mutant's test run raises its soft limit to
65,536, or to the hard limit when that is lower, and never lowers a higher
limit. The log line that names where the mutation session ran ends with
open-file limit: workers <soft>, tests <limit> (hard <hard>): the limit the
workers have, the limit of each test run and the hard limit. Check
ulimit -Hn in a job: a low hard limit is a matter for your site. Do not set
an unlimited soft limit yourself. Some programs then close every possible file
descriptor when they start a process, which makes each start slow.
Java projects¶
For Java, PIT runs engine_workers as its number of threads. Each
Java process takes a quarter of the job's memory limit as its largest heap
unless told otherwise, and Testuary has no setting for the heap of a Java
project's processes. Set one for every Java process of the job, for example
export JDK_JAVA_OPTIONS=-Xmx1g. Java then notes "Picked up
JDK_JAVA_OPTIONS" on its error output, which is harmless.
A Java run that Testuary runs directly keeps PIT's files and reports, the
compiled suites and its own copies of the project in the system's temporary
directory. A node-local TMPDIR keeps them off shared storage. That is a matter
of speed, not of the results.
On nodes without internet access, Maven needs a local repository that already
holds the project's dependencies, and its offline switch, for example
export MAVEN_ARGS=-o. LangAdapter downloads the JUnit launcher,
JaCoCo and PIT into its cache on first use, ~/.cache/langadapter/java
unless LANGADAPTER_JAVA_TOOLS names another directory. Fill that cache
beforehand and set LANGADAPTER_JAVA_TOOLS_OFFLINE=1 in the job.
With more than one worker, LangAdapter splits the mutants of the class under test into groups of at most 10 for PIT, so the threads share the work. Each group starts a Java process with its own heap.
Model traffic through a proxy¶
Compute nodes often reach the internet only through a proxy. The model client follows the standard proxy variables. Set all spellings, because a site's lowercase variables can take precedence over uppercase ones:
export ALL_PROXY=<proxy URL> all_proxy=<proxy URL>
export HTTPS_PROXY=<proxy URL> https_proxy=<proxy URL>
export HTTP_PROXY=<proxy URL> http_proxy=<proxy URL>
export NO_PROXY=localhost,127.0.0.1,::1 no_proxy=localhost,127.0.0.1,::1
A SOCKS proxy works with a socks5h://host:port address. The h resolves
host names at the proxy, which matters when the node has no name service for
outside addresses. NO_PROXY must cover the loopback addresses, because the
Python mutation workers talk to each other over loopback HTTP. Keep the model
key out of job scripts: Testuary reads it from TESTUARY_<PROVIDER>_API_KEY
or from a private models.ini in the user directory (see the
configuration guide).
Starter request concurrency¶
The starter phase asks the model for several independent suites. These
settings in [starter] decide how many requests run at once:
concurrent_requests: how many requests the starter phase sends at the same time.concurrent_requests_ollamatakes its place whenever the provider isollama, local or hosted.--concurrent-requestson the command line sets both.cache_warmup: send the first starter request alone, then the rest at once, up to that limit, so a provider can cache the shared beginning of the prompt. Whether it does depends on the provider, and savings are not guaranteed. When the first suite's validity check cannot be resolved, the rest is not requested.
One request followed by up to 24 at once is cache_warmup = true with
concurrent_requests = 24. Requests one at a time are
concurrent_requests = 1. A small fixed number such as 4 is
concurrent_requests = 4. Several jobs running at the
same time multiply the load on the provider: count all of them against its
rate limits.
Time limits¶
The scheduler's time limit for a job is separate from Testuary's own limits,
such as suite_run_timeout_seconds or measurement_timeout_seconds. Choose the job's
limit from the elapsed times of comparable runs with headroom. A limit that is
too short wastes the work done so far until you resume, and one that is very
long can delay the job's start.
Continuing after a cancelled job¶
Run the same command again with --resume, the same --output-dir and the
same configuration files. For each repetition of each target, Testuary skips
the phases whose records show them finished, after checking them, and starts
again at the first phase that is not. Results and logs
describes the rules. The node-local directories of the cancelled job are gone
and are created anew. Never let two jobs use the same run directory at once.
Collecting results¶
Write the run directory to shared storage that outlives the job, and copy it after the job has ended, not while it still writes. The directory holds the delivered suites, the phase records and the logs. Results and logs describes its layout and the exit status. A job that the scheduler reports as completed can still contain a target without a suite that meets the minimums: read the summary at the end of the log or the results table for each target's outcome. Leave out the virtual environment and any scratch directories when you archive a run.
Example job script¶
The script assumes one target of one project per job. It runs in the directory
that holds the settings files, and --home names the directory with
models.ini. Name the source directory and the run directory by paths
without links: a path that passes through a link, at any level, is
refused. Replace every value in angle brackets.
#!/bin/bash
#SBATCH --job-name=testuary
#SBATCH --account=<account>
#SBATCH --partition=<partition>
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=18
#SBATCH --mem-per-cpu=2G
#SBATCH --time=<hh:mm:ss>
#SBATCH --output=<log directory>/testuary-%j.out
set -euo pipefail
source <environment directory>/bin/activate
cd <configuration directory>
# Node-local scratch for the mutation engine, unique to this job.
SCRATCH="/dev/shm/testuary_$(id -u)/${SLURM_JOB_ID}"
export TESTUARY_MUTATION_COSMIC_RAY_CLONE_DIR="$SCRATCH/clones"
export TESTUARY_MUTATION_COSMIC_RAY_LOCAL_SESSION_DIR="$SCRATCH/sessions"
mkdir -p "$TESTUARY_MUTATION_COSMIC_RAY_CLONE_DIR" \
"$TESTUARY_MUTATION_COSMIC_RAY_LOCAL_SESSION_DIR"
trap 'rm -rf "$SCRATCH"' EXIT
# Model traffic through the site's proxy, loopback excluded.
export ALL_PROXY=<proxy URL> all_proxy=<proxy URL>
export HTTPS_PROXY="$ALL_PROXY" https_proxy="$ALL_PROXY"
export HTTP_PROXY="$ALL_PROXY" http_proxy="$ALL_PROXY"
export NO_PROXY=localhost,127.0.0.1,::1 no_proxy=localhost,127.0.0.1,::1
# 16 of the allowed CPUs for Testuary and its workers.
CPUS=$(python -c 'import os; print(",".join(map(str, sorted(os.sched_getaffinity(0))[:16])))')
taskset -c "$CPUS" testuary generate \
--home <user directory> --config config.ini --config cluster.ini \
--source-dir <source directory> --source-files <target> \
--output-dir <run directory> ${RESUME:+--resume}
config.ini holds your settings for the run, among them model in
[api], an alias of models.ini, which has no default. The job starts in
the configuration directory, so the project's own settings file,
.testuary/project.ini, is read only when that directory lies inside the
project. Files named with --config
replace the global config.ini of the user directory, so the script names the
run's own config.ini first. cluster.ini
comes second, so its machine settings win, with a cluster.ini such as:
[starter]
concurrent_requests = 4
[mutation]
cosmic_ray_distributor = http
engine_workers = 15
Submit with sbatch <script>. To continue a cancelled job, submit it again
with the same run directory and RESUME=1 in its environment, for example
sbatch --export=ALL,RESUME=1 <script>.