Getting started¶
Install¶
Testuary runs on Python 3.10 or newer. It is supported on Windows and Linux. On macOS it is expected to work, but this release has not run on a Mac: Tested environments says what is known. The project you test can use a different Python version, or a JDK for Java. You select it separately.
Install Testuary into a virtual environment of its own, outside the directory of
the project you test, for example by running the commands below in your home
directory. Testuary pins the versions of pytest, coverage, Cosmic Ray and PuLP
that its measurements use, and in a shared environment pip would replace other
versions of these packages. Inside the directory you test, Testuary's own
environment can also count as the project's, as
python_executable describes. pip
installs LangAdapter (package langadapter) with Testuary, which runs the tests
and tools of each language:
python3 -m venv testuary-env
source testuary-env/bin/activate
python -m pip install testuary
testuary --version
testuary --help
On Windows, create the environment with python -m venv testuary-env and
activate it with testuary-env\Scripts\activate in cmd or PowerShell. In
PowerShell, run Set-ExecutionPolicy -Scope Process RemoteSigned first: the
default policy of Windows PowerShell refuses the activation script, and this
command allows local scripts in that window only. On Debian and Ubuntu,
python3 -m venv needs a package, as
Set up your machine describes.
Testuary comes with a local web interface. Start it in the directory you work in:
testuary-web
It opens the interface in your browser, or in an SSH login prints the
ssh -L line for your own computer. The web interface
describes its pages and, under Access, how to start it
on a machine you share.
testuary --version names the installed versions of Testuary and LangAdapter.
The project you test needs its own working test tools. A Python project needs pytest, coverage and Cosmic Ray in its virtual environment. Tested environments lists the tested versions, which the commands below install. The quick check lists the versions it finds. Install them with the project's interpreter:
/path/to/project-env/bin/python -m pip install pytest==8.1.1 coverage==7.6.8 cosmic-ray==8.4.1
For a project on Python 3.14 or 3.15:
/path/to/project-env/bin/python -m pip install pytest==8.4.2 coverage==7.16.2 cosmic-ray==8.4.1
A Java project needs
a JDK and the project's Maven or Gradle build (its wrapper or the tool on the
PATH) with its test framework. LangAdapter downloads the JUnit
launcher, JaCoCo and PIT into its cache on first use, unless
LANGADAPTER_JAVA_TOOLS_OFFLINE is set to 1. Testuary runs the project's
build and test code as it is. That is not a sandbox, so test only code you
trust.
To see a run of all four phases without a model, a project or any test tools, play back the offline demonstration:
testuary --demo --output-dir demo-python
Set up with the wizard¶
testuary setup asks for what a run needs and writes it into the same files
you would edit by hand. Run it in your project's directory:
testuary setup
It has three parts, asked in this order:
- Model access, in
models.iniin your user directory: the kind of model service, its address, the model and an alias for it. Where the service needs a key, the key is read without being shown. It goes intomodels.ini, which is then readable by you only, or you name an environment variable that holds it. The wizard checks access with the service's free list of models. Before it asks, it requests that list from Ollama athttp://localhost:11434, waiting at most two seconds. If Ollama on this computer answers, it is the suggested choice. - This machine, in
config.iniin your user directory: how many mutants run at the same time (about half the CPUs Testuary may use, which a container's CPU limit lowers, at most 15) and how many concurrent requests the model service gets in the starter phase. - The project, in its settings file
.testuary/project.iniin the project directory: the project directory, whose files are tested (the current one by default), the language, for Python the directory its packages are imported from and the interpreter, for Java the JDK and, without a wrapper, the build tool, the targets (the files to generate tests for) and the model alias to use. It also creates.testuary/results/.gitignore, so the results of runs stay out of version control.
The user directory holds your models and the settings of this machine for every project. The configuration guide says where it is and describes every file Testuary reads.
Each question shows its default in brackets, and Enter accepts it. Model
access and this machine's parallel work, when already set, are passed with
one line. Every other part, the project included, can be skipped. An existing
section of models.ini or config.ini is replaced only after you confirm.
In the project's settings file, the settings the wizard asks about take
your answers and every other setting in the file stays. When its input
ends, the wizard stops before it writes the part it is asking about. Parts
already confirmed keep what they wrote. After a project is written, it runs
testuary check in the project's directory.
Run one part on its own with testuary setup model, testuary setup
machine or testuary setup project [NAME], for example to set up a second
project. NAME is the project's name in results and records, by default the
name of its directory.
In the project's directory, or in any directory below it, a run then needs no options:
testuary
testuary src/pkg/module.py
testuary check
testuary generates tests for every target, each source file the project's
settings file lists, with its model alias. Each target goes through all four
phases with its own model requests, and a run of more than one target lists
the targets before the first request. Naming
targets runs only those, and a name starts from the directory you run the
command in. A target outside the project's source directory, or not in its
language, is refused before any work. --model and the other options of
testuary generate still apply. testuary check checks the same run without
starting it.
Without the wizard, a command names the model with --model: the packaged
settings choose no model, so a run without one in an option or a settings file is
refused before any work. [api] model in the project's settings file, as
the wizard writes it, takes the option's place. testuary check without a
model reports it as a problem, with exit status 1, and still looks at the
project and its tools.
Configure a model¶
Create the configuration templates:
testuary init
This writes models.ini into your user directory and hints.json into the
project's directory, names each file it writes and keeps any file that already
exists. In models.ini, add a section for your model:
[ollama_qwen]
provider = ollama
model = qwen3-coder:30b
api_url = http://localhost:11434/v1
The section name is the model alias you pass with --model. provider =
names the kind of model service, ollama, openai or vllm. It also gives
the name of the key variable, here TESTUARY_OLLAMA_API_KEY. model = is the
model name the service knows, and api_url = is the address of the service's
OpenAI-compatible interface, which for a local Ollama ends in /v1. The model
must support tool calls. Without api_url, an ollama alias goes to Ollama
Cloud, the address [ollama] api_url gives,
which needs a key. For Ollama on your own computer, keep the api_url line.
A service that needs a key reads it from that variable, or from another
place models.ini fields names. Keep
models.ini private and out of version control. A local service without
keys, such as Ollama on your own computer, needs none. At a paid service,
every model request can cost money. The
settings reference lists every setting.
Check your setup¶
Before the first model request, run the quick check. It looks at the same inputs a run would use:
testuary check --model ollama_qwen --source-dir /path/to/project \
--python-executable /path/to/project-env/bin/python \
--source-files package/module.py --output-dir /path/to/new-results
On Windows, the interpreter of a virtual environment is
<project-env>\Scripts\python.exe, for example
--python-executable C:\path\to\project-env\Scripts\python.exe.
The check's report says what the quick check looks at. The report goes to standard error and starts with what it found and how far it looked. A run tries some required parts first, such as the API key, so the quick check reports no problem only for what it tried. For a Python project with every tool installed it looks like this (paths replaced by placeholders):
The quick check found no problem in what it tried. The required parts it left to a run are listed under Not tried yet.
A quick check runs nothing in your project and contacts no model.
Source directory: /path/to/project (on this computer)
Target: package/module.py
Model: qwen3-coder:30b via Ollama
Run directory: /path/to/new-results (a new run)
Phases: Starter phase, Coverage phase, Mutation phase, Finishing
Limits: starter phase up to 4 attempts per suite | coverage phase up to 1000 model requests, no token or time limit | mutation phase up to 1000 model requests, no token or time limit
Found:
Python: runtime CPython 3.12.10 (/path/to/project-env/bin/python)
Installed but not tried:
Python: pytest 8.1.1, coverage 7.6.8, cosmic-ray 8.4.1
run the tool check to try it
The tool check is testuary check --verify with the same options. It makes no model requests.
Not tried yet:
the API key: No API key is configured. A service without API keys works, and a service that needs one refuses the first model request.
If the service needs an API key, set TESTUARY_OLLAMA_API_KEY, or name the key's variable with api_key_env in the [ollama_qwen] section of models.ini.
Python: running generated tests, listing the mutation tool's operators, running the mutation tool
run the tool check to try it
Python: removing single tests from a test file, counting the tests and assertions of a test file
start a run, which tries it for the first time
Full setup report: the same command with --display detailed
Next step: testuary check --verify with the same options (tries the installed tools, no model requests).
The items are grouped by what they mean, as the
terminal guide lists. Each problem
comes with the fix the check reports, and the report always ends with one
next step. Add
--display detailed to see every item by its plain name with its key,
status and reason, or --json for the report as data on standard output.
The tool check, --verify, also starts each tool once and tries your
project in a copy as a run does, without running a test. It contacts no
model:
testuary check --verify --model ollama_qwen --source-dir /path/to/project \
--python-executable /path/to/project-env/bin/python \
--source-files package/module.py --output-dir /path/to/new-results
For Python it imports the test tools with the selected interpreter in a copy of
the project, collects the project's tests there without running one, and writes
nothing into the project or its environment. For Java it runs the selected
java and javac, then, in the copy, resolves the build, compiles and finds
the tests, and starts the measurement tools a run needs. Like a run, it
downloads what the build needs and, unless LANGADAPTER_JAVA_TOOLS_OFFLINE is
set, the JUnit launcher, JaCoCo and PIT, within setup_timeout_seconds. A
download that fails, or that the variable turns off, is reported as a problem
with the step to take. The first run tries a step that the check had no time
left for, and the build of a project with its own build wrapper, which may
download its build tool. Running the tests and their measurements is a run's
work.
The tool check lists such parts under "not tried yet" and still ends with exit
status 0 when nothing else is open. Where the project's own runs build
offline, with Maven's -o or --offline in MAVEN_ARGS or in
.mvn/maven.config, the check builds offline too and reports a needed download
or an offline failure it does not know as a problem, since the first run would
meet it too.
The check's exit statuses:
- Status 0: The check found no problem. Parts that only a real run can try are listed under "not tried yet".
- Status 1: The check found a problem, such as a missing tool or no selected model.
- Status 2: A setting was refused or a choice is ambiguous.
- Status 130: The check was stopped with Ctrl+C. No setup was changed.
Results of an earlier run that a run would continue or reuse are confirmed only by that run. The check lists them under "not tried yet", and they do not change the exit status.
Run with small limits¶
For a first run, keep the model's work small. Put these lines into the
project's settings file, .testuary/project.ini in the project's directory,
which the wizard writes or you create:
[starter]
suite_count = 3
attempts_per_suite = 2
[coverage]
max_requests_per_phase = 20
max_output_tokens_per_phase = 200000
max_wall_clock_seconds_per_phase = 1800
[mutation]
max_requests_per_phase = 20
max_output_tokens_per_phase = 200000
max_wall_clock_seconds_per_phase = 1800
Then run the target in the project's directory, where the run finds that file:
cd /path/to/project
testuary generate --model ollama_qwen \
--python-executable /path/to/project-env/bin/python \
--source-files package/module.py
A run takes each target through four phases. In the starter phase the model writes several independent test suites, the starter suites, and Testuary picks the best result from them. The coverage phase then lets the model improve that suite so that it covers more lines and branches. The mutation phase does the same for the mutation score. A mutant is a copy of the code with one small change, and a test that fails on a mutant kills it. The mutation score is the percentage of mutants the tests kill. Each loop stops at its goal, at its limits, or after too many measurements without improvement, as the settings reference sets them for each loop. At the phase end of each of these three phases, in its consolidation step, Testuary combines the suites, removes tests that add nothing and keeps the best suite. The fourth phase, Finishing, writes the results of the delivered suite: it takes the suite's coverage from the phase end that handed it on and its mutation result from the phase end of the mutation phase, so it judges the suite no second time. It measures on its own only what those did not, such as the mutation score when the mutation phase was skipped.
These lines allow 3 starter suites with up to 2 attempts per suite, so a starter suite that fails, for example a reply without tests, is generated once more. They allow 20 model requests each for the coverage phase and the mutation phase, and stop each loop after 200,000 reported output tokens or 30 minutes after its first request. That is at most 46 model requests per target, not counting requests sent again after temporary failures such as a lost connection, or with half the output limit after a starter request did not fit the model's context. Testuary does not know what these requests cost. The lines change no goal or measurement, but with small limits the results are not comparable with those of runs at the default limits. An earlier run continues only with the same limits. The settings reference lists every limit.
Java uses the same command with the project's JDK:
cd /path/to/java-project
testuary generate --model ollama_qwen --java-home /path/to/jdk \
--source-files src/main/java/example/Calculator.java
The project directory stays where it is. Put quotes around Windows paths that
contain spaces. A new run directory must be missing or empty. Without
--output-dir, the run directory is .testuary/results/<model alias> in the
project's directory. Inside the project, a run directory must lie in its
.testuary directory. To
continue or reuse earlier results, see results and logs.
Follow the run¶
In an interactive terminal the progress view stays pinned at the bottom. It
shows the whole run, the current target, its phase (Starter phase, Coverage
phase, Mutation phase or Finishing), the step within the phase and the
operations running now, with progress bars where totals are known. A Log row
names the run's plain-text log from the moment it opens. The log of finished
work scrolls above the pinned view. The terminal guide
explains each view and the --display choices.
Read the results¶
When the run ends, the run directory holds:
results/tests.csv, one row per target and repetition with its coverage and mutation results.results/<target>/r0/tests/, the delivered suite of each target, in a directory named after the target's path.r0is the first repetition. A target run more than once with--runshas a directory per repetition.logs/run.log, every log line of the run as plain text.
At the end the run prints a summary with the outcome of each target, its delivered suite, its scores and its model use. When a target is unfinished, the summary ends with the command that continues the run. Results and logs describes the directory, the exit status and how to continue a run.