Python API¶
A Python program runs Testuary through the public entry points below. The configuration guide shows them in use. They are Testuary's supported Python interface, and its other modules are internal.
Version¶
testuary.__version__ = '0.2.1'
module-attribute
¶
The installed Testuary version.
testuary.RESULTS_VERSION = '0.2'
module-attribute
¶
The version of the result files, recorded in run logs.
Settings¶
testuary.settings.resolve_settings(*, config_files=None, overrides=None, use_environment=True, base_dir=None, models=None, environment=None, home=None, project_settings=True, models_files=None, load_extensions=False)
¶
Resolve the settings from these layers, each overriding the one
before: the packaged defaults, the global or explicit files, the
project's settings file, the enabled environment and the explicit
fields. [main] extensions is the exception: each layer adds its
items (extension_items).
With load_extensions, for the start of a command that uses them, the
items are loaded after each layer's [main] is checked and before
its other sections are, so [project] may name a language an
extension adds and the settings it declares. An item that cannot load
is refused, naming where it is named. Without it, and always for an
empty list, nothing is loaded, and settings that name items keep a
[project] language or key no registered language has, named in
RunSettings.unchecked.
file_locations names the files: without explicit files the user
directory's optional config.ini, and the project.ini found from
base_dir upward. With project_settings false none is read and
base_dir is the project. An empty models mapping skips the
catalogue. models_files names the catalogue files read in order in
place of the user directory's models.ini, relative ones from
base_dir as for config_files. A later file's section replaces
the same section of an earlier one whole, where configuration files
merge key by key. An empty sequence reads none. An absent last file
reads as empty, and any other absent file is refused, as are a single
path and models_files beside models, and a file in the project
directory that sets api_url or api_key_env. Relative paths start from
the project directory.
The launch environment is always captured (environment, else the
process's). use_environment decides only whether the TESTUARY_
and LANGADAPTER_ variables are inputs. Disabled, they are left out
of it too, and platform variables such as PATH stay.
testuary.settings.RunSettings
dataclass
¶
Resolved settings. Direct construction reads no ambient configuration.
Sections override packaged defaults. models is a catalogue, not a
file name. environment is the captured launch environment of native
commands. Omitted, nothing is captured and native execution refuses to
run: pass environment=dict(os.environ) or use resolve_settings.
environment_captured says whether one was supplied. supplied
holds the (section, key) pairs a layer above the packaged defaults
sets, every key of sections when omitted. configuration_files
names the configuration and catalogue files resolve_settings read,
which project copies leave out. user_dir is the user directory whose
model catalogue and global configuration these settings were resolved
from, None when constructed directly.
results_dir
property
¶
The directory results in the project's Testuary directory. A run
writes below it unless [project] output_dir names another directory.
consumer_files
property
¶
The files Testuary reads as configuration (settings files, catalogues, prompt profile, hints file), never project inputs. Each is listed as read, as spelled on disk and resolved, so a copy leaves it out however it is spelled or linked.
configuration_layers()
¶
The captured layers in input order, read-only. Not a safe export.
export()
¶
Return non-secret effective configuration suitable for result
records: every section but those that configure no run
(UNRECORDED_SECTIONS).
testuary.settings.ConfigurationError
¶
Bases: ValueError
Configuration was refused before a command started.
testuary.settings.file_locations(base_dir=None, *, home=None, environment=None, testuary_dir='.testuary', search=True)
¶
Where runs started in base_dir (default: the current directory) find
their files, and where the setup and changes write them.
The user directory is home, else TESTUARY_HOME in environment,
else platform_user_dir(). The project search goes upward from
base_dir and stops below the system user's home directory
(Path.home()), which is never a project. Without search the
starting directory is the project. --config or
TESTUARY_CONFIG_FILE replaces the global configuration file for one
command.
testuary.settings.project_locations(base_dir=None, *, home=None, environment=None)
¶
file_locations with the Testuary directory's name a run without
command-line settings uses (its variable, else the global configuration,
else the default), reading nothing else: for other programs to find a
project's settings and results.
testuary.settings.FileLocations
dataclass
¶
Testuary's files for runs started in one directory: the user directory's global configuration and catalogue, and the project directory, the nearest upward holding a Testuary directory (else the starting directory), whose Testuary directory holds the project's settings and results.
testuary.settings.platform_user_dir()
¶
The platform's per-user configuration directory for Testuary, as
platformdirs names it: on Windows %APPDATA%\testuary, on macOS
~/Library/Application Support/testuary, elsewhere
$XDG_CONFIG_HOME/testuary or ~/.config/testuary.
testuary.settings.reference(*, languages=None)
¶
Every setting by section in the packaged config.ini's order, as
it and its comments state them, then in [project] those the installed
languages declare. A setting several languages declare names them
all, and the first declaration gives it its kind, choices and check. The
reference page and the settings views come from it. The model request
controls are described with the models.
testuary.settings.SectionDescription
dataclass
¶
A section of the configuration: its name, what it is for and its
settings in order. title names it in plain words.
title
property
¶
The section's name in plain words, such as "Starter phase".
testuary.settings.SettingDescription
dataclass
¶
One setting: section, name, purpose, kind (boolean, choice,
number, text, and for a language's project setting path and
executable), default (empty: unset), a choice's values, its check,
and every language declaring it (empty: Testuary's own).
label
property
¶
The setting's name in plain words, as a settings view titles it: a written label for the settings most runs change, else its name's words.
testuary.settings.setting_values(settings, *, languages=None)
¶
Each described setting with its value in settings and the layer
it comes from, the last that sets it. Values come from the safe view the
export gives, the sections it leaves out of records included, so no
credential appears.
testuary.settings.SettingValue
dataclass
¶
A described setting's value and its source: default, the
path of the file that sets it, environment or overrides.
sources names every layer that sets it, in order and in the same
terms, the last being source. It holds no value. active is False
for a setting the run does not use, inactive saying why.
testuary.settings.extension_items(settings)
¶
Each item of [main] extensions in the combined list's order, as
(item, source, directory): the layer that names it in the terms of
SettingValue.source (the path of a file, environment or
overrides) and the directory a file item is relative to, that of the
file that names it, else the project directory. Nothing is loaded, so
two items that name one file both appear. Settings that
dataclasses.replace made keep the rows of their resolution while
the list stays, and an item a replace adds counts as an override.
testuary.settings.update_file(path, section, values=None, *, remove=())
¶
Set values and remove remove in section of the settings
file path, keeping every other line, and write it only when the
whole file validates as the resolver's. Else ConfigurationError names
the problem and the file stays. A missing file is created.
testuary.settings.add_extension(item)
¶
Name item, a file or an installed module, in [main]
extensions of the user's config.ini when that file does not name it
yet, keeping its other items and lines, and tell whether the file
changed. A run with --config does not read that file. A file item is
relative to the user directory.
testuary.settings.remove_extension(item)
¶
Take item out of [main] extensions of the user's config.ini,
the key with its last item, and tell whether the file changed.
testuary.settings.check_project_settings_file(path)
¶
Refuse the settings file path when any of its sections, such as
[api] or a provider's, sets api_url, which decides where the
model's key goes.
The caller passes the files that are not the user's own: Testuary refuses the addresses of its project's settings file by this rule, and a program checks each other settings file it passes to its runs. A missing file passes.
Raises:
| Type | Description |
|---|---|
ConfigurationError
|
naming the file and each address it sets. |
ValueError
|
when the file is not valid INI. |
testuary.settings.check_project_catalogue_file(path)
¶
Refuse the model catalogue file path when any of its sections sets
api_url or api_key_env, which decide where the model's key goes.
The caller passes the files that are not the user's own: settings resolution checks each catalogue file in the project directory, and a program checks each other catalogue file it passes to its runs. A missing file passes.
Raises:
| Type | Description |
|---|---|
ConfigurationError
|
naming the file and each such value. |
ValueError
|
when the file is not valid INI. |
Setup¶
The questions testuary setup asks, answered through one function, and the
model catalogue and a project's settings, changed as the setup changes them.
testuary.setup.run_setup(ask, probe, say, base_dir, *, part=None, name='', announce_run=True, home=None)
¶
Run the named part, or all three, for runs started in base_dir:
models.ini and config.ini in the user directory (home when given),
the project's settings file in the project's directory. part is
"model", "machine" or "project" (PARTS).
ask(question) returns the answer to a Question as text, an
empty text for its default. probe tries the model and the tools the
answers name. say(text) shows one line to the person.
Run alone, a part always asks. In the walk of all three, a part already
set is passed with one line, and every other part offers a skip. The
result lists what was written and what a run still lacks.
announce_run names the run command at the end. testuary setup
switches it off and names the command after its check of the project.
testuary.setup.Question
dataclass
¶
One question: its stable key, the text shown, the default Enter
accepts, its choices, and the reason shown with it, if any. A closed
question takes one of its choices. An open one also takes another
answer, such as a path, that its check accepts: its choices are what
the setup found. A secret answer is read without echo and never shown.
testuary.setup.Ask = Callable[[Question], str]
module-attribute
¶
The question function run_setup takes: it returns the answer to a
Question as text, an empty text for the question's default.
testuary.setup.Say = Callable[[str], None]
module-attribute
¶
The function run_setup shows one line to the person with.
testuary.setup.terminal_ask(question)
¶
Ask on the terminal: the default in brackets, a secret without echo. Without terminal input the wizard stops instead of writing.
testuary.setup.PARTS = ('model', 'machine', 'project')
module-attribute
¶
The setup's parts, as run_setup's part names them.
testuary.setup.Probe
¶
Looks at the machine for defaults and checks. The demonstration
replaces it with answers from fixtures. cpu_count returns the number
of CPUs this process may use, and local_ollama whether an Ollama on
this computer lists its models at http://localhost:11434 within two
seconds.
cpu_count()
¶
The number of CPUs this process may use: those its affinity
allows (allowed_cpus), or the machine's where the platform has
no affinity, at most what a CPU quota leaves (_quota_cpus), 1
when none can be read.
environment(name)
¶
The environment variable name, empty when it is not set.
is_dir(path)
¶
Whether path is a directory.
is_file(path)
¶
Whether path is a file.
source_files(root, language)
¶
The language's source files below root, relative to it with
forward slashes, as the language's adapter tells them: files with its
file extensions outside the directories it ignores (ignored_dirs, and
dot directories), except the names it declares as no sources
(non_source_files), its test files (is_test_file) and empty
files, which hold nothing to test.
read_text(path)
¶
The text of the file path, or None when it cannot be read.
run(argv, timeout)
¶
The exit code and output of argv, or None when it does not
start or does not end within timeout seconds. It gets no input,
and output the locale cannot decode is kept with replacements.
which(command)
¶
The path of the program command on the search path, empty when it
is not found.
local_ollama()
¶
Whether an Ollama on this computer lists its models at
http://localhost:11434 within two seconds.
model_list(provider, address, key, *, timeout=15)
¶
The model names the service lists, a request that generates nothing, or None when it cannot be read. The key goes into the request header only.
restrict_to_user(path)
¶
Leave path readable by its owner only, where the system
allows it. POSIX files are written with permissions 0600. On Windows the
file's inherited access is replaced by its owner's.
testuary.setup.Provider
dataclass
¶
A kind of model service the wizard offers.
testuary.setup.PROVIDERS = (Provider('ollama-local', 'Ollama on this machine', 'ollama', 'http://localhost:11434/v1', 'none', starters=1, starters_reason='a local Ollama answers one request at a time'), Provider('ollama-cloud', 'Ollama Cloud', 'ollama', 'https://ollama.com/v1', 'required', starters=4, starters_reason='Ollama Cloud refuses more than a few concurrent requests'), Provider('openai', 'OpenAI', 'openai', 'https://api.openai.com/v1', 'required', starters=4, starters_reason="a cautious start, raise it within your account's rate limit"), Provider('deepseek', 'DeepSeek API', 'deepseek', 'https://api.deepseek.com/v1', 'required', starters=4, starters_reason='DeepSeek states a limit of 25 parallel requests'), Provider('vllm', 'vLLM server', 'vllm', 'http://localhost:8000/v1', 'optional', custom_address=True, starters=10, starters_reason='a vLLM server sustains parallel requests'), Provider('openai-compatible', 'another OpenAI-compatible API', '', 'http://localhost:8000/v1', 'optional', custom_address=True, starters=4, starters_reason="a cautious start, raise it within the service's limit"))
module-attribute
¶
The kinds of model service the setup offers, in the order it offers them.
testuary.setup.SetupResult
dataclass
¶
What the wizard wrote and what a run still lacks. project is the
directory of the project it wrote, where run runs it.
testuary.setup.SetupError
¶
Bases: RuntimeError
The wizard stopped, or a catalogue change was refused. The message
says why and what to do. For a refused change, problems names each
refused value as (field, problem), the field being the argument of
set_model or set_project it concerns, or the project key.
testuary.setup.models(base_dir=None, *, home=None, probe=None, models_files=None)
¶
The aliases of the model catalogue of runs in base_dir (the
current directory when omitted), the user directory's (home when given),
with where each key comes from and whether a run can use the alias, by
the rule the setup offers aliases by. probe reads the environment.
models_files names the catalogue files read in order in place of
the user directory's, joined as resolve_settings(models_files=...)
joins them.
testuary.setup.ModelEntry
dataclass
¶
One alias of the model catalogue as the setup sees it. address is
the address the catalogue names for the alias or its provider, never
with credentials, and empty for the provider's configured endpoint.
key_source says where its key comes from: models.ini,
environment, not needed for a local address, or missing.
key_variable names the variable the key is read from, or that a run
would read it from when it is missing. usable says whether a run can
use the alias. file is the catalogue file the alias's section comes
from. It holds no key.
testuary.setup.usable_aliases(base_dir=None, *, home=None, probe=None)
¶
The aliases a run can use: the setup offers these.
testuary.setup.set_model(alias, *, provider, model, address='', key='', key_variable='', base_dir=None, home=None, probe=None, models_files=None)
¶
Create or replace alias in the model catalogue, as the setup's
model part writes it: the alias's provider and model, an address (empty:
the provider's configured one, or the address its section already
holds), a key stored in the provider's section, or the variable that
holds the key. The file keeps its comments and other sections and is
restricted to its owner while it holds a key. SetupError names a value
the setup would refuse, and an alias that resolved before and would no
longer resolve as a run resolves it.
With models_files, the catalogue files read in order (models),
the change goes to the last file, which it creates when absent. A
section only an earlier file defines is written into it whole from the
earlier section's keys. No key is written: a key is refused, and so is
a last file that holds one. A section whose key an earlier file holds
needs key_variable, and no address that holds a credential is
written.
testuary.setup.remove_model(alias, *, base_dir=None, home=None, probe=None, models_files=None)
¶
Remove alias's section from the model catalogue, keeping every
other line. Its provider's section stays: other aliases can use it.
SetupError refuses a removal that leaves an alias that resolved before
unresolved. With models_files (set_model), only a section of the
last file is removed, and an earlier file's section of the alias
applies again.
testuary.setup.project(base_dir=None, *, home=None)
¶
The settings of the project of runs in base_dir (the current
directory when omitted), found from there upward as a run finds it, with
the user directory home when given.
testuary.setup.ProjectEntry
dataclass
¶
A project's settings as the setup writes them: its directory
(project_dir), its settings file (settings_file), and the file's
[project] values with the model alias of its [api] section as
model (fields).
testuary.setup.languages()
¶
The languages the setup offers: the installed ones, as LangAdapter lists them (in name order).
testuary.setup.set_project(values, *, remove=(), base_dir=None, home=None, probe=None)
¶
Set values and remove the keys remove in the settings file of
the project of runs in base_dir, as the setup's project part writes
them: model as [api] model, every other key in [project].
Without a project there, base_dir becomes one. A key a run would
refuse, a value of more than one line, a language that is not
installed and a value the setup's project part refuses (an import
directory, interpreter, JDK or build tool that does not exist, checked
with probe) are refused, each named, and nothing is written. The
file keeps its comments and other lines. home is the user directory,
whose configuration names the Testuary directory.
Runs¶
testuary.orchestrator.RunRequest
dataclass
¶
Targets and execution policy for run_pipeline(..., settings=...).
Source paths are individual strings, never comma-delimited CLI text. The model, the project, its runtime and the limits belong to RunSettings.
runs runs each target's four phases that many times, as repetitions
r0 to r(runs-1), each from a fresh copy of the project and in its own
directory. run runs only repetition K (rK) instead, the same one
runs makes when it is above K, so a missing repetition can be added alone.
targets runs only some of source_files, by position, as
--targets does, such as "1-5". reuse names the phases whose
saved results the run takes, as --reuse does, and resume
continues an interrupted run in the same run directory, as --resume
does. stop_after_phase names the phase after which each target
stops, as --stop-after-phase does. languages lists the languages
a target may be in, whose tools are checked before the run, as
--languages does.
testuary.orchestrator.run_pipeline(args, selected_languages=None, *, settings=None, event_sink=None, events=None)
¶
Run the targets of args, a RunRequest, under settings, a
RunSettings from resolve_settings, in isolated run state.
Pass settings for programmatic use, without configuration flags on
args. Command-line flags such as target selection and resume are separate.
Overlapping calls in one process are refused, so use separate processes.
event_sink is an optional read-only display observer. It cannot control
requests, retries or results.
events names an event file, which Testuary writes as testuary
--events FILE writes its file, without a command that resumes the run.
A relative path starts from the current directory, and an existing file is
replaced. A call refused before its run may start writes no file.
Before the run starts, it loads the extensions settings name and
captures their hints.
testuary.presentation.view.phase_label(phase)
¶
A phase's name as the console shows it, such as Coverage phase for
coverage: the label other views of a run, like a web page, use.
testuary.results.recorded_source_sha256(output_dir, target)
¶
The SHA-256, in hex, of a target file's raw bytes that the run in
output_dir recorded when the target's phases started, or None when
its records hold none or its phases recorded different ones. target
is the file's path relative to the run's source directory, as the
run's completion names it. A phase that recorded none, such as one
the run reused, does not count. A caller compares the result with
hashlib.sha256 of the file's bytes read in binary. Only the
records are read.
Observing a run¶
What run_pipeline's event_sink receives, the presenter that draws it
and the observations of LangAdapter's mutation and timing progress, as
the terminal guide shows them.
testuary.presentation.DisplayEvent
dataclass
¶
One observation of an operation, emitted after the producer recorded its facts. It cannot control execution, retries, reduction or what is saved.
The identity tells concurrent work apart and shows only at the debug
level. elapsed_s is the run's clock, completed and total count
actual work, not a request limit, and message is a short diagnostic,
never a prompt or a model reply. level places it in the layered view
(run, target, phase, step or operation), or None to infer it from the
identity. label is its name in plain words. log_path names the
run's detailed logs on a run-level observation. failure, on a failed
one, tells an execution failure ("execution") from a result of the method
("outcome", such as a target without a suite that meets the
minimums), or None.
state is running, waiting, retrying, succeeded, failed, cancelled,
refused or limit_reached. limit_reached is work that stopped at a limit
its settings set (a loop's requests, tokens or time), an ordinary end
and no failure.
from_dict(data)
classmethod
¶
The observation a mapping of its fields describes
(record_from_dict).
testuary.presentation.Metric
dataclass
¶
A named value in the producer's units, with its measurement status.
Percentage values use 0 to 100, not a fraction. A progress value is never described as a final score. None is unavailable, not an observed zero. goal is the configured target in the same unit (for example a coverage goal). It is shown beside the value, never as progress toward completion.
testuary.presentation.Usage
dataclass
¶
Already-accounted usage. Omitted provider observations remain unknown.
logical_request_limit is the configured ceiling for the phase, a limit and not a prediction of how many requests the phase will need.
testuary.presentation.TerminalPresenter
¶
A callable observer that draws view, a RunView, with Rich. It
installs no hooks and redirects no global stream.
As a context manager it restores the terminal on success, failure and Ctrl+C, and no display error hides an execution error. Output goes to stderr, so stdout stays machine-readable. A redirected or dumb terminal gets plain output whatever the display mode. Debug is the detailed view with internal identities.
provenance
property
¶
The provenance of the newest observation, observed before any.
write_diagnostic(write)
¶
Run one diagnostic write unchanged, with the compact view paused
around it and resumed on a fresh line. Writes and observations are
serialized, and a nested write resumes the view only after the outer
one. A display failure falls back to plain output, while write runs
once and its exceptions propagate. Writes outside this call are not
covered.
plan(plan)
¶
Show the run's plan (RunPlan) once, in one line, before the
run's work. A plan of one file and one run leaves the compact view
one bar, the Overall row's (RunView.snapshot).
redact_with(function)
¶
Also remove the run's own credentials with function, the
resolved settings' redaction, from everything shown from now on. The
patterns alone miss a key that carries no recognizable name.
clean(value, *, preserve_whitespace=False)
¶
value as it may be shown: the run's own credentials removed,
then terminal controls and recognizable credentials.
frame()
¶
What the compact view draws now, also for looking at it in a test.
testuary.presentation.measurement.mutation_progress_event(progress, *, run_id, target, elapsed_s, phase='mutation', operation=None, provenance='observed')
¶
One progress record as a running operation observation.
elapsed_s is the run clock the producer uses for its other
observations. The record's own elapsed_s counts from the start of the
mutation run and is not substituted for it. An unknown total stays
unknown.
testuary.presentation.measurement.warmed_progress_event(progress, *, run_id, target, phase='suite timing', provenance='observed')
¶
One progress record as a display observation. An entered iteration is not a sample.
testuary.presentation.measurement.warmed_result_event(result, *, run_id, target, phase='suite timing', provenance='observed')
¶
The final result: outcome, validity, counts and, only when usable, summaries.
Event files¶
The event file --events writes, and its writer for a program whose run
a page shows as a run's, such as a web interface extension's job kind.
The terminal guide
describes its records. read_completion reads a run's
end-of-run summary, the
RunCompletion in results/completion.json.
testuary.presentation.EventFile
¶
What the display shows, as JSON lines, for programs that show a run.
Testuary's --events and run_pipeline's events write through
it, and so does a program whose run a page shows as a run's, such as a
web interface extension's job kind.
A header line holds the format version (schema_version,
EVENT_FILE_SCHEMA) and the Testuary version. Each further line holds
one record, in the order the display shows it: {"event": ...} with
the fields of a
testuary.presentation.DisplayEvent, {"readiness": ...} with the
setup report before the run (RunReadiness.as_dict()), {"plan": ...}
with what the run will do (a RunPlan), and
{"completion": ...} with the end of the run (a RunCompletion).
Every text has its credentials removed as the display removes them before
it prints, with the same secrets. A failed write ends the file, not the
run or its display. Events hold finite numbers only, so the JSON needs no
NaN.
testuary.presentation.EVENT_FILE_SCHEMA = 3
module-attribute
¶
The event file's format version, in its header line. A change of the format raises it. The completion record's own version is separate.
testuary.presentation.RunPlan
dataclass
¶
The run's files (relative to the project's source directory), its runs (r0, r1 and so on), its first three phases in order and Finishing, the fourth.
testuary.presentation.events.PlannedPhase
dataclass
¶
One of the run's first three phases: its name and label, its state
(planned, off or reused, as a PlannedStep's), the steps it goes
through and its limits.
The steps form three groups: a start that runs once (steps[:cycle]),
a cycle that repeats (steps[cycle:end], empty in the starter phase)
and the phase end that runs once after it (steps[end:], the
consolidation of every phase). end None is the number of steps: no
phase-end group.
testuary.presentation.events.PlannedStep
dataclass
¶
One step of a phase, or Finishing: its name, its label
in plain words and its state: planned, off when the run stops before
it (--stop-after-phase), or reused when the run takes its results
from an earlier one (--reuse).
testuary.presentation.events.PlannedLimit
dataclass
¶
A configured limit or goal of a phase, in its unit. None is no limit.
testuary.presentation.RunCompletion
dataclass
¶
The whole invocation: its files, where results and logs are, how to continue.
resume_command is the producer's exact command for continuing this
run, or None when there is nothing to continue or no such command.
testuary.presentation.read_completion(path)
¶
The end-of-run summary at path, a run's
results/completion.json, as the pair of its RunCompletion and
its provenance as the file records it, synthetic for a
demonstration's values. Only the file is read.
Raises ValueError when the file is not JSON or does not hold this Testuary's summary format version and a completion, ValueError or TypeError when the completion's fields do not match this version's, and OSError when the file cannot be read.
Checks¶
testuary.orchestrator.check(request, *, settings, verify=False)
¶
Check the request's setup without generating tests, and return a RunReadiness without printing.
Returns the setup report as a RunReadiness and prints nothing. With
verify it is the tool check, which starts the tools and tries the
project in a copy, and for a Java project downloads what a run would
when it is missing. Without verify it downloads nothing. An item the
check cannot try is reported as not tried: only a run tries it.
testuary.readiness.RunReadiness
dataclass
¶
The setup report of testuary check, or of a run before its first
model request: its scope, status and exit code, the selections (project,
execution, source files, phases, model, run directory and whether it starts
fresh, resumes or reuses), the
limits and the items. A configured key is reported as configured, not as
accepted by the provider. The report holds the redacted facts its
producer observed and promises nothing about later requests, cost or
provider capacity.
as_dict()
¶
The report as detached JSON data, without settings values, environment or keys.
testuary.readiness.ReadinessItem
dataclass
¶
One item of the report: an operation, tool or setting by key,
whether the run needs it, its status, what was selected and its version,
the reason, the next action and a diagnostic file (diagnostic_path).
It speaks for that operation only, not for the language as a whole. The
status is one of "verified", "missing_setup", "unsupported", "ambiguous",
"unverified", "failed" and "refused" (ReadinessStatus). tried_by
names what first tries an item not tried yet: "verification"
(testuary check --verify) or "run" (only a real run), otherwise None.
It matters only while the item is unverified.
testuary.readiness.ReadinessLimit
dataclass
¶
A limit as configured for phase: a value in unit
(configured), unlimited or unknown. Only a configured limit
has a value, so None is never read as zero.
testuary.presentation.present_readiness(report, *, presenter, before_run=False, playback=False, steps=TERMINAL_STEPS)
¶
Write one report through the presenter's diagnostic boundary.
Compact and plain output summarize what was found, what is installed but
not tried, what was not tried yet and each problem with its action,
ending with one next step. The detailed view lists every item with its
raw names. Statuses, selections and actions are the producer's: nothing is
checked, installed or inferred here. before_run marks the summary
shown before a run's first model request, and playback a
demonstration's playback, which names no API key and ends by saying
that it costs nothing. steps words the reader's next actions: the
terminal's commands unless another interface passes its own.
Models for your own program¶
A model you configured for Testuary, used from your own program: a LangChain chat model for an alias of your model catalogue or a model a provider serves, with Testuary's retries and request records. The models guide shows a call.
testuary.models.catalogue_chat_model(model, *, base_dir, records_dir, label, request_values=None, config_files=None, project_settings=True, models_files=None, provider=None)
¶
A LangChain chat model for your own program's model calls, for which
model and provider resolve as [api] model and [api]
provider do in a Testuary run in base_dir: an alias of your model
catalogue with its own provider, else, with provider, a model of
that provider with its address, protocol, request settings and key, from
the run's settings files and catalogue, read only by path. config_files,
project_settings and models_files choose the settings files and
the catalogue as they do for testuary.settings.resolve_settings: an
empty list reads no configuration file, a list of files reads them in
place of the global config.ini, project_settings false reads no
project.ini, and models_files names the catalogue files read in
place of the user directory's models.ini, a later file's section
replacing the same section of an earlier one. A program whose Testuary
runs name their settings files and catalogue passes the same choice, so
that its models read the same files.
request_values sets request values above the configured [api]
values, and no phase section applies. Its keys are the request values
of models.ini that REQUEST_VALUES holds: temperature, top_p,
seed, max_output_tokens, request_timeout_seconds,
reasoning, reasoning_effort, reasoning_budget_tokens,
preserve_reasoning_content, reasoning_field_names and
request_options. The catalogue's output limits of the model still
bound the output. The model's tool binding goes through the same retries
and records.
Each request is retried under Testuary's bundled retry settings: the
packaged values of [coverage]'s max_consecutive_transient_failures,
max_transient_failures_per_phase, transient_first_wait_seconds,
max_transient_wait_seconds, max_transient_hint_seconds and
count_retries, which the settings reference shows. A
caller cannot change them. The ceiling of
max_transient_failures_per_phase counts over the model's life: once
it is reached, the first temporary failure of a later request raises
model_requests.TransportExhausted, and a new model starts over.
Every request sent is appended to
request_records.jsonl in records_dir: a request record with
the key label. Each created model appends its settings without
credentials to model_settings.jsonl there, with label. Both files
only grow, so a resumed program keeps its earlier records.
Raises ConfigurationError before any request for a name the
catalogue does not define without provider, or a provider that
neither Testuary configures nor a provider section defines, naming the
catalogue files, an absent file of models_files, an
ollama-native model, a credential that cannot be resolved, an empty
label, or a request_values key that names the identity, a
credential or no request value. A request raises
model_requests.TransportExhausted when a ceiling is reached,
model_requests.TerminalRequestError for a terminal refusal and
model_requests.ContextOverflow when the prompt does not fit the
model's context. Their messages name the provider's error type and
text without the key, and they carry no cause, since a provider's error
can hold the request and its key. Any other error, such as a request
record that cannot be written, is raised without the errors chained to
it. The records hold no key either.
testuary.models.REQUEST_VALUES = frozenset({'temperature', 'top_p', 'seed', 'max_output_tokens', 'request_timeout_seconds', 'reasoning', 'reasoning_effort', 'reasoning_budget_tokens', 'preserve_reasoning_content', 'reasoning_field_names', 'request_options'})
module-attribute
¶
The request values catalogue_chat_model accepts in request_values:
the request fields of a model catalogue entry without protocol, which
belongs to the alias's identity. The refusal of any other key lists them.
testuary.model_requests.TransportExhausted
¶
Bases: Exception
A retry ceiling was reached. The last transport failure is the cause.
attempt is the attempt of the model request that reached it.
testuary.model_requests.TerminalRequestError
¶
Bases: Exception
A request failed in a way that retrying cannot fix. The provider's error is the cause.
testuary.model_requests.ContextOverflow
¶
Bases: Exception
The provider refused a request because the prompt does not fit the model's context. It is not retried, since the caller owns its prompt and its output limit. Its message names the provider's error type and text without the model's key, and it carries no cause, since a provider's error can hold the request and its key.
Demonstrations¶
The offline demonstration, played as testuary --demo plays it.
testuary.demo.play(language='python', *, output_dir, data=None, speed='auto', duration=None)
¶
Play the offline demonstration as testuary --demo does and return
its exit status (0 when it completed, 130 when Ctrl+C stopped it).
language selects a language's demonstration, or data names
demonstration data in its place (as --demo-data,
without language). output_dir must be new or empty. speed is
auto, instant or a factor of the configured length's speed, such
as 2 for half as long (--demo-speed), and duration the paced
playback's length in seconds (--demo-playback-seconds). A demonstration must
start before any part of a run is loaded, so each call plays in a Python
process of its own: a script can play several, one after the other, and
Ctrl+C reaches the playback, which ends with its own cleanup.
Web interface¶
The local browser interface, started as testuary-web starts it.
testuary.web.serve(base_dir=None, *, port=None, access=None, open_browser=True)
¶
Start the web interface for base_dir as testuary-web does and
return its exit status (0 when Ctrl+C ended it, 2 when base_dir is no
directory, port cannot be used, access is none of the values
below or the installed Testuary does not answer in time).
base_dir is the directory Testuary works in (default: the current
directory). port is the port on 127.0.0.1 (default: [web] port of
Testuary's settings, 8765, and 0 takes a free one), access who may
use the interface (default: [web] access, open): open, every
program on this computer that reaches the port, key, only a page
opened with the printed address, whose access key is new at each start,
or stable_key, the same with one key kept in Testuary's user
directory. open_browser opens the printed address in the browser.
In an SSH login (SSH_CONNECTION set) it opens none and prints the ssh -L
line for the user's own computer instead. The call
blocks until Ctrl+C, a SIGTERM or Ctrl+Break ends it. A job that is
still running then stops first, as Ctrl+C stops a run, and keeps the
work that finished. Call it from the main thread, which receives
Ctrl+C.
The web interface's extension point, the top-level name testuary_web of
a module that [main] extensions names
(Extend the web interface).
testuary.web.WebExtension
dataclass
¶
What an extension module adds to the web interface: the extension
API version it was written for (api), its id, and optionally the
product name, tabs, Testuary's tabs' titles by id
(tab_titles), positions for Testuary's tabs other than the Overview
by id, in the words of a tab's position (tab_positions), the ids of
Testuary's tabs other than the Overview it hides (hidden_tabs), the
address the header's Guide link opens (guide): an http or https
address or the name of an HTML file next to its module, None to hide the
link, or the empty string for Testuary's guide, dir_types,
settings_groups, job_kinds and read-only routes.
testuary.web.WebTab
dataclass
¶
A tab an extension adds: its id (the address #/title, its view script (a file next to the extension's module), its
position: "first", "last", "before:replaces, the Overview excepted. A
position may name an added or a moved tab, and a position naming a tab
no one has places the tab last. A replaced tab leaves the header, and
its address and Testuary's links to it open the replacing tab.
testuary.web.DirType
dataclass
¶
A kind of directory an extension's result view opens: its id and
title, detect(named_dir) whether a directory is one, summary(named_dir)
the JSON data its view shows, read from the files the extension wrote,
and the view script.
testuary.web.SettingsGroup
dataclass
¶
Settings an extension adds to the Settings page: its id and
title, reference(project_dir) its sections as
testuary.settings.SectionDescription, values(project_dir) each
setting's value by (section, name), and update(project_dir, section,
values, remove), its validated update, which raises ValueError
with the reason to refuse a change. project_dir is the listed project the
page shows, or None for the global settings.
testuary_settings_files(project_dir), optional, names the Testuary
settings files the extension's runs in the listed project project_dir
read, in the order resolve_settings(config_files=...) reads them
and with the extension's own file last, also while it does not exist,
or None when they name none. The runs read them in place of both the
global configuration and the project's own settings file. A relative
path starts from the project directory. The Settings page shows the
settings as those runs read them and changes the last file, when it
lies in the project directory.
testuary_models_files(project_dir), optional, names the model
catalogue files those runs read, in the order
resolve_settings(models_files=...) reads them and with the
extension's own file last, also while it does not exist, or None when
they name none. The Models page shows the models as those runs read
them and changes the last file, when it lies in the project directory
or in extension_user_dir, the extension's own user directory.
end_models_import(project_dir), optional, makes those runs read the
last file alone. The Models page calls it after it copied every model
of the earlier files but one into the last file, to remove a model only
an earlier file defines, and the function changes the extension's
settings as its last step.
After the Settings page or the Models page writes a file of such a list, it calls the list's function again, and a function that raises refuses the saved file: the file returns to what it held before, and the error's message is the reason the page shows.
testuary.web.JobKind
dataclass
¶
A job an extension starts: its id and title, the module
the job runs as python -m does, command(form) the module's
arguments for the form the page sends (ValueError with the reason
refuses it), resume(arguments), the arguments Continue runs, or
None for a kind without Continue, and nested_runs(arguments), the
Testuary runs a job started, each a NestedRun its page folds into
its phase, or None for a kind whose jobs start none. Both receive the
job's arguments without the module. The job adds --events PATH,
where the module writes Testuary's event file through
testuary.presentation.EventFile.
testuary.web.NestedRun
dataclass
¶
A Testuary run that a job of an extension's kind started: the id of
the phase of the job's plan that started it, the run's event file or
run directory (path, as the page's nestedRun names a run) and the
title its folded part on the job's page shows after "Testuary
run", or None for none.
testuary.web.WebRoute
dataclass
¶
A read-only route of an extension: it answers GET at
/api/ext/<extension id>/<path> with the JSON read(named_dir, query)
returns. named_dir is the directory the query's dir names, resolved
as the Results view resolves one, or None without dir. query
holds the query's first values by name. A ValueError refuses the
request with its message, and any other error answers with the error.
testuary.web.WEB_EXTENSION_API = 1
module-attribute
¶
The extension API version this Testuary supports. Version 1 covers the
event file's schema version 3 (testuary.presentation.EVENT_FILE_SCHEMA).
testuary.web.TOOL_CHECK_LABEL = 'Tool check: also start each tool once and try the project in a copy'
module-attribute
¶
The label of the tool check option on Testuary's Check page, which the
check report quotes while that page is Testuary's own. /api/state carries
it as tool_check_label, for a replacing Check view that offers
Testuary's tool check.
Hints of extensions¶
The hints of the extensions the settings name, captured and validated as a run captures them, and the guidance one extension's hints or a hints file add to a phase's model request.
testuary.hints.extension_hints(settings)
¶
The hints of the extensions settings name, in the combined list's
order, captured now. Each mapping is written once as JSON and validated
as a hints file. An extension that cannot load, a mapping JSON cannot
hold and invalid hints raise testuary.settings.ConfigurationError
naming the item and where it is named.
testuary.hints.request_text(hints, language, phase, *, hint_profile='')
¶
The guidance hints adds to phase's model request for
language, empty when it adds none. hints is an extension's
hints from extension_hints or a hints file's parsed content.
phase is initial, coverage or mutation, and hint_profile selects
the hints file's profile cells as [project] hint_profile does.
testuary.hints.ExtensionHints
dataclass
¶
An extension's hints as one command captured them: the item that
names the extension, the extension's resolved file and its
testuary_hints as JSON bytes.
testuary.hints.PHASES = ('initial', 'coverage', 'mutation')
module-attribute
¶
The phases whose prompts read hints and a prompt profile: the starter phase and the two loops.
Generation records¶
The readers of a run's generation records, which Generation records describes.
testuary.generation_evidence.read_suite_slots(manifest, suites_root=None)
¶
The manifest's suite_slots section, checked, as a detached copy.
With suites_root the accepted references are verified too. A missing
or unknown section raises: a disposition is never inferred from counts
or original.json.
testuary.generation_evidence.read_cleaned_source(reference, suites_root)
¶
The cleaned source a reference names, checked against its recorded hash, so a file overwritten since raises.
testuary.generation_evidence.read_ancestry(record, source=None)
¶
The record's construction graph (generation_ancestry), checked
and detached. It must belong to the record's selected source, and
source when given. Anything unreadable or contradictory raises.
testuary.generation_evidence.GenerationEvidenceError
¶
Bases: ValueError
Generation evidence is missing, contradictory or cannot be resolved.