Skip to content

Python API

A Python program runs Testuary through the public entry points below. The configuration guide shows them in use. They are Testuary's supported Python interface, and its other modules are internal.

Version

testuary.__version__ = '0.2.1' module-attribute

The installed Testuary version.

testuary.RESULTS_VERSION = '0.2' module-attribute

The version of the result files, recorded in run logs.

Settings

testuary.settings.resolve_settings(*, config_files=None, overrides=None, use_environment=True, base_dir=None, models=None, environment=None, home=None, project_settings=True, models_files=None, load_extensions=False)

Resolve the settings from these layers, each overriding the one before: the packaged defaults, the global or explicit files, the project's settings file, the enabled environment and the explicit fields. [main] extensions is the exception: each layer adds its items (extension_items).

With load_extensions, for the start of a command that uses them, the items are loaded after each layer's [main] is checked and before its other sections are, so [project] may name a language an extension adds and the settings it declares. An item that cannot load is refused, naming where it is named. Without it, and always for an empty list, nothing is loaded, and settings that name items keep a [project] language or key no registered language has, named in RunSettings.unchecked.

file_locations names the files: without explicit files the user directory's optional config.ini, and the project.ini found from base_dir upward. With project_settings false none is read and base_dir is the project. An empty models mapping skips the catalogue. models_files names the catalogue files read in order in place of the user directory's models.ini, relative ones from base_dir as for config_files. A later file's section replaces the same section of an earlier one whole, where configuration files merge key by key. An empty sequence reads none. An absent last file reads as empty, and any other absent file is refused, as are a single path and models_files beside models, and a file in the project directory that sets api_url or api_key_env. Relative paths start from the project directory.

The launch environment is always captured (environment, else the process's). use_environment decides only whether the TESTUARY_ and LANGADAPTER_ variables are inputs. Disabled, they are left out of it too, and platform variables such as PATH stay.

testuary.settings.RunSettings dataclass

Resolved settings. Direct construction reads no ambient configuration.

Sections override packaged defaults. models is a catalogue, not a file name. environment is the captured launch environment of native commands. Omitted, nothing is captured and native execution refuses to run: pass environment=dict(os.environ) or use resolve_settings. environment_captured says whether one was supplied. supplied holds the (section, key) pairs a layer above the packaged defaults sets, every key of sections when omitted. configuration_files names the configuration and catalogue files resolve_settings read, which project copies leave out. user_dir is the user directory whose model catalogue and global configuration these settings were resolved from, None when constructed directly.

results_dir property

The directory results in the project's Testuary directory. A run writes below it unless [project] output_dir names another directory.

consumer_files property

The files Testuary reads as configuration (settings files, catalogues, prompt profile, hints file), never project inputs. Each is listed as read, as spelled on disk and resolved, so a copy leaves it out however it is spelled or linked.

configuration_layers()

The captured layers in input order, read-only. Not a safe export.

export()

Return non-secret effective configuration suitable for result records: every section but those that configure no run (UNRECORDED_SECTIONS).

testuary.settings.ConfigurationError

Bases: ValueError

Configuration was refused before a command started.

testuary.settings.file_locations(base_dir=None, *, home=None, environment=None, testuary_dir='.testuary', search=True)

Where runs started in base_dir (default: the current directory) find their files, and where the setup and changes write them.

The user directory is home, else TESTUARY_HOME in environment, else platform_user_dir(). The project search goes upward from base_dir and stops below the system user's home directory (Path.home()), which is never a project. Without search the starting directory is the project. --config or TESTUARY_CONFIG_FILE replaces the global configuration file for one command.

testuary.settings.project_locations(base_dir=None, *, home=None, environment=None)

file_locations with the Testuary directory's name a run without command-line settings uses (its variable, else the global configuration, else the default), reading nothing else: for other programs to find a project's settings and results.

testuary.settings.FileLocations dataclass

Testuary's files for runs started in one directory: the user directory's global configuration and catalogue, and the project directory, the nearest upward holding a Testuary directory (else the starting directory), whose Testuary directory holds the project's settings and results.

testuary.settings.platform_user_dir()

The platform's per-user configuration directory for Testuary, as platformdirs names it: on Windows %APPDATA%\testuary, on macOS ~/Library/Application Support/testuary, elsewhere $XDG_CONFIG_HOME/testuary or ~/.config/testuary.

testuary.settings.reference(*, languages=None)

Every setting by section in the packaged config.ini's order, as it and its comments state them, then in [project] those the installed languages declare. A setting several languages declare names them all, and the first declaration gives it its kind, choices and check. The reference page and the settings views come from it. The model request controls are described with the models.

testuary.settings.SectionDescription dataclass

A section of the configuration: its name, what it is for and its settings in order. title names it in plain words.

title property

The section's name in plain words, such as "Starter phase".

testuary.settings.SettingDescription dataclass

One setting: section, name, purpose, kind (boolean, choice, number, text, and for a language's project setting path and executable), default (empty: unset), a choice's values, its check, and every language declaring it (empty: Testuary's own).

label property

The setting's name in plain words, as a settings view titles it: a written label for the settings most runs change, else its name's words.

testuary.settings.setting_values(settings, *, languages=None)

Each described setting with its value in settings and the layer it comes from, the last that sets it. Values come from the safe view the export gives, the sections it leaves out of records included, so no credential appears.

testuary.settings.SettingValue dataclass

A described setting's value and its source: default, the path of the file that sets it, environment or overrides. sources names every layer that sets it, in order and in the same terms, the last being source. It holds no value. active is False for a setting the run does not use, inactive saying why.

testuary.settings.extension_items(settings)

Each item of [main] extensions in the combined list's order, as (item, source, directory): the layer that names it in the terms of SettingValue.source (the path of a file, environment or overrides) and the directory a file item is relative to, that of the file that names it, else the project directory. Nothing is loaded, so two items that name one file both appear. Settings that dataclasses.replace made keep the rows of their resolution while the list stays, and an item a replace adds counts as an override.

testuary.settings.update_file(path, section, values=None, *, remove=())

Set values and remove remove in section of the settings file path, keeping every other line, and write it only when the whole file validates as the resolver's. Else ConfigurationError names the problem and the file stays. A missing file is created.

testuary.settings.add_extension(item)

Name item, a file or an installed module, in [main] extensions of the user's config.ini when that file does not name it yet, keeping its other items and lines, and tell whether the file changed. A run with --config does not read that file. A file item is relative to the user directory.

testuary.settings.remove_extension(item)

Take item out of [main] extensions of the user's config.ini, the key with its last item, and tell whether the file changed.

testuary.settings.check_project_settings_file(path)

Refuse the settings file path when any of its sections, such as [api] or a provider's, sets api_url, which decides where the model's key goes.

The caller passes the files that are not the user's own: Testuary refuses the addresses of its project's settings file by this rule, and a program checks each other settings file it passes to its runs. A missing file passes.

Raises:

Type Description
ConfigurationError

naming the file and each address it sets.

ValueError

when the file is not valid INI.

testuary.settings.check_project_catalogue_file(path)

Refuse the model catalogue file path when any of its sections sets api_url or api_key_env, which decide where the model's key goes.

The caller passes the files that are not the user's own: settings resolution checks each catalogue file in the project directory, and a program checks each other catalogue file it passes to its runs. A missing file passes.

Raises:

Type Description
ConfigurationError

naming the file and each such value.

ValueError

when the file is not valid INI.

Setup

The questions testuary setup asks, answered through one function, and the model catalogue and a project's settings, changed as the setup changes them.

testuary.setup.run_setup(ask, probe, say, base_dir, *, part=None, name='', announce_run=True, home=None)

Run the named part, or all three, for runs started in base_dir: models.ini and config.ini in the user directory (home when given), the project's settings file in the project's directory. part is "model", "machine" or "project" (PARTS).

ask(question) returns the answer to a Question as text, an empty text for its default. probe tries the model and the tools the answers name. say(text) shows one line to the person.

Run alone, a part always asks. In the walk of all three, a part already set is passed with one line, and every other part offers a skip. The result lists what was written and what a run still lacks. announce_run names the run command at the end. testuary setup switches it off and names the command after its check of the project.

testuary.setup.Question dataclass

One question: its stable key, the text shown, the default Enter accepts, its choices, and the reason shown with it, if any. A closed question takes one of its choices. An open one also takes another answer, such as a path, that its check accepts: its choices are what the setup found. A secret answer is read without echo and never shown.

testuary.setup.Ask = Callable[[Question], str] module-attribute

The question function run_setup takes: it returns the answer to a Question as text, an empty text for the question's default.

testuary.setup.Say = Callable[[str], None] module-attribute

The function run_setup shows one line to the person with.

testuary.setup.terminal_ask(question)

Ask on the terminal: the default in brackets, a secret without echo. Without terminal input the wizard stops instead of writing.

testuary.setup.PARTS = ('model', 'machine', 'project') module-attribute

The setup's parts, as run_setup's part names them.

testuary.setup.Probe

Looks at the machine for defaults and checks. The demonstration replaces it with answers from fixtures. cpu_count returns the number of CPUs this process may use, and local_ollama whether an Ollama on this computer lists its models at http://localhost:11434 within two seconds.

cpu_count()

The number of CPUs this process may use: those its affinity allows (allowed_cpus), or the machine's where the platform has no affinity, at most what a CPU quota leaves (_quota_cpus), 1 when none can be read.

environment(name)

The environment variable name, empty when it is not set.

is_dir(path)

Whether path is a directory.

is_file(path)

Whether path is a file.

source_files(root, language)

The language's source files below root, relative to it with forward slashes, as the language's adapter tells them: files with its file extensions outside the directories it ignores (ignored_dirs, and dot directories), except the names it declares as no sources (non_source_files), its test files (is_test_file) and empty files, which hold nothing to test.

read_text(path)

The text of the file path, or None when it cannot be read.

run(argv, timeout)

The exit code and output of argv, or None when it does not start or does not end within timeout seconds. It gets no input, and output the locale cannot decode is kept with replacements.

which(command)

The path of the program command on the search path, empty when it is not found.

local_ollama()

Whether an Ollama on this computer lists its models at http://localhost:11434 within two seconds.

model_list(provider, address, key, *, timeout=15)

The model names the service lists, a request that generates nothing, or None when it cannot be read. The key goes into the request header only.

restrict_to_user(path)

Leave path readable by its owner only, where the system allows it. POSIX files are written with permissions 0600. On Windows the file's inherited access is replaced by its owner's.

testuary.setup.Provider dataclass

A kind of model service the wizard offers.

testuary.setup.PROVIDERS = (Provider('ollama-local', 'Ollama on this machine', 'ollama', 'http://localhost:11434/v1', 'none', starters=1, starters_reason='a local Ollama answers one request at a time'), Provider('ollama-cloud', 'Ollama Cloud', 'ollama', 'https://ollama.com/v1', 'required', starters=4, starters_reason='Ollama Cloud refuses more than a few concurrent requests'), Provider('openai', 'OpenAI', 'openai', 'https://api.openai.com/v1', 'required', starters=4, starters_reason="a cautious start, raise it within your account's rate limit"), Provider('deepseek', 'DeepSeek API', 'deepseek', 'https://api.deepseek.com/v1', 'required', starters=4, starters_reason='DeepSeek states a limit of 25 parallel requests'), Provider('vllm', 'vLLM server', 'vllm', 'http://localhost:8000/v1', 'optional', custom_address=True, starters=10, starters_reason='a vLLM server sustains parallel requests'), Provider('openai-compatible', 'another OpenAI-compatible API', '', 'http://localhost:8000/v1', 'optional', custom_address=True, starters=4, starters_reason="a cautious start, raise it within the service's limit")) module-attribute

The kinds of model service the setup offers, in the order it offers them.

testuary.setup.SetupResult dataclass

What the wizard wrote and what a run still lacks. project is the directory of the project it wrote, where run runs it.

testuary.setup.SetupError

Bases: RuntimeError

The wizard stopped, or a catalogue change was refused. The message says why and what to do. For a refused change, problems names each refused value as (field, problem), the field being the argument of set_model or set_project it concerns, or the project key.

testuary.setup.models(base_dir=None, *, home=None, probe=None, models_files=None)

The aliases of the model catalogue of runs in base_dir (the current directory when omitted), the user directory's (home when given), with where each key comes from and whether a run can use the alias, by the rule the setup offers aliases by. probe reads the environment. models_files names the catalogue files read in order in place of the user directory's, joined as resolve_settings(models_files=...) joins them.

testuary.setup.ModelEntry dataclass

One alias of the model catalogue as the setup sees it. address is the address the catalogue names for the alias or its provider, never with credentials, and empty for the provider's configured endpoint. key_source says where its key comes from: models.ini, environment, not needed for a local address, or missing. key_variable names the variable the key is read from, or that a run would read it from when it is missing. usable says whether a run can use the alias. file is the catalogue file the alias's section comes from. It holds no key.

testuary.setup.usable_aliases(base_dir=None, *, home=None, probe=None)

The aliases a run can use: the setup offers these.

testuary.setup.set_model(alias, *, provider, model, address='', key='', key_variable='', base_dir=None, home=None, probe=None, models_files=None)

Create or replace alias in the model catalogue, as the setup's model part writes it: the alias's provider and model, an address (empty: the provider's configured one, or the address its section already holds), a key stored in the provider's section, or the variable that holds the key. The file keeps its comments and other sections and is restricted to its owner while it holds a key. SetupError names a value the setup would refuse, and an alias that resolved before and would no longer resolve as a run resolves it.

With models_files, the catalogue files read in order (models), the change goes to the last file, which it creates when absent. A section only an earlier file defines is written into it whole from the earlier section's keys. No key is written: a key is refused, and so is a last file that holds one. A section whose key an earlier file holds needs key_variable, and no address that holds a credential is written.

testuary.setup.remove_model(alias, *, base_dir=None, home=None, probe=None, models_files=None)

Remove alias's section from the model catalogue, keeping every other line. Its provider's section stays: other aliases can use it. SetupError refuses a removal that leaves an alias that resolved before unresolved. With models_files (set_model), only a section of the last file is removed, and an earlier file's section of the alias applies again.

testuary.setup.project(base_dir=None, *, home=None)

The settings of the project of runs in base_dir (the current directory when omitted), found from there upward as a run finds it, with the user directory home when given.

testuary.setup.ProjectEntry dataclass

A project's settings as the setup writes them: its directory (project_dir), its settings file (settings_file), and the file's [project] values with the model alias of its [api] section as model (fields).

testuary.setup.languages()

The languages the setup offers: the installed ones, as LangAdapter lists them (in name order).

testuary.setup.set_project(values, *, remove=(), base_dir=None, home=None, probe=None)

Set values and remove the keys remove in the settings file of the project of runs in base_dir, as the setup's project part writes them: model as [api] model, every other key in [project]. Without a project there, base_dir becomes one. A key a run would refuse, a value of more than one line, a language that is not installed and a value the setup's project part refuses (an import directory, interpreter, JDK or build tool that does not exist, checked with probe) are refused, each named, and nothing is written. The file keeps its comments and other lines. home is the user directory, whose configuration names the Testuary directory.

Runs

testuary.orchestrator.RunRequest dataclass

Targets and execution policy for run_pipeline(..., settings=...).

Source paths are individual strings, never comma-delimited CLI text. The model, the project, its runtime and the limits belong to RunSettings.

runs runs each target's four phases that many times, as repetitions r0 to r(runs-1), each from a fresh copy of the project and in its own directory. run runs only repetition K (rK) instead, the same one runs makes when it is above K, so a missing repetition can be added alone.

targets runs only some of source_files, by position, as --targets does, such as "1-5". reuse names the phases whose saved results the run takes, as --reuse does, and resume continues an interrupted run in the same run directory, as --resume does. stop_after_phase names the phase after which each target stops, as --stop-after-phase does. languages lists the languages a target may be in, whose tools are checked before the run, as --languages does.

testuary.orchestrator.run_pipeline(args, selected_languages=None, *, settings=None, event_sink=None, events=None)

Run the targets of args, a RunRequest, under settings, a RunSettings from resolve_settings, in isolated run state.

Pass settings for programmatic use, without configuration flags on args. Command-line flags such as target selection and resume are separate. Overlapping calls in one process are refused, so use separate processes. event_sink is an optional read-only display observer. It cannot control requests, retries or results. events names an event file, which Testuary writes as testuary --events FILE writes its file, without a command that resumes the run. A relative path starts from the current directory, and an existing file is replaced. A call refused before its run may start writes no file. Before the run starts, it loads the extensions settings name and captures their hints.

testuary.presentation.view.phase_label(phase)

A phase's name as the console shows it, such as Coverage phase for coverage: the label other views of a run, like a web page, use.

testuary.results.recorded_source_sha256(output_dir, target)

The SHA-256, in hex, of a target file's raw bytes that the run in output_dir recorded when the target's phases started, or None when its records hold none or its phases recorded different ones. target is the file's path relative to the run's source directory, as the run's completion names it. A phase that recorded none, such as one the run reused, does not count. A caller compares the result with hashlib.sha256 of the file's bytes read in binary. Only the records are read.

Observing a run

What run_pipeline's event_sink receives, the presenter that draws it and the observations of LangAdapter's mutation and timing progress, as the terminal guide shows them.

testuary.presentation.DisplayEvent dataclass

One observation of an operation, emitted after the producer recorded its facts. It cannot control execution, retries, reduction or what is saved.

The identity tells concurrent work apart and shows only at the debug level. elapsed_s is the run's clock, completed and total count actual work, not a request limit, and message is a short diagnostic, never a prompt or a model reply. level places it in the layered view (run, target, phase, step or operation), or None to infer it from the identity. label is its name in plain words. log_path names the run's detailed logs on a run-level observation. failure, on a failed one, tells an execution failure ("execution") from a result of the method ("outcome", such as a target without a suite that meets the minimums), or None.

state is running, waiting, retrying, succeeded, failed, cancelled, refused or limit_reached. limit_reached is work that stopped at a limit its settings set (a loop's requests, tokens or time), an ordinary end and no failure.

from_dict(data) classmethod

The observation a mapping of its fields describes (record_from_dict).

testuary.presentation.Metric dataclass

A named value in the producer's units, with its measurement status.

Percentage values use 0 to 100, not a fraction. A progress value is never described as a final score. None is unavailable, not an observed zero. goal is the configured target in the same unit (for example a coverage goal). It is shown beside the value, never as progress toward completion.

testuary.presentation.Usage dataclass

Already-accounted usage. Omitted provider observations remain unknown.

logical_request_limit is the configured ceiling for the phase, a limit and not a prediction of how many requests the phase will need.

testuary.presentation.TerminalPresenter

A callable observer that draws view, a RunView, with Rich. It installs no hooks and redirects no global stream.

As a context manager it restores the terminal on success, failure and Ctrl+C, and no display error hides an execution error. Output goes to stderr, so stdout stays machine-readable. A redirected or dumb terminal gets plain output whatever the display mode. Debug is the detailed view with internal identities.

provenance property

The provenance of the newest observation, observed before any.

write_diagnostic(write)

Run one diagnostic write unchanged, with the compact view paused around it and resumed on a fresh line. Writes and observations are serialized, and a nested write resumes the view only after the outer one. A display failure falls back to plain output, while write runs once and its exceptions propagate. Writes outside this call are not covered.

plan(plan)

Show the run's plan (RunPlan) once, in one line, before the run's work. A plan of one file and one run leaves the compact view one bar, the Overall row's (RunView.snapshot).

redact_with(function)

Also remove the run's own credentials with function, the resolved settings' redaction, from everything shown from now on. The patterns alone miss a key that carries no recognizable name.

clean(value, *, preserve_whitespace=False)

value as it may be shown: the run's own credentials removed, then terminal controls and recognizable credentials.

frame()

What the compact view draws now, also for looking at it in a test.

testuary.presentation.measurement.mutation_progress_event(progress, *, run_id, target, elapsed_s, phase='mutation', operation=None, provenance='observed')

One progress record as a running operation observation.

elapsed_s is the run clock the producer uses for its other observations. The record's own elapsed_s counts from the start of the mutation run and is not substituted for it. An unknown total stays unknown.

testuary.presentation.measurement.warmed_progress_event(progress, *, run_id, target, phase='suite timing', provenance='observed')

One progress record as a display observation. An entered iteration is not a sample.

testuary.presentation.measurement.warmed_result_event(result, *, run_id, target, phase='suite timing', provenance='observed')

The final result: outcome, validity, counts and, only when usable, summaries.

Event files

The event file --events writes, and its writer for a program whose run a page shows as a run's, such as a web interface extension's job kind. The terminal guide describes its records. read_completion reads a run's end-of-run summary, the RunCompletion in results/completion.json.

testuary.presentation.EventFile

What the display shows, as JSON lines, for programs that show a run. Testuary's --events and run_pipeline's events write through it, and so does a program whose run a page shows as a run's, such as a web interface extension's job kind.

A header line holds the format version (schema_version, EVENT_FILE_SCHEMA) and the Testuary version. Each further line holds one record, in the order the display shows it: {"event": ...} with the fields of a testuary.presentation.DisplayEvent, {"readiness": ...} with the setup report before the run (RunReadiness.as_dict()), {"plan": ...} with what the run will do (a RunPlan), and {"completion": ...} with the end of the run (a RunCompletion). Every text has its credentials removed as the display removes them before it prints, with the same secrets. A failed write ends the file, not the run or its display. Events hold finite numbers only, so the JSON needs no NaN.

readiness(report)

Write the setup report report, a RunReadiness, as a readiness record.

completion(completion)

Write the end of the run completion, a RunCompletion, as a completion record.

plan(plan)

Write what the run will do, plan, a RunPlan, as a plan record.

testuary.presentation.EVENT_FILE_SCHEMA = 3 module-attribute

The event file's format version, in its header line. A change of the format raises it. The completion record's own version is separate.

testuary.presentation.RunPlan dataclass

The run's files (relative to the project's source directory), its runs (r0, r1 and so on), its first three phases in order and Finishing, the fourth.

from_dict(value) classmethod

The plan from its dictionary (record_from_dict).

summary()

The plan in one line, as a display shows it before the run: the phases in order with their states, then Finishing.

testuary.presentation.events.PlannedPhase dataclass

One of the run's first three phases: its name and label, its state (planned, off or reused, as a PlannedStep's), the steps it goes through and its limits.

The steps form three groups: a start that runs once (steps[:cycle]), a cycle that repeats (steps[cycle:end], empty in the starter phase) and the phase end that runs once after it (steps[end:], the consolidation of every phase). end None is the number of steps: no phase-end group.

testuary.presentation.events.PlannedStep dataclass

One step of a phase, or Finishing: its name, its label in plain words and its state: planned, off when the run stops before it (--stop-after-phase), or reused when the run takes its results from an earlier one (--reuse).

testuary.presentation.events.PlannedLimit dataclass

A configured limit or goal of a phase, in its unit. None is no limit.

testuary.presentation.RunCompletion dataclass

The whole invocation: its files, where results and logs are, how to continue.

resume_command is the producer's exact command for continuing this run, or None when there is nothing to continue or no such command.

unfinished property

The files a resume would continue: those without a suite that meets the minimums that the producer did not mark as not resumable.

from_dict(data) classmethod

The completion a mapping of its fields describes (record_from_dict).

testuary.presentation.read_completion(path)

The end-of-run summary at path, a run's results/completion.json, as the pair of its RunCompletion and its provenance as the file records it, synthetic for a demonstration's values. Only the file is read.

Raises ValueError when the file is not JSON or does not hold this Testuary's summary format version and a completion, ValueError or TypeError when the completion's fields do not match this version's, and OSError when the file cannot be read.

Checks

testuary.orchestrator.check(request, *, settings, verify=False)

Check the request's setup without generating tests, and return a RunReadiness without printing.

Returns the setup report as a RunReadiness and prints nothing. With verify it is the tool check, which starts the tools and tries the project in a copy, and for a Java project downloads what a run would when it is missing. Without verify it downloads nothing. An item the check cannot try is reported as not tried: only a run tries it.

testuary.readiness.RunReadiness dataclass

The setup report of testuary check, or of a run before its first model request: its scope, status and exit code, the selections (project, execution, source files, phases, model, run directory and whether it starts fresh, resumes or reuses), the limits and the items. A configured key is reported as configured, not as accepted by the provider. The report holds the redacted facts its producer observed and promises nothing about later requests, cost or provider capacity.

as_dict()

The report as detached JSON data, without settings values, environment or keys.

testuary.readiness.ReadinessItem dataclass

One item of the report: an operation, tool or setting by key, whether the run needs it, its status, what was selected and its version, the reason, the next action and a diagnostic file (diagnostic_path). It speaks for that operation only, not for the language as a whole. The status is one of "verified", "missing_setup", "unsupported", "ambiguous", "unverified", "failed" and "refused" (ReadinessStatus). tried_by names what first tries an item not tried yet: "verification" (testuary check --verify) or "run" (only a real run), otherwise None. It matters only while the item is unverified.

testuary.readiness.ReadinessLimit dataclass

A limit as configured for phase: a value in unit (configured), unlimited or unknown. Only a configured limit has a value, so None is never read as zero.

testuary.presentation.present_readiness(report, *, presenter, before_run=False, playback=False, steps=TERMINAL_STEPS)

Write one report through the presenter's diagnostic boundary.

Compact and plain output summarize what was found, what is installed but not tried, what was not tried yet and each problem with its action, ending with one next step. The detailed view lists every item with its raw names. Statuses, selections and actions are the producer's: nothing is checked, installed or inferred here. before_run marks the summary shown before a run's first model request, and playback a demonstration's playback, which names no API key and ends by saying that it costs nothing. steps words the reader's next actions: the terminal's commands unless another interface passes its own.

Models for your own program

A model you configured for Testuary, used from your own program: a LangChain chat model for an alias of your model catalogue or a model a provider serves, with Testuary's retries and request records. The models guide shows a call.

testuary.models.catalogue_chat_model(model, *, base_dir, records_dir, label, request_values=None, config_files=None, project_settings=True, models_files=None, provider=None)

A LangChain chat model for your own program's model calls, for which model and provider resolve as [api] model and [api] provider do in a Testuary run in base_dir: an alias of your model catalogue with its own provider, else, with provider, a model of that provider with its address, protocol, request settings and key, from the run's settings files and catalogue, read only by path. config_files, project_settings and models_files choose the settings files and the catalogue as they do for testuary.settings.resolve_settings: an empty list reads no configuration file, a list of files reads them in place of the global config.ini, project_settings false reads no project.ini, and models_files names the catalogue files read in place of the user directory's models.ini, a later file's section replacing the same section of an earlier one. A program whose Testuary runs name their settings files and catalogue passes the same choice, so that its models read the same files.

request_values sets request values above the configured [api] values, and no phase section applies. Its keys are the request values of models.ini that REQUEST_VALUES holds: temperature, top_p, seed, max_output_tokens, request_timeout_seconds, reasoning, reasoning_effort, reasoning_budget_tokens, preserve_reasoning_content, reasoning_field_names and request_options. The catalogue's output limits of the model still bound the output. The model's tool binding goes through the same retries and records.

Each request is retried under Testuary's bundled retry settings: the packaged values of [coverage]'s max_consecutive_transient_failures, max_transient_failures_per_phase, transient_first_wait_seconds, max_transient_wait_seconds, max_transient_hint_seconds and count_retries, which the settings reference shows. A caller cannot change them. The ceiling of max_transient_failures_per_phase counts over the model's life: once it is reached, the first temporary failure of a later request raises model_requests.TransportExhausted, and a new model starts over. Every request sent is appended to request_records.jsonl in records_dir: a request record with the key label. Each created model appends its settings without credentials to model_settings.jsonl there, with label. Both files only grow, so a resumed program keeps its earlier records.

Raises ConfigurationError before any request for a name the catalogue does not define without provider, or a provider that neither Testuary configures nor a provider section defines, naming the catalogue files, an absent file of models_files, an ollama-native model, a credential that cannot be resolved, an empty label, or a request_values key that names the identity, a credential or no request value. A request raises model_requests.TransportExhausted when a ceiling is reached, model_requests.TerminalRequestError for a terminal refusal and model_requests.ContextOverflow when the prompt does not fit the model's context. Their messages name the provider's error type and text without the key, and they carry no cause, since a provider's error can hold the request and its key. Any other error, such as a request record that cannot be written, is raised without the errors chained to it. The records hold no key either.

testuary.models.REQUEST_VALUES = frozenset({'temperature', 'top_p', 'seed', 'max_output_tokens', 'request_timeout_seconds', 'reasoning', 'reasoning_effort', 'reasoning_budget_tokens', 'preserve_reasoning_content', 'reasoning_field_names', 'request_options'}) module-attribute

The request values catalogue_chat_model accepts in request_values: the request fields of a model catalogue entry without protocol, which belongs to the alias's identity. The refusal of any other key lists them.

testuary.model_requests.TransportExhausted

Bases: Exception

A retry ceiling was reached. The last transport failure is the cause. attempt is the attempt of the model request that reached it.

testuary.model_requests.TerminalRequestError

Bases: Exception

A request failed in a way that retrying cannot fix. The provider's error is the cause.

testuary.model_requests.ContextOverflow

Bases: Exception

The provider refused a request because the prompt does not fit the model's context. It is not retried, since the caller owns its prompt and its output limit. Its message names the provider's error type and text without the model's key, and it carries no cause, since a provider's error can hold the request and its key.

Demonstrations

The offline demonstration, played as testuary --demo plays it.

testuary.demo.play(language='python', *, output_dir, data=None, speed='auto', duration=None)

Play the offline demonstration as testuary --demo does and return its exit status (0 when it completed, 130 when Ctrl+C stopped it).

language selects a language's demonstration, or data names demonstration data in its place (as --demo-data, without language). output_dir must be new or empty. speed is auto, instant or a factor of the configured length's speed, such as 2 for half as long (--demo-speed), and duration the paced playback's length in seconds (--demo-playback-seconds). A demonstration must start before any part of a run is loaded, so each call plays in a Python process of its own: a script can play several, one after the other, and Ctrl+C reaches the playback, which ends with its own cleanup.

Web interface

The local browser interface, started as testuary-web starts it.

testuary.web.serve(base_dir=None, *, port=None, access=None, open_browser=True)

Start the web interface for base_dir as testuary-web does and return its exit status (0 when Ctrl+C ended it, 2 when base_dir is no directory, port cannot be used, access is none of the values below or the installed Testuary does not answer in time).

base_dir is the directory Testuary works in (default: the current directory). port is the port on 127.0.0.1 (default: [web] port of Testuary's settings, 8765, and 0 takes a free one), access who may use the interface (default: [web] access, open): open, every program on this computer that reaches the port, key, only a page opened with the printed address, whose access key is new at each start, or stable_key, the same with one key kept in Testuary's user directory. open_browser opens the printed address in the browser. In an SSH login (SSH_CONNECTION set) it opens none and prints the ssh -L line for the user's own computer instead. The call blocks until Ctrl+C, a SIGTERM or Ctrl+Break ends it. A job that is still running then stops first, as Ctrl+C stops a run, and keeps the work that finished. Call it from the main thread, which receives Ctrl+C.

The web interface's extension point, the top-level name testuary_web of a module that [main] extensions names (Extend the web interface).

testuary.web.WebExtension dataclass

What an extension module adds to the web interface: the extension API version it was written for (api), its id, and optionally the product name, tabs, Testuary's tabs' titles by id (tab_titles), positions for Testuary's tabs other than the Overview by id, in the words of a tab's position (tab_positions), the ids of Testuary's tabs other than the Overview it hides (hidden_tabs), the address the header's Guide link opens (guide): an http or https address or the name of an HTML file next to its module, None to hide the link, or the empty string for Testuary's guide, dir_types, settings_groups, job_kinds and read-only routes.

testuary.web.WebTab dataclass

A tab an extension adds: its id (the address #/), its title, its view script (a file next to the extension's module), its position: "first", "last", "before:" or "after:", and the id of the Testuary tab it replaces, the Overview excepted. A position may name an added or a moved tab, and a position naming a tab no one has places the tab last. A replaced tab leaves the header, and its address and Testuary's links to it open the replacing tab.

testuary.web.DirType dataclass

A kind of directory an extension's result view opens: its id and title, detect(named_dir) whether a directory is one, summary(named_dir) the JSON data its view shows, read from the files the extension wrote, and the view script.

testuary.web.SettingsGroup dataclass

Settings an extension adds to the Settings page: its id and title, reference(project_dir) its sections as testuary.settings.SectionDescription, values(project_dir) each setting's value by (section, name), and update(project_dir, section, values, remove), its validated update, which raises ValueError with the reason to refuse a change. project_dir is the listed project the page shows, or None for the global settings.

testuary_settings_files(project_dir), optional, names the Testuary settings files the extension's runs in the listed project project_dir read, in the order resolve_settings(config_files=...) reads them and with the extension's own file last, also while it does not exist, or None when they name none. The runs read them in place of both the global configuration and the project's own settings file. A relative path starts from the project directory. The Settings page shows the settings as those runs read them and changes the last file, when it lies in the project directory.

testuary_models_files(project_dir), optional, names the model catalogue files those runs read, in the order resolve_settings(models_files=...) reads them and with the extension's own file last, also while it does not exist, or None when they name none. The Models page shows the models as those runs read them and changes the last file, when it lies in the project directory or in extension_user_dir, the extension's own user directory. end_models_import(project_dir), optional, makes those runs read the last file alone. The Models page calls it after it copied every model of the earlier files but one into the last file, to remove a model only an earlier file defines, and the function changes the extension's settings as its last step.

After the Settings page or the Models page writes a file of such a list, it calls the list's function again, and a function that raises refuses the saved file: the file returns to what it held before, and the error's message is the reason the page shows.

testuary.web.JobKind dataclass

A job an extension starts: its id and title, the module the job runs as python -m does, command(form) the module's arguments for the form the page sends (ValueError with the reason refuses it), resume(arguments), the arguments Continue runs, or None for a kind without Continue, and nested_runs(arguments), the Testuary runs a job started, each a NestedRun its page folds into its phase, or None for a kind whose jobs start none. Both receive the job's arguments without the module. The job adds --events PATH, where the module writes Testuary's event file through testuary.presentation.EventFile.

testuary.web.NestedRun dataclass

A Testuary run that a job of an extension's kind started: the id of the phase of the job's plan that started it, the run's event file or run directory (path, as the page's nestedRun names a run) and the title its folded part on the job's page shows after "Testuary run", or None for none.

testuary.web.WebRoute dataclass

A read-only route of an extension: it answers GET at /api/ext/<extension id>/<path> with the JSON read(named_dir, query) returns. named_dir is the directory the query's dir names, resolved as the Results view resolves one, or None without dir. query holds the query's first values by name. A ValueError refuses the request with its message, and any other error answers with the error.

testuary.web.WEB_EXTENSION_API = 1 module-attribute

The extension API version this Testuary supports. Version 1 covers the event file's schema version 3 (testuary.presentation.EVENT_FILE_SCHEMA).

testuary.web.TOOL_CHECK_LABEL = 'Tool check: also start each tool once and try the project in a copy' module-attribute

The label of the tool check option on Testuary's Check page, which the check report quotes while that page is Testuary's own. /api/state carries it as tool_check_label, for a replacing Check view that offers Testuary's tool check.

Hints of extensions

The hints of the extensions the settings name, captured and validated as a run captures them, and the guidance one extension's hints or a hints file add to a phase's model request.

testuary.hints.extension_hints(settings)

The hints of the extensions settings name, in the combined list's order, captured now. Each mapping is written once as JSON and validated as a hints file. An extension that cannot load, a mapping JSON cannot hold and invalid hints raise testuary.settings.ConfigurationError naming the item and where it is named.

testuary.hints.request_text(hints, language, phase, *, hint_profile='')

The guidance hints adds to phase's model request for language, empty when it adds none. hints is an extension's hints from extension_hints or a hints file's parsed content. phase is initial, coverage or mutation, and hint_profile selects the hints file's profile cells as [project] hint_profile does.

testuary.hints.ExtensionHints dataclass

An extension's hints as one command captured them: the item that names the extension, the extension's resolved file and its testuary_hints as JSON bytes.

testuary.hints.PHASES = ('initial', 'coverage', 'mutation') module-attribute

The phases whose prompts read hints and a prompt profile: the starter phase and the two loops.

Generation records

The readers of a run's generation records, which Generation records describes.

testuary.generation_evidence.read_suite_slots(manifest, suites_root=None)

The manifest's suite_slots section, checked, as a detached copy. With suites_root the accepted references are verified too. A missing or unknown section raises: a disposition is never inferred from counts or original.json.

testuary.generation_evidence.read_cleaned_source(reference, suites_root)

The cleaned source a reference names, checked against its recorded hash, so a file overwritten since raises.

testuary.generation_evidence.read_ancestry(record, source=None)

The record's construction graph (generation_ancestry), checked and detached. It must belong to the record's selected source, and source when given. Anything unreadable or contradictory raises.

testuary.generation_evidence.GenerationEvidenceError

Bases: ValueError

Generation evidence is missing, contradictory or cannot be resolved.