The web interface¶
The web interface is Testuary's local browser interface. It starts the same Testuary commands you would type at a terminal and shows what they report: the quick check and the tool check, a run's progress, its results and the demonstration. Every value and text on its pages comes from Testuary, so a page and the console never disagree. The interface is optional: Testuary works the same without it.
Install¶
The web interface comes with Testuary: installing Testuary installs the
testuary-web command. It needs nothing else, except git for adding a
project from its GitHub address. The pages are plain HTML, JavaScript and
CSS served from the package, and the server uses the Python standard
library and psutil.
Start¶
Open a terminal in your project's directory, or in any directory you want to
work from, and start the interface. Testuary reads your models
(models.ini) and global settings (config.ini) from your Testuary user
directory, and each project's settings from .testuary/project.ini in the
project's directory:
testuary-web
It prints the directory, the installed Testuary and LangAdapter versions and
its address on 127.0.0.1, at the port of
[web] port, and opens that address in your
browser. Without an access key it adds that every program on this computer
that reaches the port can use it. Options:
| Option | Meaning |
|---|---|
--dir DIR |
The directory to work from, instead of the current directory. Its project, if it has one, is listed first. A directory that is no project lists the projects directly below it. Demonstrations go there |
--port PORT |
The port on 127.0.0.1, in place of [web] port. A port that is taken ends the start with a message that names what holds it: your own interface (then --replace stops that one first), an interface of another Testuary user directory or user, or another program (then choose another port) |
--access MODE |
Who may use the interface, in place of [web] access: open, key or stable_key, described below |
--no-browser |
Print the address without opening a browser |
--replace |
First stop your interface running on this port, as Ctrl+C stops it, then start |
--stop |
Stop your running interfaces, with --port PORT only that port's, as Ctrl+C stops them, and start none |
--version |
Print the version, which is Testuary's |
testuary-web exits with status 0 when the interface is stopped, or when
--stop ends every interface it finds, 1 when --stop finds one that does
not end, and 2 when the interface cannot start or an option is refused.
Access¶
With open the interface needs no access key, and its address is
plain. Other web pages in your browser still cannot use it (see Safety),
but every program on this computer that reaches the port can: on a
machine you share with other users, they can start runs with your models
and your model key, and read your results. There, start it with an access
key:
--access key: the printed address carries a key, new at each start, after#token=. Open the address as it is. The page keeps the key in the browser's local storage for the server's address, its port included, so new tabs, copied addresses and bookmarks of the same running interface work without it. After a new start, open its new address.--access stable_key: the same, with one key kept in your Testuary user directory (web/access-key), so the address stays the same from start to start. Use it only where nobody else can listen on the port while the interface is down: another user who does can take the key from a page opened at that address and use it at the next start. Deleting the file makes a new key.
[web] port and [web] access in your config.ini set the port and the
access for every start, and the Settings page shows them under Web
interface. As any setting, a project's .testuary/project.ini can set
them for starts in its directory, and TESTUARY_WEB_PORT and
TESTUARY_WEB_ACCESS for starts in that environment.
Ctrl+C in the terminal stops the interface. A job that is still running is stopped first, the way Ctrl+C stops it at a terminal, and the interface waits for its cleanup. A second Ctrl+C ends the job at once. One job runs at a time in the working directory, also across several starts of the interface there.
An interface running elsewhere, in another terminal or in the background,
stops the same way with testuary-web --stop, or with
testuary-web --stop --port PORT only the one on that port. Each start
records itself in your Testuary user directory (web/running/<port>.json)
with a stop secret, and only a request that carries the secret stops it.
Another program on the port and another user's interface are never
touched. An interface started with another TESTUARY_HOME has its record
in that directory: --stop and --replace find it with the same
TESTUARY_HOME. A record whose interface no longer runs is removed.
--stop and --replace wait until the interface has ended, after its
job's cleanup.
Over SSH¶
The interface listens on 127.0.0.1 of the machine it runs on, so a browser
on another computer reaches it only through an SSH tunnel. Started in an
SSH login, testuary-web opens no browser on that machine. After the
address it prints the ssh -L line for your own computer, for example:
ssh -L 8765:127.0.0.1:8765 you@203.0.113.7
Run that line in a terminal on your own computer and keep that login open,
then open the printed address in your own computer's browser. The line
gives the interface's port on both sides, adds -p when the SSH server
listens on a port other than 22, and names your login and the address
this SSH connection arrived at. Behind a router or on a cloud machine,
that address can be a private one your computer cannot reach: use the
name you log in with in its place.
The port must be the same on both sides of ssh -L, because the
interface answers only requests that name its own port. The port stays the
same from start to start, so one tunnel serves every start: log in with
the ssh -L line, run testuary-web in that login and open the address it
prints. If the port is taken on either computer, choose another one with
--port or [web] port and use it in all three places of the ssh -L line. A
machine others log in to is a shared one: start the interface there with
--access key.
The pages¶
Every page fits a window as narrow as a phone's. In a narrow window, these
tables show each row as a block, each value under its column's name: a
check's items and limits, the Jobs list, a run's operations, the Results
page's targets and repetitions, how each phase ended and a phase end's
candidates. In a narrow window, a repetition's row of tests.csv shows
each name above its value. Another wide table or a file shown as text
scrolls in its own box. On a job's page, a text box follows new output
while you are at its end. Scrolled up, it keeps your place, and a part you
opened or closed stays so. The Guide link in every page's header opens
this guide for the installed version, in a tab of its own.
Overview, Projects and Models¶
The overview shows your projects with their latest results, your models,
the recent jobs and the main actions: start a run, check a project, open
results. A model reads default model when your global settings choose it
([api] model in your config.ini, which names none until you set it).
It reads tenacity's model, for example, when tenacity is the project of the
directory you started in and names it ([api] model in its
.testuary/project.ini, which the project setup writes). Each recent job
names its run directory, its model and its values, as the Jobs page does.
An open overview follows the jobs without a reload: a job shows there as it
starts and ends, and the projects' latest results are drawn again once a
job has ended. A project's latest results are the newest that Testuary
summarized in the project's .testuary/results: each target
with its fully measured repetitions and the mean of each value across them.
A project is a directory with its own .testuary directory, which holds its
settings file, project.ini, and its results.
Projects lists the project of the directory you started in (in a directory that
belongs to none, the projects directly below it) and your recent projects,
which the interface keeps in your Testuary user directory. Add a project by
choosing its directory and answering Testuary's project setup there, one
question at a time, as testuary setup project asks them. With a GitHub
address (https://github.com/owner/project), git clones the project into a
new directory, named after it, in the directory you choose, and the setup
follows there. It takes HTTPS addresses on github.com only and needs git.
Without git it says so. A directory of that name that exists already is refused.
git clones without the project's submodules, asks for no login and has 15
minutes. The setup offers what it finds. The languages come with the one with
the most of the project's files first. The interpreters are the project's
environments and the Pythons on your PATH, each a choice of its own with its
whole path, and Another path takes an interpreter of yours. The JDKs are the
ones it finds, the source files are checkboxes, and the models are yours, or a
new model added in place. Testuary checks each answer and shows a refused one's
reason beside it. Back returns to the question before. The last step lists the
settings to write by their names, with the file's text folded below, and Write
the settings saves them. An address in the setup's text, such as the settings
reference the file names, is a link. The list then shows the new project. Edit
settings sets its language, its source files, its model and its language's
tools, and links all of the project's settings on the Settings page. Forget
takes a recent project off the list and leaves its directory, settings and
results. The project of the directory you started in and the projects below it,
marked this directory's project and in this directory, stay listed.
Models lists your model aliases as Testuary's setup reads them, with where
each key comes from, and marks the default model and the project's model
as the overview does. A model's badge reads SET UP when the alias has its
key, or needs none at a local address, and NO KEY otherwise. Whether its
service answers, only a connection test tells: Test the connection reaches
the service as a run would, with the same address and key, and asks it for
the models it serves, which generates nothing and costs nothing. The
answer names the address, without credentials, and says whether the
service answered and lists the model. Add, edit and remove an alias there.
Add refuses an alias that exists, with the reason beside the field, and
Edit changes an existing model, opening with its own kind of service and
provider name. Removing a model keeps its provider's section in
models.ini, since other models of that provider use its address, and the
confirmation says so. A model's key line says whether a key is stored in
models.ini or comes from an environment variable, and never shows the
key. To replace a stored key, Edit the model and type a key in its Key
field. An empty Key field keeps the stored key.
When an installed extension names the model catalogue files its runs in a project read, the Models for choice offers them for that project beside Your model catalogue. Each model then shows the file it comes from, its full path on hover, and a change goes to the last file, which holds key variables and no key, and no address or key variable at all when it lies in the project directory. Editing a model an earlier file defines saves a copy of its section in the last file, where it takes the place of the earlier one, and removing the copy brings the earlier model back. A new address for a model of another OpenAI-compatible API also copies its provider section, shown with its own Remove. A model only an earlier file defines is removed by ending the import, where the extension offers it. The page asks first and says that every other model is copied into the last file, which the runs then read alone, and asks for a key variable for each key an earlier file holds. It then checks that the runs read the last file alone, and otherwise puts the file back and says what holds now. A last file outside the project directory and the extension's own directory is shown read-only with the reason.
Settings¶
Every setting by group (Model, Coverage phase, Mutation phase and so on), each with its name, its value, where the value comes from and Save, at any window width: Testuary's default, your global settings, this project's settings file, the environment or a run's options, told apart by color. The groups a run usually changes come first and open: the model, the project, the starter phase and both loops. Each shows its common settings and every setting a file or the environment sets, and the rest of the group is one fold away. The other groups follow under More settings. Find a setting searches every setting by its name, label, purpose or group. A setting the run does not use, such as another provider's service or another mutation engine's options, is grayed with the reason. The page opens on the project of the directory you started in, when there is one, else on the global settings. Choose all projects to change the global settings, or one project to change its own. An installed extension can name the Testuary settings files its own runs in a project read, its own file last. The project then has a further choice, Testuary settings for the extension, which shows those settings with the file each value comes from and changes the last file. Those runs read the files in place of your global settings and the project's own settings, so a value either of them sets does not apply to them unless the extension names their file. Each change goes through Testuary's validated update, which refuses a value a run would refuse and keeps the file's comments. Save and Reset keep the page where it is and say what they did, and a refused value reads "Not saved:" with the reason beside its setting. About this installation, at the end, names the installed Testuary and LangAdapter versions.
Run¶
Starts a run of a project in its directory, as testuary runs there. The
project's source files are a checklist: the targets its settings name are
ticked, or every file when they name none. Select all and Deselect all
tick or clear every one, beside how many are ticked. A project whose
settings name no language lists the files of each language it holds, each
language with its own Select all and Deselect all. A target the settings
name that the project's directory does not hold is named above the list, with
what to do. Choose the model, all repetitions (how many per target) or one
repetition only (such as one that did not finish), and the run directory.
Left empty, the model choice names the model the run asks: the project's
model, else the default model. A relative run directory starts at the
project directory, and an empty one takes
[project] output_dir. When the run
directory already holds an earlier run, the page offers to continue it
while targets remain unfinished, or to start a new run in a new run
directory. A run sends model requests, which can cost money with a paid
provider. The project's settings can hold small limits for a first run.
Check¶
Its button, Run the check (sends nothing to a model), runs the quick
check, testuary check, in a project's directory for the targets ticked in its
checklist, with Select all and Deselect all as on the Run page. With "Tool
check: also start each tool once and try the project in a copy" ticked it
runs the tool check (testuary check --verify), which also starts each test,
coverage and mutation tool the settings select once and, in a copy of the
project, tries its build where a run builds it and finds its tests without
running one, unless the project has its own build wrapper, which may download
its build tool. When the project's run directory already holds a run, the check
judges the directory the next run from the Run page uses: it continues that
earlier run while targets remain unfinished, else it takes the free directory
beside it.
The page names that directory before the check starts. It shows Testuary's
summary with its next step, the report's facts, items and limits, and the
detailed listing. Where the console's next step names a command, the page
names its own controls: a tool check with its box ticked, the detailed
report below, or a run from the Run page. The check sends no model request
and never shows a key.
Jobs¶
Lists the jobs this start of the interface has run, each with its run directory, its model and each target's values as its end-of-run summary states them. The list follows the jobs without a reload. A demonstration's model is the recorded one, and its values are marked as a demonstration's. A run's page shows its plan as Testuary made it: the four phases (the starter phase, the coverage phase, the mutation phase and Finishing) with their steps. The starter phase does Generate and Measure suites. Each loop does Read tests and First measurement once, then Edit tests, Run tests and Measure in a cycle. Every phase ends with consolidation, which combines, reduces and compares the candidate suites, and a loop picks its suite there. The starter phase picks its suite there too, after measuring the best candidates under the name they would be delivered with. Finishing has no steps: while it runs, its box says what it does. Each step reads current, finished or upcoming, beside each loop's goal and limits, and the current step shows how long it has been under way. The starter suites' first two steps count the suites generated and measured, each loop counts its model requests against its limit, and a loop that has ended shows its measured values. A loop that needed none of its steps says why. A loop that ends at its request, token or time limit names that limit, in amber: its settings set that end, so it is no failure. Below are the rows the console's compact view draws, from Testuary's own display rows, their times moving with the run's clock: the Overall row's phases done, such as "1 of 4 phases done", the phases counted 1 to 4, each phase end, and for a failed run the cause in one sentence above the details. When the cause is a model service that gave no answer after its retries, a link leads to the connection test on the Models page. What happened in order follows, and the run's text output folded below it. Paths are shown short. The full path is on hover, and Copy path copies it.
Stop works as Ctrl+C at a terminal: the work in flight cleans up first, and work that finished is kept. If a stopped job is still cleaning up after the time Testuary states for that, Force stop ends it with every process it started. While a job stops, its header reads STOPPING, and a stopped job's badge and label share one word, STOPPED or FORCE-STOPPED. The diagram marks the work under way at the stop as stopped. A run that stops before writing any results says that it wrote no results, offers no results link, and its cause panel shows Testuary's own error text. When a run ends, its page states its outcome in Testuary's words, such as how many targets have a suite that meets the minimums, and links to its results. A run that left targets without such a suite reads MINIMUMS NOT MET, and when Testuary can finish them, Continue the run runs it again with the same options in the same run directory, keeping the finished work. Other jobs say what their exit status means, in Testuary's words. A run whose results record warnings, such as a time limit reached or a loop stopped at its limit before its goal, lists them there in Testuary's words, each with the setting it names and its value.
Results¶
Choose a project: its runs are listed, the newest first, from its results
directory (.testuary/results) wherever the project lives. Another run
directory, such as a demonstration's, opens by its path.
- The project across its targets: for each value the mean, median, range and standard deviation over the targets, a mark per target on a 0 to 100 percent track, and beside them each target's starter suites before the loops: the median of its starter suites, and with several repetitions the mean of the repetitions' medians. A project of one target shows each value in that target, without statistics across targets.
- Values per target: each repetition's delivered suite, with several repetitions their mean, lowest and highest value, and the target's starter suites as above.
- Targets and repetitions: each repetition's final values, how each phase ended and its
warnings, such as a time limit reached or a loop stopped at its limit
before its goal. Select
a target for its details: each phase's values, tests, model requests,
tokens, tool calls and time, Finishing's values, the starter suites'
statistics, where the time went, the delivered suite's files and every
column of the repetition's row in
tests.csv. - How each phase end chose its suite, in a target's details: for the starter suites and each loop, what it chose and why. That is the best score in the order its record names, such as mutation score, then branch coverage, then line coverage, or equal scores with the reduced suite preferred, or the fastest of equal suites, or, when equal suites could not be timed, the input order. Each candidate is listed with its tests, coverage, mutation score and outcome: chosen, measured but not chosen, passes but not fully measured, does not pass, or not built, the last three with their reason. The reduced suite's mutation score, when derived from the merged suite's kill matrix rather than measured, reads "(derived, not measured)" beside it. A test's output folds behind its first line. A loop that ended with the suite it received says so, and the starter suites' end names the passing suites that left without being fully measured.
- Measurements a tool could not make, in a target's details when there were any: each measurement that the coverage tool or the mutation tool refused in a loop, under that loop's name, with the tests that stopped the tool, or a note saying it named no test. The mutation phase's measurement also runs the coverage tool, so a refusal by the coverage tool can stand under the mutation phase.
- Under a run's suite directory, how to use the delivered suite: copy the
test file to where the project keeps its tests and run it with the
project's own test command. Where its tests ran with a setup file beside
them, the page names it and says that they can need it: copy it into the
same directory too. For Python this is the
conftest.pythat makes the project importable for the tests. Where that directory already has a file of that name, the two have to be joined by hand. Each file that goes into the project has its copy button, such as Copy test_calc.py into the project and Copy conftest.py into the project. The test file's button offers the directory that holds most of the project's tests of that language, in the delivered suite's package, or else the project's directory, and a setup file's button offers its test file's directory. A missing directory is made, a directory in.testuaryis refused, and nothing is written over a file. A demonstration's tests belong to its recorded project and are not offered. - The end of the run, as the console printed it, from the run directory's
completion.json, for every run. - The run's log and every file of the directory, shown as text.
The values and their names are Testuary's, unchanged, from the summaries it
writes (summary.json per run, per target and for the project): an empty
value is unknown, and the interface computes no values of its own.
Demonstration¶
Plays back one recorded run of this Testuary release, for Python or Java.
Under Speed, Normal speed plays it over the length that Length at normal
speed sets, a minimum, and Instant plays it without pauses. It sends no model
request, runs no test or mutation tool and needs no setup. Each playback
writes its record into a new directory in the interface's working directory,
which the page names, and the Results page opens it. Its stepwise playback, one
step per Enter press, needs a terminal: testuary --demo --demo-step.
Safety¶
- The interface listens on 127.0.0.1 only and loads nothing from the internet: no fonts, scripts or services. During a run, only Testuary's own model requests leave the machine. Besides a run, only the model setup and a model's connection test reach a model service, and they ask it only for the models it serves. Adding a project from its GitHub address runs git, which reaches GitHub over HTTPS only.
- Requests must name this server in their Host header, so other web pages cannot reach it through another name for 127.0.0.1, and changes must come from its own page, so other web pages cannot make them. It sets no cookie. Under every access setting these checks hold.
- Without an access key, other programs and other users of the machine
can use the interface through its port (see Access), so on a computer
you share with other users, start it with
--access key. With one, the key in the address's#part is never sent to a server by the browser. The page keeps it in the browser's local storage for this server's address and sends it as a header with every request. - No answer of the server holds a stored model key, and no log, event file or address holds one. The key goes only to its model's service address, in a Test or a run. The lists come from Testuary's setup and key-free export, and the results views never show Testuary's configuration files. The progress events Testuary writes for a job have keys removed as the console removes them.
- The results views open only directories below the working directory or in a listed project, and follow no link or junction.
- The interface's own log in the terminal records actions and exit statuses only.
Extensions¶
An extension named in [main] extensions can add tabs, result views,
settings groups and jobs to the interface and change its product name and
tabs, while Testuary's own pages, settings, access key and jobs stay as they
are. Extend the web interface lists what an extension
can change and shows how to write one.
The interface and the console¶
Every action is a Testuary command, run in the project's directory, and a
job's page shows it, for example
testuary package/module.py --display plain --no-color --model=ollama_qwen.
A run writes the
same files as at the console, and results from either can be opened in the
other. The progress rows, the summaries and the exit sentences are
Testuary's own. Among what stays at the console are the setup's machine
part, editing the settings files by hand, the demonstration's stepwise
playback, the toy and tutorial demonstrations, testuary setup --demo,
init, provision, new-language and tutorial. The
pages ask the setup's project part, and the Models page its model part.
Files the interface writes¶
The interface writes in these places:
- In the working directory, below
.testuary-web/:jobs/<job>/, each job's terminal text (stdout.txt,stderr.txt) and, for a run or a demonstration, Testuary's progress events (events.jsonl), andrunning.jsonwhile a job runs, so that a second start of the interface starts no job in the same directory. - In the working directory, each demonstration's run directory,
testuary-demo-<language>-<YYYYMMDD-HHMMSS>/. - In Testuary's user directory (
TESTUARY_HOME, else the platform's user directory):web/projects.jsonwith the recent projects' directories only,web/access-keywith--access stable_key,models.inifrom the Models page, through Testuary's setup, andconfig.inifrom the Settings page with all projects, through Testuary's validated settings update. - In Testuary's user directory, while an interface runs:
web/running/<port>.jsonwith its process, port, directory, start time and stop secret, which--stopand--replaceread. The interface removes it when it ends. - In a project's directory:
.testuary/project.ini, from adding a project, Edit settings and the Settings page for one project, always through Testuary's setup or its validated update, and the.testuary/results/.gitignorethat the setup's write creates. A delivered suite's test file copied from a run's results goes into the directory of the project you choose. - In the directory you choose for a GitHub address: the project's new directory, which git clones.
A run started from a page writes its results where the same run at the console writes them. That is Testuary's writing, not the interface's.
The results pages read each run's own run directory, its end-of-run
summary results/completion.json included, not the job history. The Jobs
page lists only the jobs of the interface's current start. When no job runs,
deleting .testuary-web/ removes only the stored jobs' terminal text and
event files. A run that ended before writing results/completion.json takes
its end-of-run summary from its job's event file, and that summary is then
lost.