Terminal output¶
Testuary shows its progress in the terminal and keeps it separate from the results and logs it saves. The display mode changes only what you see. It does not change generation, measurements, limits or which suite is chosen.
Preview the views with the offline demonstration, which needs no model:
testuary --demo --display compact
testuary --demo --display detailed
testuary --demo --display plain
testuary --demo --display detailed --no-color
For your own project, add the same options to the generate command from
getting started.
Views¶
Select a view with --display. The
command-line reference names its default.
autousescompactin an interactive terminal andplainwhen the output is redirected or the terminal has few capabilities.compactkeeps a progress view pinned at the bottom of the terminal.detailedwrites every update as a line of its own, without the pinned view. A count that only grows gets a new line at each ten percent, or every 30 seconds when its total is unknown, and work that waits gets an occasional line.plainwrites the same lines without color or cursor movement, for CI and log files.debugis the detailed view with internal identifiers added, and it also echoes every log line, for diagnosing a run.
An explicit compact or detailed also falls back to plain lines outside an
interactive terminal. No full-screen interface is used. Every plain or detailed
record is one line, so paths and commands are never split. The terminal wraps
long lines only on screen. Colors follow --no-color, or without it the
NO_COLOR environment variable. Progress goes to standard error. The display
does not capture standard output, and a few diagnostic lines still go there, such
as the list of a project's targets before a run and, with --display debug,
the path of the structured log.
The pinned view¶
The compact view shows the run in layers, from the top down: the whole run, the current target, its phase, the step within the phase and the operations running now. This frame comes from the offline Python demonstration during the coverage phase, in a terminal 100 columns wide (log path shortened):
Testuary | demo playback
Overall RUNNING
████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 25% 1 of 4 phases done
Log demo-python/run/logs/run.log (every step: --display detailed)
Target tenacity/wait.py
Phase Coverage phase | phase 2 of 4 | RUNNING
Step Read tests | RUNNING
State Operation Progress Playback
RUNNING Model reads the tests playback 00:00:03
Model use, Coverage phase: model requests 1 of 20, 3,635 input tokens, 33 output tokens
In a demonstration the clock column reads Playback and counts playback time. In a normal run it reads Elapsed and counts the time the run has taken.
The Overall row counts the phases done of all targets together, and
of every repetition when each target runs more than once (--runs), such as
"5 of 8 phases done (2 targets)" or "(2 targets x 3 repetitions)". Here one target
has four phases and the first is done, so it shows 1 of 4. Finishing is the
fourth phase and counts as one, so 100% means that Finishing is done too. Its
row reads "Finishing: measuring the delivered suite and writing the results". In
a run of one target and one repetition, as here, the Overall bar is the only
one: the Target row names the target, and the Phase row its phase, its place
among the phases and its state. With several targets or repetitions, the Target
row counts finished targets and the Phase row the phases of the current target,
each with a bar of its own. The Step row names the step of the phase that is
running, here the coverage phase's Read tests.
The four phases are the starter phase, the coverage phase, the mutation phase and Finishing, as getting started explains. The starter phase's steps are Generate, Measure suites and Consolidation. A loop's steps are Read tests, First measurement, Edit tests, Run tests, Measure and Consolidation. Above the pinned view, a log scrolls with the start and the end of each run, target, phase and step and the outcome of every operation. Count updates stay in the pinned view. Recent failures stay visible while other work continues.
The Log row names the run's plain-text log from the moment it opens, before the
first target starts. Use --display detailed to see every update as it
happens. A path too long for the terminal keeps its end, the log's directories
and file name, behind .... The run's records keep the whole path.
Progress bars and estimates¶
A bar appears wherever Testuary knows how much work there is, except on the Target and Phase rows of a run of one target and one run, which would repeat the Overall count. It shows the percentage, the count and how long that item has run. Percentages are rounded down, so 100% and a full bar mean that all work is done. Work that is counted without a known total shows "total unknown" instead of a bar. Mutation testing counts the mutants run and the caught mutants re-checked, never the mutants caught, so its count, such as "9 of 37 mutants run", has no percentage that could read as a score.
Evenly sized measurement work, such as mutants or timing iterations, also shows an estimate like "about 3 min left" once three items have finished while in view. Model phases never get an estimate, because the time of a model request varies too much. The Overall row shows the total time of the run once it ends.
A model request is one question to the model, and the request limit counts these. A request sent again after a connection error is not counted again. The count reads "model requests 6 of 8", and the requests sent follow only when they differ, as in "model requests 6 of 8 (7 sent, 1 sent again after connection errors)". The detailed view always shows both. A request limit is a ceiling, not a forecast of how many requests a phase will need, so it never appears as a bar. A coverage or mutation goal appears beside the measured value and its basis, for example "Line coverage: 80% (goal 100%) [measured]". Before the value is measured, only the goal is shown. Request counts and their limit belong to the current target and phase and start again with the next one.
Outcomes¶
A target that ends without a suite that meets the minimums is a result, not a failure to execute. It is shown as MINIMUMS NOT MET in a warning color. A target that failed to execute, for example because it crashed or could not be set up, stays FAILED in red.
A loop that ends at its request, token or time limit ends as its settings
say, which is no failure either. The request refused at the limit and the
step and work the loop ends name that limit in yellow, and they are not
among the recent failures that stay visible. A failure in the same loop
still is. The end-of-run summary names each loop that a limit stopped before
its goal in a Warning: line.
Measured zero is different from unavailable, incomplete, unsupported or failed data. Progress values and reused results are labeled as such. Model requests, requests sent, input and output tokens, cache-hit tokens and cost are shown separately where they are known. A value that is not known is not shown as zero. When a failed request has unknown usage, the total token count of the run stays unavailable rather than counting only the requests that succeeded. Reused mutation results, a reused list of mutants and the model service's cache-hit tokens are different things and are not merged into one rate.
Tool rows name what the model does in plain words, such as "Model reads the
tests" or "Tests run", and its outcome, not its arguments or output. The
detailed view adds the tool's own name, such as read_tests. While work runs, metrics that are not measured yet are left out. A
finished operation shows the state of each metric. Processing a saved response
is labeled as recorded work, not as a new model request. Phases that are
already complete show their reused results only after their resume checks.
Prompts and model responses are not part of the terminal view. What the saved logs contain does not depend on the display, and results and logs describes them. The display hides common credential patterns, but that does not replace protecting the run directories and your private configuration.
The check's report¶
testuary check runs the quick check and writes its report in plain words.
The first line says how many problems it found and how far it looked. The
quick check reads the settings and finds the tools. The tool check,
--verify, also starts each tool once and, in a copy of the project, tries its
build where a run builds it and finds its tests without running one, unless the
project has its own build wrapper, which may download its build tool. The
quick check downloads nothing. For a Java project the tool check downloads
what a run would when it is missing: what the build needs and, unless
LANGADAPTER_JAVA_TOOLS_OFFLINE is set, the JUnit launcher, JaCoCo and PIT.
The setup time limit, setup_timeout_seconds, bounds these downloads. Neither check
runs a test or contacts a model. Required parts that only
a run can try do not change the exit status. The first line then says that the
check found no problem in what it tried, never that it found no problems, and
names each language whose adapter has no check of its own, whose tools only a run
tries. Those parts are listed under "not tried yet" with the step to take.
The check ends with an exit
status other than 0 when it found no problem but a required part that the
check itself should have tried was not tried. The first line then says so, and
the next line says how far the check looked.
Then come the project, the targets, the model, the run directory, the phases and the request, token and time limits of each phase. The report says whether an API key is set, but never tries the key. The items follow in groups: found, installed but not tried, not tried yet, missing or blocking with the fix the check reported, optional, notes, and the files that hold more detail. Under "not tried yet", an item listed alone shows the reason the check gave, and on the next line the step to take.
The report ends with one next step:
- With one problem, the fix the check reported.
- With several problems, fixing them and running the same command again.
- After a quick check without problems,
testuary check --verifywith the same options. - After a tool check without problems, a run with
testuary generate.
When no fix was reported, the next step points to the detailed view instead of
guessing one. A refused setting stops the check before it looks at the project,
and the first line says so. No selected model is no refusal: the check reports
it as one problem, with exit status 1, and looks at the project and its tools.
A run directory named after the model alias is left out until a model is
selected.
Any other model problem, such as an alias that models.ini lacks, refuses the
check. --display detailed lists every item by its plain
name with its key, status and reason, also when the output is redirected. Its
status words are the summary's: not tried yet, found (quick check) or works
(tool check), missing, not supported, more than one found and refused.
--json writes only the report as data to standard output, with the
status values of its format.
Getting started shows a complete report.
Interruptions and problems¶
A refused configuration is not a failed run. It can happen before the run directory or any log exists. Correct the reported input, and rely on a saved log only when the report names its path.
Ctrl+C stops the run through its normal cancellation and restores the terminal. Work that already finished is kept. A second Ctrl+C does not cut the cleanup short: the run waits for the measurement or test run in progress to stop and clean up, and prints "Stopping: the work in flight ends and cleans up first, then the run ends." While the stopped run closes what it cancelled, up to 120 seconds and usually a few, further Ctrl+C presses are ignored. The command then prints "Stopped by you." with what you can do next, for example "Continue with the same command and --resume.", and ends with status 130. Results and logs lists every exit status. If the display itself fails, Testuary falls back to plain lines. A failed measurement or a missing required record stays an execution error and is never hidden by the display.
For programs¶
Observing a run¶
Pass event_sink=callback to run_pipeline to receive each observation as a
frozen testuary.presentation.DisplayEvent, including phase transitions after
checkpoints. The callback only observes: its return value does not affect
requests, retries, limits or results. Missing totals and metrics stay unknown
rather than zero. A callback that fails is switched off, and the run continues.
Saving the events to a file¶
--events FILE writes what the display of a command-line run shows to a
file, for programs that show the run elsewhere. The first line names the
format (schema_version 3) and the Testuary version. Each further line is one
JSON object with one record, in the order the display shows them:
{"event": ...} holds a DisplayEvent with its fields, {"readiness": ...}
the setup report before the run, as RunReadiness.as_dict() gives it,
{"plan": ...} what the run will do, a RunPlan with its fields, and
{"completion": ...} the end of the run, a RunCompletion with its fields.
Credentials are removed from every text, as the display removes them. The
file is written line by line, so a program can follow it while the run goes
on. The directory must exist, and an existing file is replaced. With --demo
the file holds the playback's records, with the demonstration's provenance.
If a write fails, the file stops there, one line on standard error says so,
and the run and its display continue.
A program that starts a run with run_pipeline names its event file with
events=PATH, and Testuary writes the same records into it, without a
command that resumes the run. A relative path starts from the program's
current directory. The path is checked before the run, and a call refused
before its run may start writes no file. A program of your own writes the
same format through testuary.presentation.EventFile, with the plan and
completion types it records (the
API reference). The format's version is
testuary.presentation.EVENT_FILE_SCHEMA.
Writing through the presenter¶
When you use TerminalPresenter directly, send your own diagnostic output
through its callback so that the pinned view makes room for it:
from testuary.presentation import TerminalPresenter
with TerminalPresenter("detailed") as presenter:
presenter.write_diagnostic(lambda: print("Preparing selected inputs", flush=True))
The callback runs once, on its own stream, with its own text. The compact view pauses around it and resumes on a new line. Plain and detailed views leave the write alone. Exceptions from the write propagate, including cancellation. The presenter does not hide secrets in or save this output. That stays the caller's logging policy. It does not capture other libraries' output or replace the standard streams. Use detailed or plain output when your writes cannot go through this callback.
Showing a setup report¶
present_readiness(report, presenter=presenter) writes a
testuary.readiness.RunReadiness value within the same presenter. It renders
the report's scope, selected inputs, limits and fixes without checking or
starting anything. Statuses and fixes come only from the report. A set API
key is not a tried key, and says nothing about the provider's quota, context
size or cost. Log paths appear only when the report contains them.
Before its first model request, generation shows the report with
before_run=True. It then opens with "Setup before the run", in the detailed
view too, without a check's name or an exit status, and advises no further
check. It ends with "Generation starts now" only when there is no problem and
the report's exit status is 0, otherwise with the reported fix or a pointer to
the detailed view. A demonstration's playback adds playback=True: it sends no
model request, so the report leaves out the API key and ends with "The
playback starts now".
Showing the end of a run¶
Describe each target in a FileCompletion from
testuary.presentation.terminal. Its outcome is one of complete (a suite
that meets the minimums), no_complete_suite (no suite that meets the
minimums), failed (failed to execute), cancelled (stopped before
finishing), already_complete (a suite that meets the minimums, recorded
earlier) and stopped (ended after the phase that --stop-after-phase
requested). Add the reason, the delivered suite paths,
the final metrics with their basis, the model usage, and the fallbacks
and warnings of its runs, each a sentence. Set
resumable=False for a target that a resume would skip. Collect the
targets in a RunCompletion with the results_directory, the log_path, the
exact resume_command that continues the run and cancelled=True when the run
was stopped.
present_completion(completion, presenter=presenter) writes one headline that
counts the targets with a suite that meets the minimums (including those recorded
earlier), the targets without one, the targets that failed to execute, the targets
stopped before finishing and the targets that ended after the requested phase. One block per target
follows and, when a resume would continue some targets and a command is given,
that command as the next step. Values are
shown as given. Unknown values are left out, never guessed.
Mutation progress¶
Where the engine can observe its mutants, which is the case for local Python
runs, LangAdapter's mutation request reports MutationProgress records through
its progress callback. Show them on the run's clock:
from testuary.presentation.measurement import mutation_progress_event
def show(progress):
presenter(mutation_progress_event(progress, run_id="run-1", target="src/calc.py",
elapsed_s=run_clock()))
The row "Mutation testing" counts the mutants run so far, with a bar
when the total is known and "total unknown" otherwise, and the time estimate
after three mutants. A count is progress, not a result, so it shows no
percentage. No score is shown,
because a mutant's result can still change until mutation testing ends.
The producer ends the mutation-measurement operation after each mutation run
with an observation of its outcome. The next run then starts its own count and
clock.
Warmed Java timing¶
LangAdapter's warmed suite timing (tests.benchmark_warmed, described
in the timing guide of LangAdapter's documentation) reports progress and a result record. Show both with the same
presenter. Here adapter is LangAdapter's adapter for the project and suite
the suite to time.
from langadapter.runs import WarmedSuiteBenchmarkRequest
from testuary.presentation import TerminalPresenter
from testuary.presentation.measurement import warmed_progress_event, warmed_result_event
with TerminalPresenter() as presenter:
def show(progress):
presenter(warmed_progress_event(progress, run_id="timing", target="CounterTest"))
result = adapter.tests.benchmark_warmed(WarmedSuiteBenchmarkRequest(
suite, "native_lifecycle", iteration_timeout_s=60, total_timeout_s=900,
on_progress=show))
presenter(warmed_result_event(result, run_id="timing", target="CounterTest"))
Progress counts finished warmup and measured iterations. An iteration that has only started is not a sample. Single iteration times are shown as raw values, never as a summary. The five summaries (mean of fork means, median of fork medians, smallest and largest sample, standard deviation between forks) appear only for a successful, complete result, exactly as LangAdapter computed them. Otherwise the line states the validity and "no timing summary". One fork has no standard deviation between forks. Durations of zero are marked as limited by the timer's resolution. No comparison or speedup is claimed.