A prompt profile¶
A prompt profile changes the text Testuary sends to the model. It does not
change tools, limits, validity rules or how suites are measured, so two runs that
differ only in their profile measure the same way.
testuary tutorial walks a prompt
profile on a small project without contacting a model.
Selecting a profile¶
Select a profile with --prompt-profile, TESTUARY_MAIN_PROMPT_PROFILE or
[main] prompt_profile, in that order.
Relative paths says where a
relative path starts. The file is read and validated once, when the command
starts, and kept for the whole run. A profile with an empty phases object
leaves every prompt unchanged:
[main]
prompt_profile = profile.json
{"format": 1, "name": "unchanged", "phases": {}}
Checking a profile¶
testuary check checks the selected profile, for example
testuary check --prompt-profile profile.json --model ALIAS --source-files FILE.
A missing file, an unknown component, section heading or tool message, or
an invalid placeholder refuses the command with exit status 2 before any
work. The refusal of a wrong section heading, tool message name or
placeholder lists the allowed ones: the phase's ## headings, such as
## WORKFLOW and ## Hard rules, and its tool messages, as its bundled
prompt names them. The report of a check that passes does not name the
profile, and the check does not show the changed prompts. A check refuses a
run directory that is not empty, so after a run give it a new
--output-dir.
A run records the profile under prompt_profile in each phase's
archive/<phase>/generation/record.json, with a copy of it and the parts
it changed in changes
(generation records).
No file of a run holds the system prompt as sent: the phase's chat log holds
its user prompt and the conversation. testuary.hints.request_text returns
the guidance hints add.
The format¶
A profile is a JSON file with "format": 1, an optional name and a phases
object. Each of initial (the starter phase), coverage and mutation
can change these components:
system_promptanduser_prompt_template, the whole text, withappendorreplace,system_prompt_sections, named sections of the system prompt, withappend,replaceorremove, whereremovetakestrue,tool_messages, the text of existing tool messages, withappendorreplace. A profile cannot add tools or change their descriptions.
The text of an operation is a string or a list of lines.
A section is named by its heading line as the bundled prompt spells it,
such as ## Hard rules. The initial phase has no sections and no tool
messages, so only its system_prompt and user_prompt_template can change.
A replace of the whole system prompt cannot be combined with section
operations.
This profile appends one sentence to the system prompt of every phase:
{
"format": 1,
"name": "boundary-tests",
"phases": {
"initial": {
"system_prompt": {"append": "Exercise empty, boundary and error inputs."}
},
"coverage": {
"system_prompt": {"append": "Use explicit expected values."}
},
"mutation": {
"system_prompt": {"append": "Keep assertions tied to observable behavior."}
}
}
}
This profile changes the coverage phase's sections and one of its tool
messages. It appends a sentence to the section ## Hard rules, removes the
section ## Test libraries available and appends a sentence to the tool
message edit_not_found:
{
"format": 1,
"name": "independent-tests",
"phases": {
"coverage": {
"system_prompt_sections": {
"## Hard rules": {"append": "Keep every test independent of the others."},
"## Test libraries available": {"remove": true}
},
"tool_messages": {
"edit_not_found": {"append": "Read the test file before the next edit."}
}
}
}
}
A per_language entry in a phase changes a component for one language. For
that language it replaces the general change to the same component, or to
the same section or tool message, so a language entry must repeat any
general text it keeps. A language that LangAdapter has not registered, built
in or added by an extension, is refused. The
example that testuary init --examples copies,
testuary-examples/language-profile.json, which Testuary's tests resolve
unchanged, does this for Python and Java:
{
"format": 1,
"name": "example-language-guidance",
"phases": {
"coverage": {
"system_prompt": {"append": "Cover each branch with one focused test."},
"per_language": {
"python": {
"system_prompt": {"append": "Cover each branch with one focused test. Use pytest.raises for expected exceptions."}
},
"java": {
"system_prompt": {"append": "Cover each branch with one focused test. Use assertThrows for expected exceptions."}
}
}
}
}
}
Profile operations run before the template placeholders are filled and before
hints are added. Keep the placeholders the bundled text uses: a profile that
drops one is not refused, and the model then gets the text without that
value. A profile may use the placeholders of the component it changes and
the language's placeholders, such as $lang_name. Any other placeholder
refuses the run, and $$ writes a literal dollar sign.
Comparing profiles¶
To compare profiles, use a separate run directory for each profile and keep every other setting the same. Testuary records which prompts each phase used, with a copy of the profile and hints files. A resume or reuse with a different profile is refused rather than silently generating again, and saved model responses keep the usage recorded when they were made.