Skip to content

Your own model client

A model service that speaks the OpenAI chat protocol needs only a model alias in models.ini in your user directory, as the models guide shows. When the model needs a different client library, add a client factory: a Python function, named by chat_model, that returns the chat model Testuary sends its requests through.

Client factories

The chat model a client factory returns is a LangChain chat model. The model alias names the factory with chat_model = ITEM:factory. ITEM is a Python file, such as my_chat.py, or the name of an installed module, such as my_package.chat. A file needs no installation. Relative paths says where a relative file name starts. The text before the last colon is the item, so a Windows path keeps its drive letter.

chat_model selects the protocol added, and protocol = added states it. Any other protocol together with chat_model is refused. A chat_model in a service's section, or in its variable such as TESTUARY_OPENAI_CHAT_MODEL, applies to every alias of that service.

The provider = field names the model service, not the factory. For a service that is not one of the kinds Testuary configures, ollama, openai and vllm, name a section of its own as the provider:

[my_service]
api_url = https://service.example/api

[my_alias]
provider = my_service
model = my-model
api_key_env = MY_SERVICE_API_KEY
chat_model = my_chat.py:factory

The fields of models.ini explains api_url, api_key_env and the other fields.

Testuary loads the item once and calls the function with these keyword arguments: model, base_url, api_key, timeout in seconds and max_retries=0, then temperature, top_p, seed (the sampling seed of a starter suite) and max_tokens (the output-token limit in effect) when they are set. api_key is an empty string when no key is configured. Each key of the alias's request_options arrives as a keyword argument of its own. request_options cannot set a keyword that Testuary passes itself.

The returned model must:

  • be a LangChain BaseChatModel,
  • stream its answers for the starter suites,
  • support bind_tools for the coverage phase and the mutation phase, which pass only a list of LangChain tools, and return a runnable model,
  • accept system messages after the first one, and tool messages,
  • report each answer's token counts in usage_metadata, in a streamed answer too, since Testuary counts the tokens from it,
  • send no request while it is created or bound to tools.

Testuary passes max_retries=0 and keeps it that way, because Testuary itself retries failed requests and counts them. Reasoning settings for a client factory's model go in the alias's request_options, not in the built-in reasoning controls.

Each client serves one target and one phase, and each starter suite has its own client. A request sent again after a connection error reuses the same client. Do not rely on state shared between clients, or on keywords given when tools are bound overriding those given when the client is created. Each repetition's settings record, resolved_settings.json, holds the factory's file as extension_file and its SHA-256 as extension_sha256, and for an installed module the version of its package as extension_version, which Testuary reads from the package's installed metadata. A resume accepts saved work only while these are unchanged, and its refusal names the client factory's file that changed. An installed module goes into the Python environment Testuary runs in, not into the virtual environment of the project under test.

What a request carries and expects

A starter request carries no tools. Its answer holds the test file in a fenced code block. Testuary takes the last block fenced with the language's tag, such as python, else the last block between plain fences. An answer that ends before the closing fence holds no test file.

A request of the coverage phase or the mutation phase carries the tools of the current step and the phase's result tool, emit_coverage_test_result or emit_mutation_test_result, which every step offers. Each tool carries its description and its argument schema. The first request offers read_tests, the second the phase's measure tool, measure_coverage or measure_mutation. Later requests offer read_source, read_tests and edit_tests, and after an edit run_tests as well. After a passing run of the edited tests, they offer read_source, read_tests and the measure tool, without edit_tests and run_tests, until the measurement. Only when that passing run carried a STANDS ALONE note is edit_tests offered with the measure tool, so that the tests the note names can be fixed first. Such a note names tests that leave the module under test or a library setting changed, or that fail in reverse order.

read_tests and read_source take start_line and end_line. edit_tests takes old_text and new_text, or edits, a list of such pairs, each with an optional replace_all. run_tests and the measure tool take no arguments. The model answers with tool calls. Testuary runs each call and sends its result back as a tool message. A call of a tool the step does not offer is answered with a tool message that says what makes the tool available, and the phase continues. Each request ends with a system message that states where the phase stands.

The phase's final answer is a call of its result tool. Its required arguments are test_file_name, explanation and targeted_functions in the coverage phase, and mutation_test_file_name, explanation and targeted_mutants in the mutation phase. test_file_name and mutation_test_file_name hold the name of the working test file, as the phase's first prompt names it, such as test_sign.py. Testuary ends the phase itself once a measurement reaches the phase's goal and extra_measurements_after_goal further measurements have run, or, with allow_early_stop, which is on by default, after max_measurements_without_improvement measurements without improvement. At the goal, the end waits while the latest passing run carries a STANDS ALONE note. The tool message of the ending measurement asks for the result call, and the call is accepted. An earlier result call is accepted only when the last test run passed, the current tests are measured, and the goal is met or, with allow_early_stop, the measurements stopped improving. Otherwise the call is refused: a tool message answers the call, and the reason follows as a human message. With allow_early_stop, once max_final_answers_without_measurement result calls in a row came without a new measurement between them, the phase ends at the last of them. When Testuary ends the phase at the goal, after the stall or at this cap, the phase's result is the one Testuary builds from its own records, not the call's arguments. The settings reference lists these settings. A reply without a tool call counts as a final answer that is not usable: Testuary tells the model why, and the phase continues within its request limit.

The example

testuary init --examples copies a complete client factory into testuary-examples. Testuary's tests resolve these files unchanged.

The factory, testuary-examples/testuary_example_chat.py, wraps the OpenAI-compatible client:

"""A client factory for Testuary in one file.

The alias's ``chat_model = testuary_example_chat.py:factory`` names this file
and its ``factory``. Testuary loads the file and calls ``factory`` with the
resolved constructor parameters and request options as keyword arguments,
including ``max_retries=0``: Testuary owns retries and their accounting. The
factory returns a LangChain chat model that streams and can bind tools. This
example wraps the OpenAI-compatible client. A different client maps the
parameters itself.
"""
from langchain_openai import ChatOpenAI


def factory(**parameters):
    return ChatOpenAI(stream_usage=True, **parameters)

The alias in testuary-examples/models.ini names the factory's file, which lies beside it:

# A model alias served through the example's client factory, the file
# testuary_example_chat.py beside this file. It needs no installation.
# Set LOCAL_MODEL_API_KEY before running, even for a local endpoint that
# accepts any key. Keep keys out of files.

[example_added]
provider = openai
model = qwen3-coder:latest
api_url = http://localhost:11434/v1
api_key_env = LOCAL_MODEL_API_KEY
chat_model = testuary_example_chat.py:factory
protocol = added

Set the key variable the alias names, LOCAL_MODEL_API_KEY, even for a local endpoint that accepts any key. Then run the quick check. controls.ini, also in the examples directory, selects the alias example_added, the example prompt profile and the hints. --home . makes Testuary read the examples' models.ini in place of the one in your user directory. Run the command in testuary-examples. --no-project-settings keeps a project's settings file from applying when that directory lies inside a project:

export LOCAL_MODEL_API_KEY=<key>
testuary check --home . --config controls.ini --no-project-settings --source-dir /path/to/project \
  --python-executable /path/to/project-env/bin/python --source-files package/module.py

In PowerShell, set the variable with $env:LOCAL_MODEL_API_KEY = "<key>", write the command on one line and name the interpreter as C:\path\to\project-env\Scripts\python.exe.

The quick check loads the factory's file, so the file's top-level code runs, and reports a file or module that cannot be found, fails to load or lacks the factory. Its default report does not name the factory. With --display detailed, each phase's model settings item reads Selected: added, and its reason says that the client factory imports. Neither the quick check nor testuary check --verify calls the factory, creates a client or contacts the model.

Trying the client in a run

A run creates the clients. Before any phase runs, it calls the factory for each phase the run generates in, each phase up to --stop-after-phase that it does not reuse, and binds the tools of the coverage phase and the mutation phase among them. A client that cannot be created or bound refuses the run with exit status 2 before any request is sent. The refusal does not show the factory's own error, since that error can hold the key. Call the factory yourself in Python, with the same keyword arguments, to see it.

To try the client without a model service, set the alias's api_url to a local server you control, which reaches the factory as base_url, or let the factory return a model with scripted answers. This factory returns a scripted model for a target sign.py whose function sign(x) returns -1 below zero and 1 otherwise. It answers a starter request with a test of a positive value. In the coverage phase it chooses its call by the tools a step offers: it reads the tests, measures, adds a test of a negative value with one edit, runs the tests and measures again. That measurement reaches the goal, and its result call is accepted.

from langchain_core.language_models import BaseChatModel
from langchain_core.messages import AIMessage
from langchain_core.outputs import ChatGeneration, ChatResult

FENCE = "`" * 3
STARTER = f"{FENCE}python\nfrom sign import sign\n\n\ndef test_positive():\n    assert sign(2) == 1\n{FENCE}"
EDIT = {"old_text": "def test_positive():",
        "new_text": "def test_negative():\n    assert sign(-2) == -1\n\n\ndef test_positive():"}
RESULT = {"test_file_name": "test_sign.py", "explanation": "Covers both signs.",
          "targeted_functions": ["sign"]}
USAGE = {"input_tokens": 0, "output_tokens": 0, "total_tokens": 0}


class ScriptedChat(BaseChatModel):
    edited: bool = False

    @property
    def _llm_type(self):
        return "scripted"

    def bind_tools(self, tools, **kwargs):
        return self.bind(tools=tools, **kwargs)

    def _generate(self, messages, stop=None, run_manager=None, tools=(), **kwargs):
        offered = {tool.name for tool in tools}
        call = None
        if "run_tests" in offered or "measure_coverage" in offered:
            call = ("run_tests" if "run_tests" in offered else "measure_coverage", {})
        elif "edit_tests" in offered and not self.edited:
            self.edited, call = True, ("edit_tests", EDIT)
        elif offered == {"read_tests", "emit_coverage_test_result"}:
            call = ("read_tests", {})
        elif offered:
            call = ("emit_coverage_test_result", RESULT)
        calls = [{"name": call[0], "args": call[1], "id": f"call_{len(messages)}"}] if call else []
        message = AIMessage("" if call else STARTER, tool_calls=calls, usage_metadata=USAGE)
        return ChatResult(generations=[ChatGeneration(message=message)])


def factory(**parameters):
    return ScriptedChat()

It serves the starter phase and the coverage phase only. The example defines no _stream: LangChain's stream then yields the whole answer of _generate as one piece. A factory for a different client library translates the keyword arguments itself.

Save the example as scripted_chat.py beside your models.ini. The alias still needs an address, which the scripted factory ignores, and no key variable:

[scripted_service]
api_url = http://127.0.0.1:9/scripted

[my_alias]
provider = scripted_service
model = scripted
chat_model = scripted_chat.py:factory

A short trial run generates one starter suite and skips the mutation measurement at the phase ends. Put this in the project's .testuary/project.ini:

[main]
phase_end_mutation = false

[starter]
suite_count = 1

Then run this in the project's directory, with your alias, interpreter and target:

testuary generate --model my_alias --stop-after-phase coverage \
  --python-executable /path/to/project-env/bin/python --source-files package/module.py

When [project] source_files and [project] python_executable in the settings file name the target and the interpreter, the two options can be left out. The run sends the starter phase's and the coverage phase's requests through your client and stops after the coverage phase. It ends with exit status 0 and reports the target as ended after the requested phase, with MINIMUMS NOT MET on its last line. A second trial needs a new --output-dir: a run directory that holds results is refused, and a resume is refused once the factory's file changed.