Concepts
Tool calling
Utmost supports OpenAI-style function tools on chat completions, function tools on Responses, and Anthropic tools on Messages. This page uses chat completions.
Check capabilities.tools in GET /v1/models. With tools attached, the router only picks models that support them. A pinned model is not checked; the request goes to it as sent.
Defining tools
{
"model": "auto",
"messages": [{ "role": "user", "content": "Is CVE-2024-3094 in xz 5.6.1?" }],
"tools": [
{
"type": "function",
"function": {
"name": "lookup_cve",
"description": "Look up a CVE and the affected versions.",
"parameters": {
"type": "object",
"properties": { "id": { "type": "string" } },
"required": ["id"]
},
"strict": true
}
}
]
}typemust befunction.name: 1 to 64 letters, digits,_or-.parameters: a JSON Schema object.- Up to 128 tools per request.
Tools count toward input tokens and toward the router's difficulty estimate (any tools add one point; see Routing).
tool_choice
| Value | Meaning |
|---|---|
"auto" |
The model decides. |
"none" |
No tool calls. |
"required" |
The model must call at least one tool. |
{"type": "function", "function": {"name": "lookup_cve"}} |
The model must call this tool. |
parallel_tool_calls: false asks for at most one call per turn. Both settings are passed to the backend; how strictly they are honoured depends on the model.
The response
{
"choices": [{
"index": 0,
"message": {
"role": "assistant",
"content": null,
"tool_calls": [{
"id": "call_a5fcd44539fe088d",
"type": "function",
"function": { "name": "lookup_cve", "arguments": "{\"id\":\"CVE-2024-3094\"}" }
}]
},
"finish_reason": "tool_calls"
}]
}arguments is a JSON string. Parse it, and validate it before acting on it.
Sending results back
Append the assistant message with its tool_calls, then one tool message per call:
{
"model": "auto",
"tools": [ ... ],
"messages": [
{ "role": "user", "content": "Is CVE-2024-3094 in xz 5.6.1?" },
{ "role": "assistant", "content": null, "tool_calls": [
{ "id": "call_a5fcd44539fe088d", "type": "function", "function": { "name": "lookup_cve", "arguments": "{\"id\":\"CVE-2024-3094\"}" } }
]},
{ "role": "tool", "tool_call_id": "call_a5fcd44539fe088d", "content": "{\"affected\": [\"5.6.0\", \"5.6.1\"]}" }
]
}With an auto alias, each turn is routed on its own, so turns of one conversation can run on different models. Pin a model if that matters.
Parallel calls
A model may return several entries in tool_calls. Answer each with its own tool message, matching tool_call_id, in any order.
Streaming
Tool calls stream as deltas on choices[].delta.tool_calls. Each delta has an index. The first delta for an index carries id, type and function.name. Later deltas carry pieces of function.arguments to append.
data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":"call_a5fc...","type":"function","function":{"name":"lookup_cve","arguments":""}}]},"finish_reason":null}], ...}
data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\"id\":\"CVE-"}}]},"finish_reason":null}], ...}
data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"2024-3094\"}"}}]},"finish_reason":null}], ...}
data: {"choices":[{"index":0,"delta":{},"finish_reason":"tool_calls"}], ...}Assemble by index:
import OpenAI from "openai";
const client = new OpenAI({ apiKey: process.env.UTMOST_API_KEY, baseURL: "https://api.utmost.site/v1" });
const stream = await client.chat.completions.create({ model: "auto", stream: true, tools, messages });
const calls: { id: string; name: string; arguments: string }[] = [];
for await (const chunk of stream) {
for (const d of chunk.choices[0]?.delta?.tool_calls ?? []) {
const c = (calls[d.index] ??= { id: "", name: "", arguments: "" });
if (d.id) c.id = d.id;
if (d.function?.name) c.name = d.function.name;
c.arguments += d.function?.arguments ?? "";
}
}
console.log(calls.map((c) => ({ ...c, arguments: JSON.parse(c.arguments) })));calls = {}
for chunk in client.chat.completions.create(model="auto", stream=True, tools=tools, messages=messages):
if not chunk.choices:
continue
for d in chunk.choices[0].delta.tool_calls or []:
c = calls.setdefault(d.index, {"id": "", "name": "", "arguments": ""})
c["id"] = d.id or c["id"]
if d.function and d.function.name:
c["name"] = d.function.name
c["arguments"] += (d.function.arguments or "") if d.function else ""Notes
- The refusal retry never runs on a response that contains tool calls.
- Tool results are ordinary input. They are checked against the hard limits like any other content.