LobstackHow Lobstack measures whether a model calls tools correctly through the Lobstack API. The test set, the scorer and this page are versioned together; a score is comparable only to another score on the same suite version.
What is tested
80 cases, written by hand for this suite, in six groups:
| Group | Cases | What it tests |
|---|---|---|
| Selection | 14 | One clear request, one right tool, among many offered |
| Arguments | 16 | Line ranges, exact strings, arrays, enums, booleans, bounded integers, quotes, newlines, non-ASCII text, relative dates |
| Near misses | 12 | Two tools that look alike: email or Slack, diff or status, file or URL, issue or pull request, insert or update |
| No call needed | 14 | Small talk, questions the model can answer, missing details it should ask for, requests no offered tool can do |
| Parallel | 12 | Two or three independent calls that belong in one response |
| Chains | 12 | Two to four steps, each using the previous step's result |
Every case has the same system prompt. It names the workspace, the date (2026-10-21) and the time zone (UTC), asks the model to answer in text when no tool is needed, and to make independent calls in the same response. The tools are OpenAI-style function tools with JSON Schema parameters.
The suite is published as JSON at /evals/suite.json.
How a run works
- Each model is called through the Lobstack API (
https://www.lobstack.ai/api/gateway/v1/chat/completions), with the case's messages and tools,temperature: 0, and no other parameters.autois Nex, Lobstack's router, which picks a model per request. - Some models do not accept
temperature. The API drops it and says so inx-lobstack-dropped-params; the results file recordstemperature_ignoredfor that model. - In a chain, after each step the runner appends the model's own tool calls and the case's scripted tool results, then asks again. A chain stops at the first step that fails.
- Retries happen only on transport errors: the request did not complete (network error, or no answer in 120 seconds), or the API answered 429, 502, 503 or 504. Up to 3 retries, 1, 2 and 4 seconds apart. Any other answer is the model's result.
- A case whose request still fails, or that the API answers with another error status, is errored. Errored cases are counted and shown, and left out of every rate below. A model with every case errored has no scores.
Judging one step
A step expects either a call (a list of calls) or no call.
- No call: passes when the response has no tool calls.
- A call: passes when
- the names called equal the names expected, as a multiset (the same tools, the same number of times, nothing extra), and
- every call's arguments parse as a JSON object, and
- the calls can be matched one to one with the expected calls so that every argument check holds.
Argument checks: eq (equal), subset (an object with at least these keys and values, or an array with at least these items), regex (case-insensitive), oneOf (one of these values; null also accepts the argument being left out), contains (case-insensitive substring), present, absent, range (a number in bounds). Checks test what the request asked for, not the model's wording: a Slack message must contain "resolved", not equal a sentence.
The metrics
Every rate is a share from 0 to 100%, over the cases or calls it applies to, ignoring errored cases. A rate with nothing to apply to is shown as a dash.
| Metric | Over | Passes when |
|---|---|---|
| Overall | every case | every step of the case passes |
| Right tool (correct tool selection) | cases with at least one call step | at the case's first call step, the names called equal the names expected |
| Valid arguments (schema-valid on first try) | every call the model made to a tool the case offered | the arguments parse as JSON and validate against that tool's JSON Schema. The runner never asks twice, so every call is a first try |
| Made-up tool (hallucinated or unknown name) | every call the model made | the name is not a tool the case offered. Lower is better |
| Chains (multi-step success) | cases with two or more steps | every step passes |
| Parallel | cases with a step expecting two or more calls | that step passes, with all the calls in one response |
| No call needed (correct abstention) | cases whose first step expects no call | the model made no call |
Also reported per model:
- Latency p50: the median wall time of every API call in the run, measured by the runner, network included.
- Cost per 100 cases: the sum of
x-lobstack-cost-usdover the run's calls, divided by the scored cases, times 100. This is what the calls cost on Lobstack's keys. If any call was unpriced the figure is not shown.
The JSON Schema validator supports type, enum, const, properties, required, additionalProperties, items, minItems, maxItems, minimum, maximum, minLength, maxLength and pattern. Every tool in the suite uses only these, and the build checks it.
What the scores do not say
- They measure tool calling through the Lobstack API, with this prompt and these tools. A different prompt or tool set can score differently.
- At temperature 0 most models are close to deterministic, not exactly. Two runs of the same suite can differ by a case or two.
- Through
auto, Nex picks the model per request, so its score is the score of its choices.
Versions
- 1.0.0 (2026-10-21): first version, 80 cases.
A minor version adds cases or changes a check. A major version removes cases or changes a rule on this page.