Evals · suite v1.0.0 · 80 cases

Which models call tools well.

The same 80 agent tasks, sent to every model through the Lobstack API at temperature 0.

First run pending

Scoreboard

Scores from the latest run.

Sort by any column. Rates leave out cases whose request failed, and the table says how many.

First run pending. Scores appear here after the suite has run against the API. Nothing is estimated in the meantime.

Method

Overall: Every step of the case passed.
Right tool: At the first step that needs a call, exactly the expected tools were called.
Valid arguments: A call’s arguments matched the tool’s JSON Schema, on the first try.
Made-up tool: A call named a tool the case did not offer. Lower is better.
Chains: A two- to four-step task passed at every step.
Parallel: Independent calls arrived together in one response.
No call needed: The model answered or asked instead of calling a tool.