Skip to content

Record cost and turns per arm in the eval baseline - #31

Merged
kevinccbsg merged 1 commit into
mainfrom
evals/cost-and-turns
Oct 26, 2026
Merged

kevinccbsg merged 1 commit into
mainfrom
evals/cost-and-turns

Conversation

@kevinccbsg

Copy link
Copy Markdown
Member

Adds a second baseline table with the mean cost and turns per run, with and without the plugin, for Opus, Sonnet and Haiku, from the runs behind the existing score table.

The score alone hides a skill that reaches the same result at a higher cost: twd-mock-component on Opus scores 1.00 in both arms and costs 57% more with the plugin. The note says where to read costUsd and turns per run in aggregate-result.json.

The score alone hides a skill that reaches the same result at a higher
cost. twd-mock-component on Opus scores 1.00 in both arms and costs 57%
more with the plugin.
@kevinccbsg
kevinccbsg merged commit 22fb214 into main Oct 26, 2026
1 check passed
@kevinccbsg
kevinccbsg deleted the evals/cost-and-turns branch October 26, 2026 10:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant