Skip to content

Strengthen the eval suite and record a per-model baseline - #29

Merged
kevinccbsg merged 9 commits into
mainfrom
evals/strengthen-suite
Oct 25, 2026
Merged

kevinccbsg merged 9 commits into
mainfrom
evals/strengthen-suite

Conversation

@kevinccbsg

Copy link
Copy Markdown
Member

What

  • README: a Tested Scenarios table, so plugin users know which scenarios a skill change cannot quietly break.
  • Runs: three runs per case by default. evals/README.md documents a Quick mode (one run, plugin only) and a Full mode (--threshold 0.9, agent and judge models pinned).
  • New cases:
    • twd-not-for-unit-tests: the skill stays out of a Vitest request.
    • twd-mock-component: a camera scanner replaced with twd.mockComponent().
    • twd-run-loop: a filtered --test run before the closing run, and run.json read. It runs against a stand-in twd-cli the scaffold writes, so no browser is needed.
  • Skill fix (1.3.7): twd no longer runs with context: fork. One eval run invoked it with no arguments; the forked skill got no goal, and the main agent then wrote the tests without the skill's rules.
  • Baseline: WITH / Δ per case for Opus 5.5, Sonnet 5.5 and Haiku 5.5, in evals/README.md.

Findings left for follow-up PRs

  1. twd skill: with Sonnet, a green filtered run is where the skill stops, with no closing run and no run.json read. Opus does both every time. The rule needs stating more strongly. twd-run-loop is already red for Sonnet.
  2. Environment: with Node from nvm, npx cannot run inside the eval sandbox. tsc and twd-lint only ever get attempted, never run, and most of the Sonnet and Haiku twd-run-loop misses come from this. Running the evals with a Node installed outside the home directory should remove the noise and the extra grants.
  3. Haiku: after one denied command, Haiku decides the shell is off for the rest of the session, despite the note in the prompt. To decide: harden the skill against this, or accept that Haiku is not a target.

Replace the Supported Frameworks section with the scenarios the plugin
evals check, so users can see which setups a skill change cannot
quietly break.
One run of a non-deterministic agent cannot tell a broken skill from a
bad draw. Cases now use the default three runs; the README documents a
quick one-run mode for skill edits and a full mode with the no-plugin
baseline and a 0.9 threshold for releases and model changes.
A Vitest unit test for a pure function, in a project that also has TWD,
must not invoke the twd skill. The never-invoked grader is scored in
both arms, so the case guards the skill description against grabbing
any testing request.
Document which model plays which role and change one per comparison, so
a score drop points at the skill edit or at the model, not both.
A camera scanner wrapped in MockedComponent: the twd skill has to
replace it with twd.mockComponent before visit, drive the page through
the fake's onScan with the scanner's payload, and clear component mocks
in beforeEach.
Forked, the skill saw only its args. A model that invoked it without
the goal got a setup check back and wrote the tests itself, without the
skill's rules: an eval run put clearComponentMocks in afterEach that
way. Inline, the skill reads the request from the conversation, and the
empty-goal branch shrinks to asking what to test.
A denied exploratory ls led one run to conclude the shell was off and
skip the type check and twd-lint it was allowed to run. The note stays
neutral: naming twd-lint taught the no-plugin arm what the skill adds
and flattened the delta.
The scaffold writes a twd-cli stand-in that honours --test and writes
run.json, so the run loop can be graded without a browser. npx cannot
run inside the eval sandbox when Node comes from nvm, so the commands
grant the stand-in's binary as well, and the README says why tsc and
twd-lint graders only check the attempt.
Grant every local binary so a denied twd-lint does not stop the
run-loop agent before twd-cli, and note what the misses were.
@kevinccbsg
kevinccbsg merged commit ca02220 into main Oct 25, 2026
1 check passed
@kevinccbsg
kevinccbsg deleted the evals/strengthen-suite branch October 25, 2026 23:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant