Repository navigation
Strengthen the eval suite and record a per-model baseline - #29
Merged
Merged
Conversation
Replace the Supported Frameworks section with the scenarios the plugin evals check, so users can see which setups a skill change cannot quietly break.
One run of a non-deterministic agent cannot tell a broken skill from a bad draw. Cases now use the default three runs; the README documents a quick one-run mode for skill edits and a full mode with the no-plugin baseline and a 0.9 threshold for releases and model changes.
A Vitest unit test for a pure function, in a project that also has TWD, must not invoke the twd skill. The never-invoked grader is scored in both arms, so the case guards the skill description against grabbing any testing request.
Document which model plays which role and change one per comparison, so a score drop points at the skill edit or at the model, not both.
A camera scanner wrapped in MockedComponent: the twd skill has to replace it with twd.mockComponent before visit, drive the page through the fake's onScan with the scanner's payload, and clear component mocks in beforeEach.
Forked, the skill saw only its args. A model that invoked it without the goal got a setup check back and wrote the tests itself, without the skill's rules: an eval run put clearComponentMocks in afterEach that way. Inline, the skill reads the request from the conversation, and the empty-goal branch shrinks to asking what to test.
A denied exploratory ls led one run to conclude the shell was off and skip the type check and twd-lint it was allowed to run. The note stays neutral: naming twd-lint taught the no-plugin arm what the skill adds and flattened the delta.
The scaffold writes a twd-cli stand-in that honours --test and writes run.json, so the run loop can be graded without a browser. npx cannot run inside the eval sandbox when Node comes from nvm, so the commands grant the stand-in's binary as well, and the README says why tsc and twd-lint graders only check the attempt.
Grant every local binary so a denied twd-lint does not stop the run-loop agent before twd-cli, and note what the misses were.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
evals/README.mddocuments a Quick mode (one run, plugin only) and a Full mode (--threshold 0.9, agent and judge models pinned).twd-not-for-unit-tests: the skill stays out of a Vitest request.twd-mock-component: a camera scanner replaced withtwd.mockComponent().twd-run-loop: a filtered--testrun before the closing run, andrun.jsonread. It runs against a stand-intwd-clithe scaffold writes, so no browser is needed.twdno longer runs withcontext: fork. One eval run invoked it with no arguments; the forked skill got no goal, and the main agent then wrote the tests without the skill's rules.evals/README.md.Findings left for follow-up PRs
run.jsonread. Opus does both every time. The rule needs stating more strongly.twd-run-loopis already red for Sonnet.npxcannot run inside the eval sandbox. tsc and twd-lint only ever get attempted, never run, and most of the Sonnet and Haikutwd-run-loopmisses come from this. Running the evals with a Node installed outside the home directory should remove the noise and the extra grants.