Skip to content

docs(skills): load less text per run, type-check before the first run - #18

Merged
kevinccbsg merged 3 commits into
mainfrom
docs/skills-token-diet
Oct 12, 2026
Merged

kevinccbsg merged 3 commits into
mainfrom
docs/skills-token-diet

Conversation

@kevinccbsg

@kevinccbsg kevinccbsg commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

Text-only pass on the three skills used most (twd, setup, ci-setup): load only the material a run needs, and catch wrong API calls with the type checker instead of a browser run.

Load less per run

Run Before After
twd, writing a test ~56 KB ~32 KB (−43%)
ci-setup, GitHub Action path ~39 KB ~19 KB (−51%)
setup, Vite project ~29 KB ~24 KB (−16%)
  • twd
    • test-writing.md is now the core (32 KB → 8 KB).
    • Full mock options, URL matching, waitFor, Sinon, and the uncommon interactions and assertions move to test-api.md, read on demand.
    • The philosophy section (already in SKILL.md) and the WRONG/RIGHT blocks the type checker now catches are gone.
  • setup: the non-Vite entry-file blocks and the server-state cache table and scaffolds move to setup/references/, read only when detection finds them.
  • ci-setup
    • ci.md is replaced by ci-setup/references/{custom-workflow,contracts,recording}.md, each read only when that option is chosen.
    • The parts that duplicated the skill (Action template, coverage steps, runtime-environment rationale) are dropped.
    • The old skills/twd/references/ci.md path only resolved from the plugin root.

Type-check before the first run

  • twd runs the project's type checker on the tests it wrote before the first browser run.
  • Verified on twd-shadcn with tsc --noEmit -p tsconfig.app.json, which catches body: for response:, positional mockRequest arguments, headers: and Jest matchers.
  • rule.request.body (typed any) and a missing await are not caught, so those stay as written rules.
  • setup records the command as Type-check command in twd-patterns.md: a typecheck script if present, vue-tsc for Vue, otherwise tsc -p tsconfig.app.json.

Fix

  • Contract strict defaults to true (twd-cli contracts.js); the old table said false.

Plugin version 1.3.2 → 1.3.3.

Validation and evals

  • .github/workflows/validate.yml
    • Runs claude plugin validate --strict on the manifests and skills, with the CLI pinned to 2.1.295 and no credentials.
    • Checks that every references/*.md a skill names exists. validate doesn't look at skill text, and this PR moved several references.
  • evals/: four claude plugin eval cases (twd flow test, twd component test, setup on Vite + React + TanStack Query, ci-setup with recording). Each scaffolds a small project and grades the files the skill writes. See evals/README.md.
    • On this branch all four score 1.00, about $1.50 for one run each.
    • Run locally before a release (~$1.50), not in CI.
    • While building the suite, an Angular case run against the skills from before docs(skills): match the current twd docs #17 scored 0.50 (isDevMode(), no define), so this kind of case does catch regressions. It was swapped for the more common Vite + React + TanStack Query setup.
    • Eval runs refuse writes into .claude/, so the setup case grades the attempted twd-patterns.md write from the trace.

- split test-writing.md into a core reference and test-api.md, read on
  demand; the twd skill's write task loads 43% less text
- twd type-checks the tests it writes before the first browser run;
  setup records the project's type-check command in twd-patterns.md
- setup moves the non-Vite entry-file blocks and the server-state cache
  material into references read only when detected
- ci-setup reads the custom workflow, contracts and recording references
  only when that option is chosen; ci.md moves under ci-setup and its
  duplicates of the skill go away
- contract strict defaults to true, as twd-cli does
- validate.yml runs `claude plugin validate --strict` on the manifests and
  skills, and checks that every references/*.md a skill names exists
- evals/ holds four `claude plugin eval` cases for twd (flow and component
  tests), setup on Angular and ci-setup with recording; each scaffolds a
  project and grades the files the skill writes
…ular

Angular is a residual case; the eval should cover what most projects
run. The new case also exercises the server-state cache reference this
PR moved out of the setup skill.
@kevinccbsg
kevinccbsg merged commit e954270 into main Oct 12, 2026
1 check passed
@kevinccbsg
kevinccbsg deleted the docs/skills-token-diet branch October 12, 2026 17:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant