diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json index 388c059..681d232 100644 --- a/.claude-plugin/plugin.json +++ b/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "twd", - "version": "1.3.7", + "version": "1.4.0", "description": "TWD (Test While Developing) — deterministic in-browser testing plugin for Claude Code. Writes, runs, and fixes component/page-level browser tests for SPAs.", "author": { "name": "BRIKEV", diff --git a/README.md b/README.md index 38143c5..6ba6dc2 100644 --- a/README.md +++ b/README.md @@ -65,110 +65,6 @@ The agent follows a 5-phase workflow: 4. **Run & Fix** — probes the dev server, runs tests headlessly via twd-cli, isolates failures with `--test`, fixes them, closes with a full-suite run 5. **Report** — summarizes coverage and results -### `/twd:test-flow-gallery` Command - -Reads TWD test files and generates visual Mermaid flowcharts with plain-language summaries — living documentation auto-generated from real tests. - -``` -/twd:test-flow-gallery -``` - -What it does: -- Reads all `*.twd.test.{ts,tsx,js}` files in the project -- Generates a `.flows.md` file next to each test file with Mermaid diagrams -- Creates a `test-flow-gallery.md` index at the project root -- Each `it()` block gets a plain-language summary + color-coded flowchart - -Example output per test: - -``` -## Items Page - -**What this tests:** A user navigates to the items page and sees a list of -all available items. The page loads data from the API and displays each item -as a list entry. -``` - -Followed by a Mermaid flowchart showing the user flow (purple visit nodes, blue actions, green assertions) and API layer (orange request/response subgraphs). - -### `/twd:test-gaps` Command - -Scans project routes and cross-references against TWD test files to find untested pages, classify risk, and generate a prioritized gap report. - -``` -/twd:test-gaps -``` - -What it does: -- Detects your framework and reads the route config (React Router, Vue Router, Angular, Next.js, Nuxt, SolidJS) -- Discovers all TWD test files and extracts tested routes -- Identifies UNTESTED and PARTIALLY TESTED pages -- Classifies risk: HIGH (mutations, payments), MEDIUM (auth, complex loading), LOW (static, read-only) -- Outputs a prioritized "Start Here" top-5 list - -Example output: - -``` -## Summary -- Pages/routes discovered: 22 -- Pages with tests: 15 -- Pages partially tested: 2 -- Pages untested: 5 - -## UNTESTED Pages — HIGH Risk -| Page | Path | Why High Risk | -|-------------|-------------------|----------------------------------| -| Checkout | /checkout | Handles payments — direct impact | -| User Settings | /settings/account | Has delete account flow | -``` - -If no TWD tests are found, the skill detects other test frameworks (Playwright, Jest, etc.) and explains how TWD complements them. - -### `/twd:test-quality` Command - -Reads TWD test files, evaluates quality across four dimensions, assigns letter grades (A/B/C/D), and generates an improvement report. - -``` -/twd:test-quality -``` - -What it does: -- Grades each test file across 4 dimensions: - - **Journey Coverage** (35%) — complete user flows vs visibility-only checks - - **Interaction Depth** (20%) — variety of `userEvent` types (click, type, keyboard, etc.) - - **Assertion Quality** (25%) — `be.visible` (weak) to `deep.equal` on API payloads (strong) - - **Error & Edge Cases** (20%) — error states, empty states, cancel flows -- Assigns a final letter grade per file and an overall project grade -- Generates specific grade-up suggestions per file -- Ranks improvements by impact - -Example output: - -``` -# TWD Test Quality Report -Files analyzed: 8 -Overall grade: C - -## Grade Distribution -| Grade | Count | Files | -|-------|-------|------------------------------------------| -| A | 2 | user-list.twd.test.ts, order-list.twd.test.ts | -| D | 2 | payment-list.twd.test.ts, item-create.twd.test.ts | - -### payment-list.twd.test.ts — Grade D -| Dimension | Grade | Notes | -|------------------|-------|------------------------------------| -| Journey Coverage | D | 1 test, visibility check only | -| Interaction Depth| D | No userEvent calls | -| Assertion Quality| D | Only be.visible and greaterThan(1) | -| Error & Edge Cases| D | Only happy path | - -**To reach grade C:** Add a search test with userEvent.type + URL assertion. -Add column header assertions with have.text. -``` - -Works best alongside `/twd:test-gaps` — gaps tells you **what's untested**, quality tells you **if existing tests are any good**. - ## Tested Scenarios The skills are checked with [plugin evals](https://code.claude.com/docs/en/plugin-evals): each scenario scaffolds a small project, runs the skill without a human in the loop, and grades the files it writes. A scenario listed here is one a skill change cannot quietly break. diff --git a/skills/test-flow-gallery/SKILL.md b/skills/test-flow-gallery/SKILL.md deleted file mode 100644 index 4e594e5..0000000 --- a/skills/test-flow-gallery/SKILL.md +++ /dev/null @@ -1,316 +0,0 @@ ---- -name: test-flow-gallery -description: Reads TWD test files and generates visual Mermaid flowcharts with plain-language summaries as living documentation. -disable-model-invocation: true -allowed-tools: [Read, Write, Glob, Grep] ---- - -# Test Flow Gallery - -You are generating a **Test Flow Gallery** — visual Mermaid flowcharts with plain-language summaries from TWD test files. The output serves as living documentation for PMs, QA, and developers, auto-generated from real tests. - -You are **framework-agnostic** — you read TWD test files regardless of whether the app uses React, Vue, Next.js, Nuxt, or any other framework. TWD tests describe user behavior, not framework internals. - -## Workflow - -### Step 1 — Discover test files - -Parse the argument passed to this skill: - -- **Specific file** (ends with `.twd.test.ts`, `.twd.test.tsx` or `.twd.test.js`): use that file only -- **Directory**: glob `*.twd.test.{ts,tsx,js}` within that directory -- **No argument**: glob `**/*.twd.test.{ts,tsx,js}` from the project root - -If no test files are found, tell the user and stop. - -### Step 2 — Read all test files - -Read each discovered test file in full. - -### Step 3 — Generate per-file markdown - -For each test file, produce a `.flows.md` file **co-located next to the test file**. - -Naming: strip the `.twd.test.ts`, `.twd.test.tsx` or `.twd.test.js` extension and append `.flows.md`. -- `checkout.twd.test.ts` → `checkout.flows.md` -- `add-dialog.twd.test.tsx` → `add-dialog.flows.md` -- `user-dashboard.twd.test.js` → `user-dashboard.flows.md` - -Each `it()` block gets its own section with: -1. A plain-language summary (2-4 sentences, business terms, no jargon) -2. A Mermaid flowchart following the interpretation rules below - -Group `it()` blocks under their parent `describe` name as a shared `##` heading. - -If a `describe` has only one `it()`, collapse to a single section — no redundant nesting. - -If a `describe` has multiple `it()` blocks, each gets its own `###` subsection with summary + diagram. - -### Step 4 — Generate index - -Create `test-flow-gallery.md` at the **project root** with the index template (see Output Templates below). - -### Step 5 — Report - -Print to the user: -- Paths to all generated `.flows.md` files -- Path to the index file -- Total count of flows diagrammed - ---- - -## Interpretation Rules - -Map TWD test commands to semantic diagram nodes. - -### User Flow Nodes - -| TWD Command | Node Type | Mermaid Shape | Class | -|---|---|---|---| -| `twd.visit('/route')` | Page entry | Stadium `([...])` | `visit` | -| `render()` (Testing Library) | Component entry — label it with the component name | Stadium `([...])` | `visit` | -| `userEvent.click()` | User action | Rectangle `[...]` | `action` | -| `userEvent.dblClick()` | User action | Rectangle `[...]` | `action` | -| `userEvent.type()` | User input | Rectangle `[...]` | `action` | -| `userEvent.clear()` | User action | Rectangle `[...]` | `action` | -| `userEvent.selectOptions()` | User selection | Rectangle `[...]` | `action` | -| `userEvent.keyboard()` | Keyboard action | Rectangle `[...]` | `action` | - -### Assertion Nodes - -| TWD Command | Node Type | Mermaid Shape | Class | -|---|---|---|---| -| `twd.should(..., "be.visible")` | Positive assertion | Hexagon `{{...}}` | `assert` | -| `twd.should(..., "be.disabled")` | Disabled/blocked | Hexagon `{{...}}` | `disabled` | -| `twd.should(..., "have.attr")` | State assertion | Hexagon `{{...}}` | `assert` | -| `twd.should(..., "have.text")` | Text assertion | Hexagon `{{...}}` | `assert` | -| `twd.should(..., "have.value")` | Value assertion | Hexagon `{{...}}` | `assert` | -| `twd.should(..., "have.class")` | State assertion | Hexagon `{{...}}` | `assert` | -| `twd.should(..., "be.checked")` | State assertion | Hexagon `{{...}}` | `assert` | -| `twd.should(..., "not.be.visible")` | Absence assertion | Hexagon `{{...}}` | `disabled` | -| `expect()` | Verification | Hexagon `{{...}}` | `assert` | -| `expect(...).to.be.null` | Absence check | Hexagon `{{...}}` | `disabled` | -| `screenDom.findByRole()` | Async wait + assert | Hexagon `{{...}}` | `assert` | -| `screenDom.findByText()` | Async wait + assert | Hexagon `{{...}}` | `assert` | -| `screen.findBy*()` (component tests) | Async wait + assert | Hexagon `{{...}}` | `assert` | -| `twd.notExists()` | Element gone assertion | Hexagon `{{...}}` | `disabled` | -| `twd.url().should(...)` | URL assertion | Hexagon `{{...}}` | `assert` | -| Warning/alert assertions | Warning | Hexagon `{{...}}` | `warning` | - -> **Note:** Element method-style assertions (`el.should("have.text", ...)`) follow the same mapping as `twd.should()`. No special treatment needed. - -### API Nodes - -| TWD Command | Node Type | Mermaid Shape | Class | -|---|---|---|---| -| `twd.mockRequest()` GET | API read | Rectangle `[...]` | `api` | -| `twd.mockRequest()` POST/PUT/PATCH/DELETE | API mutation | Rectangle `[...]` | `api` | -| API response | Response | Cylinder `[("...")]` | `response` | - -Example: `R1["GET /api/items"]:::api --> R2[("200 - 2 items")]:::response` - -### Ignore List (do not diagram) - -These items should NOT appear as nodes in the diagram: - -- `import` statements -- `beforeEach` / `afterEach` setup boilerplate — **except** `visit()` and `mockRequest()` inside them. If `beforeEach` contains `visit()` or `mockRequest()`, prepend those nodes to every `it()` diagram in the same `describe` scope. Treat them as implicit first steps of each flow. -- Variable declarations and helper function definitions -- `twd.clearRequestMockRules()` / `twd.clearComponentMocks()` -- Test utility setup (language, feature flags, user mocks) — mention in summary text instead -- `userEvent.setup()` — boilerplate, not a meaningful user action -- `cleanup()` / `restorePage()` / `componentHost()` — component test plumbing -- `Sinon.stub()` / `Sinon.restore()` — module stubbing setup, mention in summary text instead -- `twd.mockComponent()` / `twd.clearComponentMocks()` — component mocking setup, mention in summary text instead -- `twd.waitForRequest()` / `twd.waitForRequests()` — internal plumbing, not user-visible flow -- `twd.wait(ms)` — timing detail, not user-visible flow - ---- - -## Mermaid Conventions - -### Node IDs - -- User flow nodes: `A`, `B`, `C`... (sequential) -- API nodes: `R1`, `R2`, `R3`... (sequential) - -### Styling (classDef) - -Always include ALL of these classDef declarations at the bottom of every diagram: - -``` -classDef visit fill:#7c3aed,stroke:#5b21b6,color:#fff,font-weight:bold -classDef action fill:#2563eb,stroke:#1d4ed8,color:#fff -classDef assert fill:#16a34a,stroke:#15803d,color:#fff,font-weight:bold -classDef api fill:#ea580c,stroke:#c2410c,color:#fff -classDef response fill:#f3f4f6,stroke:#9ca3af,color:#374151 -classDef disabled fill:#dc2626,stroke:#b91c1c,color:#fff,font-weight:bold -classDef warning fill:#d97706,stroke:#b45309,color:#fff,font-weight:bold -``` - -### Edge Types - -- **User flow**: `A --> B` (solid arrow) -- **API side effects**: `E -.-> R1` (dotted arrow, from user flow node to API subgraph) -- **API chain**: `R1 --> R2` (solid arrow, within API subgraph) - -### Emojis in Labels - -Use these emojis as prefixes in node labels: - -- 🔗 Page visit -- 🖱️ Click action -- ⌨️ Type/input action -- ✅ Positive assertion -- ❌ Disabled/blocked/absent assertion -- ⚠️ Warning assertion -- 🌐 API subgraph title - -### Mermaid Syntax Gotchas (critical for valid output) - -- NO em dashes (`—`) in subgraph names — use hyphens (`-`) -- ALL text inside `{{}}` nodes MUST be wrapped in double quotes: `{{"✅ text here"}}` -- NO `[/ /]` trapezoid syntax with URLs — use `[""]` rectangles instead -- NO raw `&` in URLs — simplify or rephrase -- Subgraph IDs must be unique across the entire diagram (e.g., `API1["🌐 API - Page Load"]`) -- Use `direction LR` inside API subgraphs for horizontal layout - ---- - -## Output Templates - -### Per-file `.flows.md` - -```markdown -# {Module Name} - Test Flow Gallery - -> Auto-generated from TWD test suites. Do not edit manually — regenerate with `/twd:test-flow-gallery`. -> -> **Legend:** 🔗 Page visit | 🖱️ Click | ⌨️ Input | ✅ Pass | ❌ Blocked/Absent | ⚠️ Warning | 🌐 API - ---- - -## {Describe Block Name} - -### {it() block name} - -**What this tests:** {2-4 sentence plain-language summary in business terms. No code references, no jargon. Focus on what the user does, what they see, why it matters. Mention edge cases and business rules being validated.} - -```mermaid -flowchart TD - %% nodes and edges - %% classDef styles -``` - ---- -``` - -### Index file (`test-flow-gallery.md` at project root) - -```markdown -# Test Flow Gallery - Index - -> Auto-generated from TWD test suites. Regenerate with `/twd:test-flow-gallery`. - -| Test Suite | Flow File | Flows | -|---|---|---| -| {top-level describe name} | [{filename}.flows.md](relative/path/to/file.flows.md) | {number of it() blocks} | -``` - ---- - -## Granularity Rules - -- **One diagram per `it()` block** (not per `describe`) -- If a `describe` has only one `it()`, collapse to a single section -- If a `describe` has multiple `it()` blocks, each gets its own section with summary + diagram -- Group all `it()` blocks under a shared heading matching the `describe` name - ---- - -## Full Example - -### Input test - -```typescript -describe("Items Page", () => { - beforeEach(() => { - twd.clearRequestMockRules(); - twd.clearComponentMocks(); - }); - - it("should load and display items", async () => { - await twd.mockRequest("getItems", { - method: "GET", - url: "/api/items", - response: [{ id: 1, name: "Item One" }, { id: 2, name: "Item Two" }], - status: 200, - }); - - await twd.visit("/items"); - await twd.waitForRequest("getItems"); - - const heading = await screenDom.findByRole("heading", { name: "Items" }); - twd.should(heading, "be.visible"); - const items = await screenDom.findAllByRole("listitem"); - expect(items).to.have.length(2); - }); -}); -``` - -### Expected output - -Since the `describe` has only one `it()`, collapse to a single section: - ---- - -## Items Page - -**What this tests:** A user navigates to the items page and sees a list of all available items. The page loads data from the API and displays each item as a list entry. This verifies the basic data-fetching and rendering pipeline for the items listing. - -```mermaid -flowchart TD - A(["🔗 Visit /items"]):::visit - B{{"✅ Heading 'Items' is visible"}}:::assert - C{{"✅ 2 list items displayed"}}:::assert - - A --> B --> C - - A -.-> R1 - - subgraph API1["🌐 API - Page Load"] - direction LR - R1["GET /api/items"]:::api --> R2[("200 - 2 items")]:::response - end - - classDef visit fill:#7c3aed,stroke:#5b21b6,color:#fff,font-weight:bold - classDef action fill:#2563eb,stroke:#1d4ed8,color:#fff - classDef assert fill:#16a34a,stroke:#15803d,color:#fff,font-weight:bold - classDef api fill:#ea580c,stroke:#c2410c,color:#fff - classDef response fill:#f3f4f6,stroke:#9ca3af,color:#374151 - classDef disabled fill:#dc2626,stroke:#b91c1c,color:#fff,font-weight:bold - classDef warning fill:#d97706,stroke:#b45309,color:#fff,font-weight:bold -``` - ---- - -**Key observations from this example:** -- `beforeEach` boilerplate (`clearRequestMockRules`, `clearComponentMocks`) is ignored — not diagrammed -- `twd.waitForRequest("getItems")` is ignored — internal plumbing -- `twd.mockRequest()` becomes an API subgraph with request + response nodes -- `twd.visit()` is the entry point with a dotted edge to the API subgraph -- Assertions become green hexagon nodes in the user flow -- All 7 `classDef` styles are included even though not all are used in this diagram - ---- - -## Summary Writing Guidelines - -When writing the "What this tests" summary: - -- Write 2-4 sentences in plain business language -- Describe what the **user** does and sees, not what the code does -- No code references, function names, or technical jargon -- Focus on: the user journey, the expected outcome, and why it matters -- Mention edge cases and business rules being validated -- If `beforeEach` contains Sinon stubs, module mocks, or feature flag setup, mention these in the summary (e.g., "The test runs with authentication disabled to focus on the page behavior") diff --git a/skills/test-gaps/SKILL.md b/skills/test-gaps/SKILL.md deleted file mode 100644 index 51f4aba..0000000 --- a/skills/test-gaps/SKILL.md +++ /dev/null @@ -1,304 +0,0 @@ ---- -name: test-gaps -description: Scans project routes and cross-references against TWD test files to find untested pages, classify risk, and generate a prioritized gap report. -disable-model-invocation: true -allowed-tools: [Read, Glob, Grep] ---- - -# Test Gap Analysis - -You are performing a **Test Gap Analysis** — scanning a project to find all pages/routes and cross-referencing them against existing TWD test files. Your output is a prioritized markdown report answering: **which pages have no tests, and which matter most?** - -You are **framework-agnostic** — you work with any project that uses TWD tests, regardless of whether the app uses React, Vue, Angular, Next.js, Nuxt, SolidJS, or any other framework. TWD tests describe user behavior, not framework internals. - -## Invocation - -``` -/twd:test-gaps -``` - -No flags. No arguments. Auto-discover everything and output the full report to console. Do not accept or parse any flags. - -## Workflow - -### Step 1 — Pre-flight: Detect TWD presence - -Glob for `**/*.twd.test.{ts,tsx,js}`. - -**If zero TWD tests found → STOP.** Do not generate a report. Instead: - -- Read `package.json` and scan for other test frameworks: `playwright`, `@playwright/test`, `cypress`, `jest`, `vitest`, `@testing-library/react`, `@testing-library/vue` -- If other frameworks found: output the gate message below -- If no tests at all: output the no-tests gate message below -- Either way, stop. No gap report without TWD tests. - -**Gate output when other frameworks detected:** - -``` -## No TWD Tests Found - -I found {framework} tests in this project, but no TWD test files (*.twd.test.{ts,tsx}). - -TWD complements {framework} by adding deterministic, in-browser page-level tests -that run inside your Vite dev server — fast feedback during development with -mocked APIs, no flaky network calls. - -**To get started:** run `/twd:setup` to install TWD, then `/twd` to write your -first tests. - -No gap report generated — TWD tests are needed for analysis. -``` - -**Gate output when no tests at all:** - -``` -## No Tests Found - -This project has no TWD tests and no other test framework detected. - -Shipping without tests means bugs reach users undetected. Pages with forms, -payments, or access control are especially risky — a single untested mutation -can corrupt data silently. - -**To get started:** run `/twd:setup` to install TWD, then `/twd` to write -tests for your highest-risk pages first. -``` - -### Step 2 — Discover all routes - -Use three strategies in priority order. Try Strategy A first. If it yields no routes, try B. If B yields nothing, fall back to C. - -**Strategy A: Framework route config (preferred)** - -Detect framework from `package.json` dependencies, then find and read the route config: - -| Framework | Typical route file | What to extract | -|---|---|---| -| React Router | `routes.tsx`, `AppRoutes.tsx`, `router.tsx` | `path`, `element`/`component` | -| Vue Router | `router/index.ts`, `router.ts` | `path`, `component` | -| Angular | `app-routing.module.ts`, `app.routes.ts` | `path`, `component` | -| SolidJS | `routes.tsx` | `path`, `component` | -| Next.js (App Router) | `app/**/page.tsx` | directory structure = routes | -| Nuxt | `pages/**/*.vue` | file structure = routes | - -For file-based routing (Next.js App Router, Nuxt), the directory/file structure IS the route config — glob for page files and derive routes from paths. - -**Strategy B: Page component glob (fallback)** - -If no framework route config is found, glob for page components in common patterns: -- `src/pages/**/*.{tsx,vue,svelte}` -- `app/**/page.{tsx,jsx,ts,js}` -- `pages/**/*.{tsx,vue,svelte}` - -Each file represents a likely route. Derive the route path from the file path. - -**Strategy C: TWD tests only (last resort)** - -If neither A nor B yields results, use only TWD test `twd.visit()` URLs as the known route universe. Add a prominent warning banner to the report: - -``` -> ⚠️ **Route discovery: TWD tests only.** No route config or page components found. -> This report shows tested routes but CANNOT identify untested pages. -> Consider adding a route config or organizing pages in a standard directory structure. -``` - -In this mode, the UNTESTED section is omitted entirely since there's no route universe to compare against. The PARTIALLY TESTED section still works (visit-only detection). - -### Step 3 — Discover tested routes - -Read all TWD test files found in Step 1. For each file, extract: - -- **All `twd.visit()` calls** — URL strings (string literals, template literals, same-file constant references). Include `visit()` calls inside `beforeEach` blocks — these apply to every `it()` in the same `describe` scope. -- **All `twd.mockRequest()` calls** — method + URL for each mock (including those in `beforeEach`) -- **Interaction depth** — does the test have `userEvent.*` calls (click, type, clear, selectOptions, keyboard, dblClick) or is it visit-only? - -**Component tests cover no route.** A file that calls Testing Library's `render()` and never `twd.visit()` tests a component in isolation. It adds nothing to the tested-route set — do not credit the page that uses the component. Mention the components it covers in that page's "What's Covered" cell when the page is otherwise partially tested. - -**URL extraction rules:** - -| Pattern | How to extract | -|---|---| -| `twd.visit("/checkout")` | Literal string `/checkout` | -| `twd.visit(\`/orders/${id}\`)` | Replace `${...}` with `:param` → `/orders/:param` | -| `twd.visit(CHECKOUT_URL)` | Trace `const CHECKOUT_URL = "..."` in same file | -| Variable from import | Cannot resolve — note as "unresolved visit URL" | - -### Step 4 — Match and classify - -**4a. Match routes to tests** - -| Strategy | How it works | -|---|---| -| Exact URL match | Strip base path from `visit()` URLs, compare to route paths | -| Parameterized match | Replace `${var}` in visit URLs with `:param` wildcard, match route patterns | -| File naming fallback | Match test directory/file name to route segment (e.g., `checkout/` → `/checkout`) | - -**4b. Classify coverage status** - -- **TESTED** — route has a matching test file with `twd.visit()` and `userEvent.*` interactions -- **PARTIALLY TESTED** — route has tests but: - - Tests only visit the page with no `userEvent.*` interactions (visit-only) - - Route component has forms/mutations but no test mocks POST/PUT/PATCH/DELETE requests -- **UNTESTED** — route has zero matching test files - -### Step 5 — Risk classification - -For each UNTESTED and PARTIALLY TESTED route, read the component file to classify risk. - -**HIGH — Test this first** - -Any of these is true: -- **Has mutations** — forms, POST/PUT/PATCH/DELETE API calls, data creation/updates/deletes -- **Handles money** — payments, refunds, billing, subscriptions, pricing -- **Write-permission-gated** — elevated permissions for destructive actions - -**MEDIUM — Test this soon** - -Any of these is true: -- **Has access control** — permission-gated, role-based, auth-required -- **Complex data loading** — multiple API calls, deferred/streaming data -- **Detail page with multiple views** — tabs, modals, conditional sections -- **Custom caching/revalidation** — non-standard data refresh behavior - -**LOW — Nice to have** - -- **Read-only display** — simple data display, no interactions -- **Error/utility pages** — 404, 403, maintenance -- **Static pages** — no data fetching, no user input - -### Step 6 — Generate report - -Output markdown to console using the output template below. Omit empty sections (e.g., if no HIGH risk gaps, skip that subsection). If all routes are covered, congratulate the team and note any partial coverage that could be improved. - -**"Start Here" ranking:** HIGH risk first, then PARTIAL over UNTESTED at the same risk level, then by mutation density (more form actions = higher priority within the same tier). - -**Monorepo note:** v1 assumes a single-app project. If the repo appears to be a monorepo (multiple `package.json` files, `apps/` or `packages/` directory), note this in the report and analyze only the root-level app or ask the user which app to scan. - ---- - -## Output Template - -```markdown -# Test Gap Analysis Report -Generated: {date} - -## Summary -- **Pages/routes discovered:** {N} -- **Pages with tests:** {N} -- **Pages partially tested:** {N} -- **Pages untested:** {N} -- **Route discovery method:** {framework route config / page component glob / TWD tests only} - -## UNTESTED Pages - -### HIGH Risk -| Page | Path | Why High Risk | -|------|------|---------------| -| ... | ... | ... | - -### MEDIUM Risk -| Page | Path | Why Medium Risk | -|------|------|-----------------| -| ... | ... | ... | - -### LOW Risk -| Page | Path | Why Low Risk | -|------|------|--------------| -| ... | ... | ... | - -## PARTIALLY TESTED Pages -| Page | Path | What's Covered | What's Missing | -|------|------|----------------|----------------| -| ... | ... | ... | ... | - -## Start Here (Top 5 Priorities) -1. **{Page}** — {reason this is #1} -2. ... - ---- - -> **Tip:** For large codebases, the Serena MCP server can reduce token usage -> during gap analysis by enabling symbol-level navigation instead of full file reads. -``` - ---- - -## Partial Coverage Detection - -A route is PARTIALLY TESTED when: - -| Signal | What it means | -|---|---| -| Test has `twd.visit()` but zero `userEvent.*` calls | Visit-only test — page loads but no interactions verified | -| Route component has forms/submit handlers but test has no POST/PUT/PATCH/DELETE mocks | Mutations exist but are untested | - -This is deliberately simple. Full quality analysis (happy-path-only detection, error state coverage, multi-view coverage) is out of scope. - ---- - -## Full Example - -### Example output - -```markdown -# Test Gap Analysis Report -Generated: 2026-03-17 - -## Summary -- **Pages/routes discovered:** 22 -- **Pages with tests:** 15 -- **Pages partially tested:** 2 -- **Pages untested:** 5 -- **Route discovery method:** React Router config (src/routes.tsx) - -## UNTESTED Pages - -### HIGH Risk -| Page | Path | Why High Risk | -|------|------|---------------| -| User Settings | /settings/account | Has form mutations (update profile, change password, delete account) | -| Admin Role Create | /admin/roles/create | Creates RBAC roles affecting permissions for all users | -| Checkout | /checkout | Handles payments — direct financial impact | - -### MEDIUM Risk -| Page | Path | Why Medium Risk | -|------|------|-----------------| -| Billing Detail | /billing/:id | Financial data display with nested detail views | -| Product Config | /products/:id/config | Detail page with multiple tabs and conditional sections | - -## PARTIALLY TESTED Pages -| Page | Path | What's Covered | What's Missing | -|------|------|----------------|----------------| -| Order Detail | /orders/:id | Page loads, data displays | No userEvent interactions — forms and buttons untested | -| Dashboard | / | Card links verified | Page has POST /api/dashboard/settings mock but no form interaction tests | - -## Start Here (Top 5 Priorities) -1. **Checkout** — Handles payments. Zero tests. A bug here costs real money. -2. **Admin Role Create** — Creates RBAC roles. Zero tests. A bug affects access for all users. -3. **User Settings** — Has delete account flow. Zero tests. Untested destructive action. -4. **Order Detail** — Most visited detail page. Has tests but no interaction coverage. -5. **Billing Detail** — Financial data. Zero tests. Nested route may have edge cases. - ---- - -> **Tip:** For large codebases, the Serena MCP server can reduce token usage -> during gap analysis by enabling symbol-level navigation instead of full file reads. -``` - -**Key observations from this example:** -- Route discovery used React Router config — the most reliable strategy -- 5 untested pages found, classified by risk (3 HIGH, 2 MEDIUM, 0 LOW) -- 2 partially tested pages — one is visit-only, one has mocks but no form interactions -- "Start Here" priorities rank HIGH risk first, then by mutation density -- Empty sections (LOW risk) are omitted -- Serena tip included at the bottom - ---- - -## Limitations - -1. **TWD only.** Only scans `*.twd.test.{ts,tsx,js}` files. Tests in other frameworks (Vitest, Playwright, Cypress, Jest) are invisible to the gap analysis (but detected during pre-flight). -2. **Heuristic route matching.** Route-to-test matching is best-effort. Cannot trace runtime navigation or dynamic route registration. -3. **No quality analysis.** A route with a single `be.visible` assertion shows as TESTED same as one with 20 thorough tests. Partial detection is limited to visit-only and missing mutation mocks. -4. **Variable resolution is same-file only.** Cannot follow imported URL constants across modules. -5. **Risk classification is heuristic.** Based on reading component code — may miss business context (e.g., a "simple" page that's actually critical for compliance). diff --git a/skills/test-quality/SKILL.md b/skills/test-quality/SKILL.md deleted file mode 100644 index 6c4d098..0000000 --- a/skills/test-quality/SKILL.md +++ /dev/null @@ -1,361 +0,0 @@ ---- -name: test-quality -description: Reads TWD test files, evaluates quality across four dimensions (journeys, interactions, assertions, error handling), assigns letter grades, and generates an improvement report. -disable-model-invocation: true -allowed-tools: [Read, Glob, Grep] ---- - -# Test Quality Scoring - -You are evaluating **TWD test quality** — reading existing TWD test files, grading each across four weighted dimensions, and producing a report explaining what each file does well and what it's missing. - -This skill answers: **"Are the existing tests any good?"** — not just whether tests exist, but whether they exercise meaningful user journeys, verify real outcomes, and handle failures. - -You are **framework-agnostic** — you evaluate TWD test quality regardless of the app's framework. All scoring is based on TWD API usage patterns, not framework-specific code. This is **static analysis only** — you read test source code, you do not execute tests. - -## Invocation - -``` -/twd:test-quality -``` - -No flags. No arguments. Auto-discover all TWD test files and output the full report to console. Do not accept or parse any flags. - -## Workflow - -### Step 1 — Pre-flight: Detect TWD presence - -Glob for `**/*.twd.test.{ts,tsx,js}`. - -**If zero TWD tests found → STOP.** Do not generate a report. Instead: - -- Read `package.json` and scan for other test frameworks: `playwright`, `@playwright/test`, `cypress`, `jest`, `vitest`, `@testing-library/react`, `@testing-library/vue` -- If other frameworks found: output the gate message below -- If no tests at all: output the no-tests gate message below -- Either way, stop. No quality report without TWD tests. - -**Gate output when other frameworks detected:** - -``` -## No TWD Tests Found - -I found {framework} tests in this project, but no TWD test files (*.twd.test.{ts,tsx}). - -TWD complements {framework} by adding deterministic, in-browser page-level tests -that run inside your Vite dev server — fast feedback during development with -mocked APIs, no flaky network calls. - -**To get started:** run `/twd:setup` to install TWD, then `/twd` to write your -first tests. - -No quality report generated — TWD tests are needed for analysis. -``` - -**Gate output when no tests at all:** - -``` -## No Tests Found - -This project has no TWD tests and no other test framework detected. - -Shipping without tests means bugs reach users undetected. Pages with forms, -payments, or access control are especially risky — a single untested mutation -can corrupt data silently. - -**To get started:** run `/twd:setup` to install TWD, then `/twd` to write -tests for your highest-risk pages first. -``` - -### Step 2 — Read all test files - -Read each discovered TWD test file in full. For each file, analyze: - -- All `it()` blocks and their content -- All `twd.visit()` calls (including in `beforeEach` — these apply to every `it()` in the same `describe` scope) -- All `userEvent.*` calls — types and frequency -- All `twd.should()` and `expect()` assertions — what they verify -- All `twd.mockRequest()` calls — methods and status codes -- `beforeEach`/`afterEach` setup patterns -- Testing Library `render()` calls — a file that renders and never calls `twd.visit()` is a **component test file** (see "Component Test Files" below) - -### Step 3 — Grade each file - -Apply the four scoring dimensions (see below) to each file. Assign a letter grade (A/B/C/D) per dimension, then compute the final grade using the weighted average formula. - -### Step 4 — Generate suggestions - -For each file below grade A, generate 2-3 specific, actionable suggestions to reach the next grade. Suggestions must: - -- Reference the actual file content (specific endpoints, UI elements, mock data) -- Be actionable ("add a test that..." not "improve assertions") -- Target the weakest dimensions first -- Not reference patterns from other files in the project (keep it self-contained) - -### Step 5 — Generate report - -Output markdown to console using the output template below. Omit empty grade distribution rows. If all files are grade A, congratulate the team and skip the Improvements section. - ---- - -## Scoring Dimensions - -Each test file is graded A/B/C/D across four dimensions. The final grade is a weighted average. - -### Dimension 1: Journey Coverage (35% weight) - -Measures whether tests walk through complete user journeys (visit → interact → assert outcome) or merely check that elements are visible. - -| Grade | Criteria | -|---|---| -| **A** | 3+ complete journeys with distinct user goals (e.g., create, search, delete) | -| **B** | 2 complete journeys, or 1 journey + significant breadth across UI sections | -| **C** | 1 complete journey covering a single happy path | -| **D** | Only visibility/rendering checks, no interaction chains | - -**What counts as a "complete journey":** A test that calls `twd.visit()`, performs at least one `userEvent` interaction (click, type, select, clear), and ends with a meaningful assertion (API payload, URL change, state transition, or specific DOM content — not just `be.visible`). - -**Detection signals:** -- Count `twd.visit()` calls and `userEvent.*` calls per `it()` block -- A journey has: visit + at least 1 userEvent + at least 1 non-visibility assertion -- Pure visibility tests: only `twd.should(el, "be.visible")` with no prior interaction - -### Dimension 2: Interaction Depth (20% weight) - -Measures the variety and realism of user interactions tested. - -| Grade | Criteria | -|---|---| -| **A** | 4+ distinct interaction types (click, type, clear, keyboard, selectOptions), form submissions | -| **B** | 3 interaction types, or 2 types used across multiple flows | -| **C** | 2 interaction types (typically click + type) | -| **D** | Only `userEvent.click()` or no `userEvent` calls at all | - -**Interaction types tracked:** -- `userEvent.click` — basic click -- `userEvent.type` — text input, search fields -- `userEvent.clear` — clearing input fields (tests overwrite behavior) -- `userEvent.keyboard` — keyboard navigation, enter/escape key handling -- `userEvent.selectOptions` — dropdown/select interactions -- `userEvent.dblClick` — double click -- Form submission via button click + `twd.waitForRequest()` on a POST/PUT/DELETE — full submit cycle - -### Dimension 3: Assertion Quality (25% weight) - -Measures whether assertions verify meaningful outcomes or just check that something is visible. - -| Grade | Criteria | -|---|---| -| **A** | Verifies API request payloads (`deep.equal`), URL changes (`url().should()`), and specific attribute values | -| **B** | Verifies specific text content (`have.text`), attribute values (`have.attr`), exact counts (`have.length`), form values (`have.value`) | -| **C** | Mix of `be.visible` and specific text checks, but no payload or state verification | -| **D** | Only `be.visible` and loose count assertions like `greaterThan(1)` | - -**Assertion hierarchy (weakest → strongest):** -1. `be.visible` — element exists and is shown (weakest) -2. `greaterThan(N)` — loose count check -3. `have.length(N)` — exact count -4. `have.text("exact text")` — specific content -5. `have.value("exact value")` — form state -6. `have.attr("href", "/path")` — link/attribute verification -7. `be.disabled` / `not.be.disabled` — interactive state -8. `expect(el).to.be.null` — absence verification -9. `twd.url().should("contain.url", "...")` — URL state verification -10. `expect(rule.request).to.deep.equal(...)` — API payload verification (strongest) - -The grade is determined by the **highest level used in 2+ test blocks** across the file, not by a single isolated occurrence. Look at the overall distribution of assertion types — a file that mostly uses `be.visible` with one `deep.equal` is still grade C/D, not A. - -### Dimension 4: Error & Edge Case Coverage (20% weight) - -Measures whether tests handle failure scenarios and boundary conditions. - -| Grade | Criteria | -|---|---| -| **A** | Tests API errors (400/500), empty states, and cancel/abort flows | -| **B** | Tests 2 of: error responses, empty states, cancel flows | -| **C** | Tests 1 edge case | -| **D** | Only happy path with status 200, single data shape | - -**Detection signals:** -- `status: 400` or `status: 500` in `mockRequest` → error state testing -- `response: { results: [] }` or `count: 0` or empty array → empty state testing -- `expect(el).to.be.null` after an action → verifying removal/hiding -- Dialog cancel flow: open → Cancel → verify closed -- Multiple mock data shapes for the same endpoint - ---- - -## Component Test Files - -A file that calls Testing Library's `render()` and never `twd.visit()` tests one component in isolation. Grade it on the same four dimensions and weights, with `render()` standing in for `twd.visit()`: - -- **Journey Coverage** — a complete journey is `render()` + at least one `userEvent` interaction + a non-visibility assertion. A dialog opened, filled and submitted counts; rendering and checking `be.visible` does not. "Distinct user goals" are the component's behaviours (validation, open/close, sort), not app pages. -- **Interaction Depth, Assertion Quality, Error & Edge Cases** — unchanged. Queries use `screen` instead of `screenDom`; read them the same way. - -Do not mark a component test file down for having no `twd.visit()` or no URL assertion — it has no route to navigate. - -Check its setup and put any problem first in the file's suggestions, whatever the grade: - -- `render()` without `{ container: componentHost() }` — the component lands after the app and queries find duplicates -- `cleanup()` and `restorePage()` missing from `afterEach` (or placed in `beforeEach`) — renders stack up, and the app stays detached for later flow tests -- `screenDom` queries against the rendered component — they cannot find it -- Sinon stubs of the project's own hooks or context — component tests use the real providers and mock only the network - -In the report, tag the file heading `— Grade X (component)` so readers can tell the two kinds apart. - ---- - -## Final Grade Calculation - -Convert dimension grades to numbers (A=4, B=3, C=2, D=1), compute weighted average, map back to letter: - -| Weighted Average | Final Grade | -|---|---| -| 3.5 or higher | **A** | -| 2.5 – 3.49 | **B** | -| 1.5 – 2.49 | **C** | -| Below 1.5 | **D** | - -The overall project grade is the average of all file grades. - ---- - -## Output Template - -```markdown -# TWD Test Quality Report -Generated: {date} -Files analyzed: {N} -Overall grade: {X} - -## Grade Distribution -| Grade | Count | Files | -|---|---|---| -| A | N | file1, file2 | -| B | N | ... | -| C | N | ... | -| D | N | ... | - -## Per-File Breakdown - -### {filename} — Grade {X} -| Dimension | Grade | Notes | -|---|---|---| -| Journey Coverage | {A-D} | ... | -| Interaction Depth | {A-D} | ... | -| Assertion Quality | {A-D} | ... | -| Error & Edge Cases | {A-D} | ... | - -**To reach grade {next}:** {2-3 specific suggestions referencing this file's content} - ---- - -## Improvements -Ranked by impact (lowest-graded files first): -1. **{file}** (Grade D): {what to improve and why it matters} -2. ... - ---- - -> Run `/twd:test-gaps` to find pages with no tests at all. -``` - ---- - -## Grade-Up Suggestions - -For each file below grade A, generate 2-3 specific suggestions to reach the next grade. - -**Suggestion patterns by dimension:** - -Journey Coverage suggestions: -- "Add a test that completes the [action] flow: [specific steps based on what the page does]" -- "Add a test for [user goal] — the page has [element] but no test exercises it" - -Interaction Depth suggestions: -- "Add `userEvent.clear(input)` before typing to test overwrite behavior" -- "Add `userEvent.keyboard('[Escape]')` to test [modal/dropdown] dismissal" -- "Add `userEvent.selectOptions` for the [dropdown element] on this page" - -Assertion Quality suggestions: -- "Replace `greaterThan(1)` with exact `have.length(N)` for [element]" -- "Add `expect(rule.request).to.deep.equal({...})` to verify the [POST/PUT] payload" -- "Add `twd.url().should('contain.url', '...')` to verify URL updates on [action]" - -Error & Edge Case suggestions: -- "Add a test mocking `status: 500` on `[endpoint]` and verify the error state" -- "Add a test with empty response for `[endpoint]` and verify the empty state message" -- "Add a cancel flow test: open [dialog/form] → click Cancel → verify it closes" - ---- - -## Full Example - -### Example output - -```markdown -# TWD Test Quality Report -Generated: 2026-03-17 -Files analyzed: 8 -Overall grade: C - -## Grade Distribution -| Grade | Count | Files | -|---|---|---| -| A | 2 | user-list.twd.test.ts, order-list.twd.test.ts | -| B | 2 | dashboard.twd.test.ts, item-detail.twd.test.ts | -| C | 2 | product-config.twd.test.ts, billing.twd.test.ts | -| D | 2 | payment-list.twd.test.ts, item-create.twd.test.ts | - ---- - -### user-list.twd.test.ts — Grade A -| Dimension | Grade | Notes | -|---|---|---| -| Journey Coverage | A | 7 tests: table display, search with debounce, column toggle, actions menu, delete flow, pagination, empty state | -| Interaction Depth | A | click, type, keyboard (Enter/Escape), selectOptions for column selector | -| Assertion Quality | A | Column headers by index, cell text values, URL contains query param, deep.equal on DELETE payload | -| Error & Edge Cases | B | Empty search response tested, delete confirmation; missing: API 500 on delete | - -**To reach A across all dimensions:** Add a test mocking `status: 500` on the `DELETE /api/users/:id` endpoint and verify the error state is displayed. - ---- - -### payment-list.twd.test.ts — Grade D -| Dimension | Grade | Notes | -|---|---|---| -| Journey Coverage | D | 1 test — visits page, checks heading is visible, counts rows with greaterThan(1) | -| Interaction Depth | D | No userEvent calls at all | -| Assertion Quality | D | Only be.visible and greaterThan(1) — weakest assertions | -| Error & Edge Cases | D | Only happy path with status 200 | - -**To reach grade C:** Add a test that searches the payment list — `userEvent.type` in the search field, verify URL updates with `twd.url().should()`. Add column header assertions with `have.text` to verify specific content instead of just visibility. - ---- - -## Improvements -1. **payment-list.twd.test.ts** (Grade D): Core financial module with only a visibility check. Add search interaction, column header verification, and row data assertions. -2. **item-create.twd.test.ts** (Grade D): Creation flow has no form interaction tests. Add fill form → submit → verify POST payload flow. -3. **product-config.twd.test.ts** (Grade C): Multi-tab page only tests first tab. Add tab switching with click + content verification for each tab. -4. **billing.twd.test.ts** (Grade C): Financial data display — add error state test (mock 500) and empty state test. - ---- - -> Run `/twd:test-gaps` to find pages with no tests at all. -``` - -**Key observations from this example:** -- Grade A file still has one B dimension — the per-dimension breakdown shows exactly where -- Grade D file gets specific suggestions referencing its actual content (search field, column headers) -- Improvements section ranks D files first, then C files -- No time estimates — just what to do and why -- Cross-reference to test-gaps at the bottom - ---- - -## Limitations - -1. **Static analysis only.** Scoring is based on reading test source code. No test execution. Cannot detect runtime issues like flaky assertions or timing problems. -2. **Heuristic grading.** Letter grades are AI interpretation of the criteria, not deterministic computation. Two runs may produce slightly different notes, but grades should be consistent for the same file. -3. **TWD only.** Only scores `*.twd.test.{ts,tsx,js}` files. Tests in other frameworks are invisible. -4. **No cross-file analysis.** Each file is scored independently. Cannot detect project-wide patterns like "all files lack error tests." -5. **No permission/access dimension in v1.** RBAC testing quality is not scored. If the project uses permission mocking, this is noted in the Journey Coverage notes but doesn't have its own dimension.