Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ twd-relay is a WebSocket relay that lets AI agents and external tools trigger an

**Relay Server** (`src/relay/`, exported as `twd-relay`) — A WebSocket server that attaches to an HTTP server. It manages exactly one browser connection and many client connections. Clients send commands (`run`, `status`); the relay forwards them to the browser. The browser sends test lifecycle events (`test:start`, `test:pass`, `test:fail`, `run:complete`, etc.); the relay broadcasts them to all clients. A `runInProgress` lock prevents concurrent test runs.

**Browser Client** (`src/browser/`, exported as `twd-relay/browser`) — Runs in the browser. Connects to the relay, listens for commands, dynamically imports `twd-js/runner` to execute tests, and streams results back. Uses native browser `WebSocket` with auto-reconnect. Reads test state from `window.__TWD_STATE__` (set by twd-js). A small `faviconManager` (in `src/browser/faviconManager.ts`) sets a colored favicon + `document.title` prefix based on connection/run state so the active TWD tab is identifiable among multiple tabs to the same origin. A sibling `runMonitor` (in `src/browser/runMonitor.ts`) tracks per-test wall-clock time; on the 3 s heartbeat tick AND at the end of every test, the browser checks whether the test exceeded `maxTestDurationMs` (default 10 s) and, if so, emits a `run:aborted` event so the CLI can exit with a clear error instead of hanging on a throttled tab.
**Browser Client** (`src/browser/`, exported as `twd-relay/browser`) — Runs in the browser. Connects to the relay, listens for commands, dynamically imports `twd-js/runner` to execute tests, and streams results back. Uses native browser `WebSocket` with auto-reconnect. Reads test state from `window.__TWD_STATE__` (set by twd-js). A small `faviconManager` (in `src/browser/faviconManager.ts`) sets a colored favicon + `document.title` prefix based on connection/run state so the active TWD tab is identifiable among multiple tabs to the same origin. A sibling `runMonitor` (in `src/browser/runMonitor.ts`) tracks per-test wall-clock time; on the 3 s heartbeat tick AND at the end of every test, the browser checks whether the test exceeded `maxTestDurationMs` (default 10 s) and, if so, emits a `run:aborted` event so the CLI can exit with a clear error instead of hanging on a throttled tab. A third helper, `diagnostics.ts`, renders the failure-diagnostics block twd-js attaches to a failed test and composes it above the error message on `test:fail`.

**Vite Plugin** (`src/vite/`, exported as `twd-relay/vite`) — A Vite plugin that hooks into `configureServer` to attach the relay to the dev server's HTTP instance, and (in `apply: 'serve'` mode) auto-injects a `<script type="module">` that imports `createBrowserClient` and calls `.connect()` via a virtual module (`virtual:twd-relay/connect`). Auto-injection is on by default; set `autoConnect: false` to opt out and wire `createBrowserClient` manually (required for non-Vite consumers). Both the relay-server path and the injected client path use the same formula (`options.path ?? base + '/__twd/ws'`); `configResolved` sets the `base` for both.

Expand Down Expand Up @@ -59,6 +59,7 @@ From repo root:
- `moduleResolution: "Bundler"` in tsconfig is required for subpath exports like `twd-js/runner`
- Use `import { type RawData } from 'ws'` (not `WebSocket.RawData` namespace access) for ESM/CJS compat
- Build externals: `ws`, `http`, `stream`, `vite`, `twd-js`, `twd-js/runner`
- `src/browser/diagnostics.ts` is a **deliberate port** of twd-js's internal `src/utils/diagnostics.ts`, not an import. twd-js keeps `formatDiagnostics` private, and its only public entry pulls in the sidebar, chai and the theme (~470 KB) — far too much to load into the page for a pure string formatter. The relay needs only the *data*, which arrives on the handler via `twd-js/runner` (already imported). The output must stay byte-identical to twd-js's, since the sidebar and `runner-ci` render the same block; `src/tests/browser/diagnostics.spec.ts` mirrors twd-js's own spec to keep them honest. **If twd-js's format changes, update this port.**

## Dependency Constraints

Expand Down Expand Up @@ -88,7 +89,7 @@ A green build is not sufficient evidence: the type rollup fails silently.

## Test Patterns

- **85 tests** across 9 files, runs in ~3s
- **95 tests** across 10 files, runs in ~3s
- Each test file uses **unique ports** (9877, 9878, 9879+) to avoid conflicts
- WebSocket tests use a **`TrackedWs` wrapper** that buffers incoming messages into a queue. This prevents race conditions — `nextMessage()` either returns a queued message or waits for the next one. This pattern is critical; without it, messages arrive before assertions are set up.
- A new browser connection replaces any existing one (closed with code 1000, reason "Replaced by new browser")
17 changes: 17 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,23 @@ The CLI prints `Run abandoned — browser tab appears frozen. Refresh the browse

When tests fail, the CLI prints a recap block at the very end of the output listing each failed test and its error. This survives `tail -N` truncation and is easy to copy as a single block.

### Failure diagnostics

When the installed `twd-js` reports them, each failure is prefixed with a diagnostics block naming the route the test was on and which mock rules never fired — usually enough to tell "the assertion is wrong" apart from "the page never loaded the data":

```
Failed tests (1):
× Catalog > lists products
── TWD diagnostics ─────────────────────────────────────
location /cg-1/settings/catalog
mock rules 6/7 triggered — catalog never requested
────────────────────────────────────────────────────────

expected 0 to be 3
```

The block sits above the error message, matching the TWD sidebar and `runner-ci`. The `mock rules` row is omitted entirely when the test registered no mock rules, and a run on an older `twd-js` that does not report diagnostics simply prints the error message as before.

---

## Manual setup (non-Vite, or opting out)
Expand Down
5 changes: 4 additions & 1 deletion src/browser/createBrowserClient.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ import type { BrowserClient, BrowserClientOptions } from './types';
import { createFaviconManager } from './faviconManager';
import { createRunMonitor } from './runMonitor';
import { selectTestIds, listTestPaths } from './filterTests';
import { composeFailureError, type TestDiagnostics } from './diagnostics';

declare global {
interface Window {
Expand Down Expand Up @@ -32,6 +33,8 @@ interface TwdHandler {
type: 'suite' | 'test';
status?: 'idle' | 'pass' | 'fail' | 'skip' | 'running';
logs: string[];
/** Set by twd-js only when the test fails; absent on older versions. */
diagnostics?: TestDiagnostics;
depth: number;
only?: boolean;
skip?: boolean;
Expand Down Expand Up @@ -218,7 +221,7 @@ export function createBrowserClient(options?: BrowserClientOptions): BrowserClie
id: test.id,
name: test.name,
suite: getSuiteName(test, handlers),
error: error.message,
error: composeFailureError(error.message, test.diagnostics),
duration: performance.now() - runStart,
});
},
Expand Down
78 changes: 78 additions & 0 deletions src/browser/diagnostics.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
/**
* Renderer for the failure-diagnostics block twd-js attaches to a failed test.
*
* This is a deliberate port of twd-js's `src/utils/diagnostics.ts`, not an import.
* twd-js keeps `formatDiagnostics` internal, and its only public entry point pulls
* in the sidebar, chai and the theme (~470 KB) — far too much to load into the page
* for a pure string formatter. The relay already defines twd-js's handler shape
* structurally (see `TwdHandler`), so this follows the same convention.
*
* The output must stay byte-identical to twd-js's, because the sidebar and
* `runner-ci` render the same block. `src/tests/browser/diagnostics.spec.ts`
* mirrors twd-js's own spec to keep the two honest.
*/

/** A snapshot of what TWD observed during a test, captured when the test fails. */
export interface TestDiagnostics {
/** Path + search + hash at the moment of failure. */
location: string;
/** Omitted entirely when the test registered no mock rules. */
mockRules?: {
registered: number;
triggered: number;
untriggered: string[];
};
}

const WIDTH = 56;
const HEADER = `── TWD diagnostics ${'─'.repeat(WIDTH - 19)}`;
const FOOTER = '─'.repeat(WIDTH);
const LABEL = ' ';
const LIST_CAP = 5;

/**
* Renders the signal, not the data: a page with fifteen mocks must not produce
* fifteen lines. The rules that did trigger are only interesting as a count — it
* is the misses that name a bug.
*/
export function formatDiagnostics(diagnostics: TestDiagnostics): string[] {
const lines = [HEADER, `location ${diagnostics.location}`];
const mockRules = diagnostics.mockRules;

if (mockRules) {
const { registered, triggered, untriggered } = mockRules;
const summary = `${triggered}/${registered} triggered`;

if (untriggered.length === 0) {
lines.push(`mock rules ${summary}`);
} else if (untriggered.length === 1) {
lines.push(`mock rules ${summary} — ${untriggered[0]} never requested`);
} else {
lines.push(`mock rules ${summary} — ${untriggered.length} never requested`);
for (const alias of untriggered.slice(0, LIST_CAP)) lines.push(`${LABEL}✗ ${alias}`);
const rest = untriggered.length - LIST_CAP;
if (rest > 0) lines.push(`${LABEL}+${rest} more`);
}
}

lines.push(FOOTER);
return lines;
}

/**
* Composes the `error` string sent on `test:fail`.
*
* The block goes *above* the message, matching the sidebar and `runner-ci`: the
* accessible-roles dump lives inside `err.message` and would otherwise bury it.
*
* `diagnostics` is absent for an older twd-js that does not report it, and for a
* test that failed one attempt then passed on retry — in both cases the message
* is returned untouched.
*/
export function composeFailureError(
message: string,
diagnostics?: TestDiagnostics,
): string {
if (!diagnostics) return message;
return `${formatDiagnostics(diagnostics).join('\n')}\n\n${message}`;
}
147 changes: 147 additions & 0 deletions src/tests/browser/createBrowserClient.diagnostics.spec.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
// @vitest-environment happy-dom
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
import { createServer, Server } from 'http';
import WebSocket from 'ws';
import { createTwdRelay } from '../../relay';
import type { TwdRelay } from '../../relay';

/**
* Drives the real `createBrowserClient` against a real relay, so the wiring
* between `onFail` and the diagnostics block is actually exercised. The rest of
* the browser suite simulates a browser-role connection instead, which cannot
* catch the composition being removed from `onFail`.
*
* `twd-js/runner` is mocked because it is a peer dependency that only resolves
* in a page; the fake runner drives `events.onFail` the way the real one does.
*/

const PORT = 9890;
const WS_URL = `ws://localhost:${PORT}/__twd/ws`;

const failure = {
message: 'expected 0 to be 3',
diagnostics: {
location: '/cg-1/settings/catalog',
mockRules: { registered: 7, triggered: 6, untriggered: ['catalog'] },
},
};

vi.mock('twd-js/runner', () => ({
TestRunner: class {
private events: {
onStart: (t: unknown) => void;
onFail: (t: unknown, e: Error) => void;
};

constructor(events: { onStart: (t: unknown) => void; onFail: (t: unknown, e: Error) => void }) {
this.events = events;
}

async runAll(): Promise<void> {
const test = {
id: 't1',
name: 'lists products',
parent: 's1',
type: 'test' as const,
logs: [],
depth: 1,
diagnostics: failure.diagnostics,
};
this.events.onStart(test);
this.events.onFail(test, new Error(failure.message));
}

async runByIds(): Promise<void> {
await this.runAll();
}
},
}));

function trackedClient(): {
ws: WebSocket;
messages: () => unknown[];
ready: Promise<void>;
} {
const ws = new WebSocket(WS_URL);
const received: unknown[] = [];
ws.on('message', (data) => received.push(JSON.parse(data.toString())));
const ready = new Promise<void>((resolve, reject) => {
ws.on('open', () => {
ws.send(JSON.stringify({ type: 'hello', role: 'client' }));
setTimeout(resolve, 50);
});
ws.on('error', reject);
});
return { ws, messages: () => received, ready };
}

describe('createBrowserClient — failure diagnostics', () => {
let server: Server;
let relay: TwdRelay;

beforeEach(async () => {
// The browser client uses the global WebSocket; happy-dom's does not open
// real sockets, so use the `ws` implementation against the local relay.
(globalThis as unknown as { WebSocket: unknown }).WebSocket = WebSocket;
window.__TWD_STATE__ = {
handlers: new Map([
['s1', { id: 's1', name: 'Catalog', type: 'suite', logs: [], depth: 0, handler: () => {} }],
[
't1',
{
id: 't1',
name: 'lists products',
parent: 's1',
type: 'test',
logs: [],
depth: 1,
handler: () => {},
},
],
]),
} as Window['__TWD_STATE__'];

server = createServer();
relay = createTwdRelay(server);
await new Promise<void>((resolve) => server.listen(PORT, resolve));
});

afterEach(async () => {
relay.close();
await new Promise<void>((resolve) => server.close(() => resolve()));
vi.resetModules();
});

it('sends the diagnostics block above the message on test:fail', async () => {
const { createBrowserClient } = await import('../../browser/createBrowserClient');
const client = trackedClient();
await client.ready;

const browser = createBrowserClient({ url: WS_URL });
browser.connect();
await new Promise((r) => setTimeout(r, 150));

client.ws.send(JSON.stringify({ type: 'run', scope: 'all' }));
await new Promise((r) => setTimeout(r, 250));

const fail = client.messages().find(
(m): m is { type: string; error: string } =>
typeof m === 'object' && m !== null && (m as { type?: string }).type === 'test:fail',
);

expect(fail).toBeDefined();
expect(fail!.error).toBe(
[
`── TWD diagnostics ${'─'.repeat(37)}`,
'location /cg-1/settings/catalog',
'mock rules 6/7 triggered — catalog never requested',
'─'.repeat(56),
'',
'expected 0 to be 3',
].join('\n'),
);

browser.disconnect();
client.ws.close();
});
});
Loading
Loading