Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 55 additions & 0 deletions packages/opencode/test/cli/cmd/report-cli.property.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
import { afterAll, describe, expect, spyOn, test } from "bun:test"
import fc from "fast-check"
import { ReportCommand } from "../../../src/cli/cmd/report"
import { CliError } from "../../../src/cli/effect-cmd"
import { disposeAllInstances } from "../../fixture/fixture"

// Property test for the `opencode report --session <id>` command (R5). Requirement IDs match traceability/anishjai.md.
// The core R5 property checks EditDecision.summary(); this one runs the real command handler, so it also catches
// the CLI swallowing SessionNotFoundError and printing a zero report (red team A2).

fc.configureGlobal({ seed: 42 })

// every run boots a project instance, so keep the run count low
const NUM_RUNS = 10

afterAll(async () => {
await disposeAllInstances()
})

describe("report command properties", () => {
// R5: A session ID that doesn't exist gives an error, not a table of zeros.
// Kind: invariant. For any session id that was never created, the command fails with a CliError
// naming that id and prints nothing.
test(
"R5: report --session with an unknown id fails instead of printing zeros",
async () => {
await fc.assert(
fc.asyncProperty(fc.stringMatching(/^[A-Za-z0-9]{1,12}$/), async (typo) => {
const session = `ses_missing_${typo}`
const printed: string[] = []
const log = spyOn(console, "log").mockImplementation((line: string) => void printed.push(line))
const out = spyOn(process.stdout, "write").mockImplementation((chunk: any) => {
printed.push(String(chunk))
return true
})
const result = await Promise.resolve(
ReportCommand.handler({ session, _: [], $0: "opencode" } as never),
).then(
() => undefined,
(error: unknown) => error,
)
log.mockRestore()
out.mockRestore()

expect(result).toBeInstanceOf(CliError)
expect((result as CliError).message).toBe(`Session not found in this project: ${session}`)
expect(printed.join("")).not.toContain("No accept/reject decisions")
}),
{ numRuns: NUM_RUNS },
)
console.log(`R5 (command): passed ${NUM_RUNS} random runs`)
},
60_000,
)
})
4 changes: 2 additions & 2 deletions traceability/anishjai.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
| R2 | C-9, O-9, O-10: "It should be locked to just the project folder you're in" | The report only counts decisions from sessions in the current project. | Project A has 3 recorded decisions and Project B has 5. Running `opencode report` in A shows Edits Proposed 3. | No | adding decisions to another project leaves this project's report unchanged (metamorphic) | [report.property.test.ts](../packages/core/test/report/report.property.test.ts): "R2: decisions in another project never change this project's report" | H2 |
| R3 | C-15, O-15, S-13: "add up the parts and see if they equal the total" | The per-session reports for every session in a project add up to the plain report. | Session X has 3 accepted / 1 rejected, Y has 1 accepted / 1 rejected. `--session X` shows 4/3/1, `--session Y` shows 2/1/1, and the overall report shows 2 sessions, 6 proposed, 4 accepted, 2 rejected. | No | summing every `--session` report equals the plain report (model / oracle) | [report.property.test.ts](../packages/core/test/report/report.property.test.ts): "R3: per-session reports add up to the plain report" | H2; H3 and red team A1 (after strengthening) |
| R4 | C-11, O-11, O-13, O-14: "It should just go back to what it was before you did those 3 edits" | New decisions show up in the report right away, and deleting a session removes its decisions. | Report starts at 2 sessions, 6 proposed, 4 accepted, 2 rejected. Accepting 3 edits in a new session gives 3 sessions, 9 proposed, 7 accepted, 2 rejected, 77.8%. Deleting that session brings it back to 2 / 6 / 4 / 2 / 66.7%. | No | adding a session raises the report by exactly its decisions, deleting it restores the original report (round-trip) | [report.property.test.ts](../packages/core/test/report/report.property.test.ts): "R4: adding then deleting a session puts the report back where it was" | H4; red team A1 |
| R5 | C-14: "Don't just show zeros, because then I'd think I didn't do anything in that session." | A session ID that doesn't exist or belongs to another project gives an error, not a table of zeros. | `opencode report --session abc123`, where abc123 is not in this project, prints "Session not found in this project: abc123" and exits with a non-zero code. | No | a typo'd id or a session from another project always gives SessionNotFoundError (invariant) | [report.property.test.ts](../packages/core/test/report/report.property.test.ts): "R5: a session id not in this project is an error, never zeros" | H5 |
| R5 | C-14: "Don't just show zeros, because then I'd think I didn't do anything in that session." | A session ID that doesn't exist or belongs to another project gives an error, not a table of zeros. | `opencode report --session abc123`, where abc123 is not in this project, prints "Session not found in this project: abc123" and exits with a non-zero code. | No | a typo'd id or a session from another project always gives SessionNotFoundError (invariant) | [report.property.test.ts](../packages/core/test/report/report.property.test.ts): "R5: a session id not in this project is an error, never zeros"; [report-cli.property.test.ts](../packages/opencode/test/cli/cmd/report-cli.property.test.ts): "R5: report --session with an unknown id fails instead of printing zeros" | H5; red team A2 (command property, after fix) |
| R6 | C-17, C-18, C-20, C-21, O-17: "that should probably have its own counter so it doesn't skew the numbers", "maybe show both? Like 1 decision, 5 edits" | Any edit that goes through without the student answering a prompt (allowed by a config rule, or by an earlier "always" reply) counts in its own Auto-Approved row and never changes Proposed, Accepted, Rejected, or the ratio. Edits that still show a prompt, including forced unfamiliar-file prompts, count as normal decisions. | With 4 accepted and 2 rejected, plus 3 edits allowed by `"edit": "allow"`, report shows Proposed 6, Accepted 4, Rejected 2, Ratio 66.7%, Auto-Approved 3. One "always" reply that also lets 2 pending edits through then adds 1 to Accepted and 2 to Auto-Approved. | Yes | Not tested yet (not built) | | None (not built) |
| R7 | C-26, O-21: "an option to export as JSON or CSV" | `opencode report --json` prints one JSON object to stdout with the same counts as the table, and `acceptRatio` as the unrounded accepted / proposed (0 to 1). A project with no decisions prints all zeros instead of a message. | Using the R1 project, `opencode report --json` output parses as JSON and equals `{"sessions":2,"proposed":6,"accepted":4,"rejected":2,"acceptRatio":0.6666666666666666}`. | Yes | Not tested yet (not built) | | None (not built) |

Expand Down Expand Up @@ -78,7 +78,7 @@ Feature claimed by **Vlad Miloserdov (vmiloser, Team Zero)**, who posted three m
| Mutant | Sent by | Requirement | Change | Result before | Fix | Result after |
|---|---|---|---|---|---|---|
| A1 | Vlad Miloserdov (vmiloser) | R3 | `decision.ts`: `countDistinct(session_id)` → `count(session_id)`, so Sessions counts decision rows | Killed by R4 (`during.sessions` must be `before.sessions + 1`, but a new session with several decisions adds several) | None needed. The strengthened R3 (Sessions vs. the generator) also kills it now. | Killed by R3, R4 |
| A2 | Vlad Miloserdov (vmiloser) | R5 | `report.ts`: CLI catches `SessionNotFoundError` and returns a zero summary instead of failing | Survived: every property calls `EditDecision.summary()` directly, which still returns the error | Not fixed. The error handling lives inside the `ReportCommand` handler, which only runs through `effectCmd` with a full `InstanceStore` bootstrap. I didn't get a property wired to that path before the deadline. It isn't equivalent: the mutated CLI prints "No accept/reject decisions recorded for session ..." and exits 0, which is exactly what C-14 warned against. The fix would be a property that generates missing/other-project ids and runs the handler, asserting it fails with `CliError`. | Survived |
| A2 | Vlad Miloserdov (vmiloser) | R5 | `report.ts`: CLI catches `SessionNotFoundError` and returns a zero summary instead of failing | Survived: every property called `EditDecision.summary()` directly, which still returns the error | New property [packages/opencode/test/cli/cmd/report-cli.property.test.ts](../packages/opencode/test/cli/cmd/report-cli.property.test.ts) "R5: report --session with an unknown id fails instead of printing zeros". It runs the real `ReportCommand` handler (full instance bootstrap) with generated never-created session ids and asserts it rejects with a `CliError` whose message is `Session not found in this project: <id>`, and that it never prints the zero-decision message. | Killed: counterexample `["A"]` (session `ses_missing_A`), the handler resolved instead of failing with `CliError` |
| A3 | Vlad Miloserdov (vmiloser) | R1 | `report.ts`: Accept Ratio printed with `toFixed(0)` instead of `toFixed(1)` | Survived: no property looked at the printed table | New property [packages/opencode/test/cli/cmd/report.property.test.ts](../packages/opencode/test/cli/cmd/report.property.test.ts) "R1: the table shows the counts and the accept ratio to one decimal". It renders `displayReport` for generated counts, parses each row, and checks it against counts and a ratio computed with integer math, not from `acceptRatio * 100`. | Killed: counterexample `[0,1,1]` (0 accepted, 1 rejected, 1 session), the table showed `0%` instead of `0.0%` |

## Agent failure case
Expand Down
Loading