Skip to content

Fix verifier, composer and reporting bugs from the #44 review - #89

Merged
pkieltyka merged 16 commits into
masterfrom
peter/44-fixes-pipeline
Nov 19, 2026
Merged

pkieltyka merged 16 commits into
masterfrom
peter/44-fixes-pipeline

Conversation

@pkieltyka

Copy link
Copy Markdown
Collaborator

Fixes #59, #60, #61, #62, #63, #64, #65, #71, #72 and #73, found in the #44 deep review. One commit per issue, plus one review-fix commit.

Verifier and publication

Composition and reporting

Planning and packets

Verification

  • Each regression test fails without its fix.
  • pnpm check, pnpm test (1601 tests) and git diff --check pass.
  • Peer review required one change for Composer errors discard verified findings #60 (primitive rejection reasons). The re-review approved 747cd06 with no findings.
  • No file overlaps with PRs merged since the branch's base.

🤖 Generated with Claude Code

pkieltyka and others added 11 commits November 19, 2026 15:48
The keep criterion requires false-positive risk that is not high, but a
keep or revise with falsePositiveRisk "high" and required evidence present
was accepted and published. normalizeSubmittedVerdict now turns it into a
reject with its own reason and a
verification_high_false_positive_risk_normalized_to_reject event. Missing
evidence keeps precedence and its existing reason.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With review.verify=false the verifier synthesized keep verdicts with
requiredEvidencePresent true, so the composer's low-confidence exception
treated unverified candidates as verified and could publish them inline.
The synthetic verdict now reports requiredEvidencePresent false: nothing
was checked. human-attention's reason-string test for the same case is
now redundant and removed, leaving one rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
revisedFinding rebuilt a revised finding from submitted schema fields and
dropped the host-owned provenance, so every verified promoted finding
(each one is revised) stopped looking promoted, and the composer's
promoted representative branch only ran when verification was off. The
revision now forwards the trusted provenance like producedBy and the
cluster fields; the verifier still cannot patch it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A revise rebuilt severity from the submitted fields only. When Stage 7 had
capped a critical finding to medium, a title-only revise resubmitted
medium, applySeverityPolicy did not cap again, and severityBeforeCap was
lost, so the never-hide-critical guarantee no longer applied. The revised
finding now keeps the original severityBeforeCap when the verifier leaves
the capped severity unchanged; a severity the verifier changes replaces it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Stage 10 error that was neither budget exhaustion nor a recoverable LLM
error was rethrown before any report existed, and review-runner assigns
retainedReview only after the composer returns, so the failure report had
zero findings although verification had finished. The spec says Stage 10
invalidity must not drop verified findings.

The composer now still builds the deterministic fallback report on that
failure path and hands it to the runner (reviewRetainedOnCompositionFailure)
before rethrowing. The run still fails, and its failure report carries
the verified findings, the fallback composition mode and the partial
banner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
refreshReviewHealth re-rendered postingPlan.reviewBody without the
composition outcome, so a review that used composer fallback lost its
"Report synthesis failed" banner on GitHub while the markdown report kept
it. It now passes result.composition like the original render.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
validatePlan prefixed "planner_empty_lenses: " to any decision whose lens
set came back empty, including a valid skip that runs no lenses. The hunk
stayed skipped but lost the diagnostic reason the component spec requires
for skips. Skips now keep the planner's reason; non-skip decisions still
record the empty-lens default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Configuration skips were disclosed as "<path>: <reason>", but planner
skips only raised the generic excluded count, so their path and reason
existed only in the coverage artifact. aggregateRunCoverage now discloses
both kinds the same way, under the existing ten-entry limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stage timings come from stage_started/stage_completed events, but
recordEvent returned before recording anything when telemetry was off,
the default, so snapshotStageTimings was empty and the report's stage
timings line disappeared. The stage start/complete timestamps are now
recorded regardless; nothing else is collected or written when telemetry
is off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The deleted-test rule (plan 92 E1) raised a packet to deep after Stage 6
had already chosen its profile, lenses, context and tool budget from the
planned coverage. A simple packet kept a 0/0/0 budget and no tools; a
standard one got 10/3/20k instead of the investigate 20/6/48k, and the
prompt kept the old profile.

buildPacket now applies the rule right after computing the test-coverage
delta, so the escalated coverage drives context, profile, lenses and tool
budget, and the packet carries coverageEscalation and emits
coverage_escalated as before. applyCoverageEscalations keeps the E2
telemetry and the run-level summary. The rule moves into the exported
coverageEscalationReason, which removes coverage-escalation's import of
packet-builder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review finding: compositionFailure doubled as the presence check, so a
composer rejecting with undefined was treated as success, and a string or
null reason could not key the WeakMap, so the runner lost the retained
review. Failure presence is now an explicit flag, and a primitive reason
is wrapped in an Error (cause = the original) that carries the handoff.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

🧞 Codegenie Review

Warning

Review completed with unresolved questions. 10 question(s) remain unresolved; absence of a confirmed finding does not establish safety.

0 confirmed findings retained from completed work. Unresolved questions require attention.

Coverage

Reviewed 38/39 hunks.
Excluded by configuration/planning: 1 hunks.
Coverage levels: deep 25, normal 9, light 4, skip 1.

  • specs/project/components/review_pipeline.md: configured skip rule

🙋 Needs Human Attention

  • Are there packet construction paths other than buildPacket (e.g. degraded/fallback packets) that previously received E1 escalation from applyCoverageEscalations and now do not?; Does src/pipeline/coverage-escalation.ts export coverageEscalationReason, and is it actually used in packet-builder.ts (an unused import would fail lint/noUnusedLocals)?; Does any ReviewPacket reach applyCoverageEscalations without passing through buildPacket (e.g. degraded/fallback packet construction), which would now miss the E1 escalation that the stage-level pass previously applied?; Is the E2 intent_mismatch telemetry-only branch of applyCoverageEscalations (plan.intentSignals refactorLike/explicitlyBehaviorPreserving plus deleted lines, skipping test paths) exercised by any test? The local plan fixture sets no intentSignals.; Does tests/coverage-escalation.test.ts still cover the aggregate coverage_escalations event and E2 candidate telemetry now that applyCoverageEscalations only counts packets instead of escalating them?

    • Files: src/pipeline/coverage-escalation.ts, src/pipeline/packet-builder.ts, src/pipeline/review-runner.ts, tests/coverage-escalation.test.ts
    • Symbols: applyCoverageEscalations, buildPacket, buildReviewPackets, coverageEscalationReason, isCoverageEscalationTestPath
    • Reason: Packet reviewer could not resolve this question from the reviewed context. Grouped from 5 related hints across 5 packets.
  • Do tests cover that with review.verify=false a synthetic keep no longer produces a stage9_verified_predicate resolution (the behavior previously enforced by the deleted reason-string guard)?; With review.verify=false, does suppressing the composer low-confidence behavior-delta exception (lowConfidencePublishableCandidateIds requires requiredEvidencePresent===true) match the intended product behavior for unverified runs?; Does any remaining test exercise human-attention's verdictResolvesPredicate for a verify=false synthetic keep after the 'verification disabled by config' regex guard was removed (src/pipeline/human-attention.ts:972-975) and the redundant human-attention test deleted?; Does verifyFindings' handling of verdict "revise" with findingUpdates merge the original finding's severity and behaviorChange into submitted.finalFinding, so the title-only case reaches revisedFinding with severity "medium" and behaviorChange "intentional_needs_confirmation"?

    • Files: src/pipeline/composer.ts, src/pipeline/human-attention.ts, src/pipeline/verifier.ts, tests/human-attention-adjudication.test.ts, tests/pipeline-phase5.test.ts, tests/verifier.test.ts
    • Symbols: applySeverityPolicy, buildVerificationResolutionIndex, lowConfidencePublishableCandidateIds, revisedFinding, verdictResolvesPredicate, verifyFindings
    • Reason: Packet reviewer could not resolve this question from the reviewed context. Grouped from 4 related hints across 4 packets.
  • When an artifact write rejects after a non-recoverable composer failure (composer.ts:417-419), the original composer error (e.g. CodegenieError "llm_call_failed") is discarded in favor of the write error, so review-runner's errorExitCode/isCodegenieError classification and error.json report the write error instead of the root cause. Is losing the composer error code intended, or should it be attached as cause?; Does review-runner.ts actually use reviewRetainedOnCompositionFailure on the composition-failure path to populate the failure report's retained review?

    • Files: src/pipeline/composer.ts, src/pipeline/review-runner.ts, tests/pipeline-phase5.test.ts
    • Symbols: dedupeRankAndComposeReview, retainReviewOn, reviewFailureRecord, reviewRetainedOnCompositionFailure, runReview
    • Reason: The packet shows only the import line; an unused or mismatched import would mean the Composer errors discard verified findings #60 fix (retained findings in the failure report) is not wired into the runner, or would fail to compile/lint. The usage site is outside this hunk. Related reasons: Packet reviewer could not resolve this question from the reviewed context. Grouped from 2 related hints across 2 packets.
  • Can any artifact or summary consumer besides allStageSummaries and snapshotStageTimings observe the zero-event byStage buckets that recordStageTiming lazily creates when telemetry is disabled?; With telemetry disabled, recordEvent now calls recordStageTiming before its early return; is there an inspected test asserting that disabled mode still writes no events/artifacts (attachRunDirectory returns an empty runDir, no events.jsonl)?

    • Files: src/telemetry/run-artifacts.ts, tests/telemetry.test.ts
    • Symbols: allStageSummaries, attachRunDirectory, createRunTelemetry, recordEvent, recordStageTiming, snapshotStageTimings
    • Reason: Packet reviewer could not resolve this question from the reviewed context. Grouped from 2 related hints across 2 packets.
  • Should the new high-false-positive-risk reject spread ...normalized (like the proof-not-established path at verifier.ts:799) instead of rebuilding a minimal verdict, so proofAssessment/behaviorChange/intentEvidence/suggestionAssessments survive into the VerificationVerdict record built at verifier.ts:620-640?; Should a verdict with requiredEvidencePresent:false AND falsePositiveRisk:"high" be covered by an explicit precedence test (it must keep the "required evidence missing" reason and requiredEvidencePresent:false rather than the new high-risk reason)? The existing missing-evidence test uses falsePositiveRisk:"medium", so it cannot detect loss of the requiredEvidencePresent === true guard in the new branch.

    • Files: src/pipeline/verifier.ts, tests/verifier.test.ts
    • Symbols: normalizeSubmittedVerdict
    • Reason: Packet reviewer could not resolve this question from the reviewed context. Grouped from 2 related hints across 2 packets.

Additional unresolved notes suppressed: 5.

Stats

  • 🤖 Model: anthropic claude-opus-5 high
  • 🧞 Codegenie: v0.8.0 (05569289fb)
  • Elapsed time: 5m 36s
  • Git: 0xPolygon/codegenie from master to peter/44-fixes-pipeline (e09ab5ab44)
  • Review completeness: complete.
  • Usage: model calls 105, tokens 3071571, cost $8.4063.
  • Effective caps: tokens 8000000.
  • Local context pressure: 15 degraded tool results, 11 degraded hunks, 5 unresolved notes suppressed.

No confirmed findings

No confirmed findings were retained. The limitations above prevent a clean conclusion.

— View Workflow Job

pkieltyka and others added 5 commits November 19, 2026 17:08
…on (#60)

PR #89 review: the deterministic fallback review was registered for the
failure path only after the stage-10 artifact writes. A write that
rejected in between reached the runner as a different error, and the
failure report had zero findings. Any error after the review is built now
carries it (reviewRetainedOnCompositionFailure); a write failure is still
the error the run reports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR #89 review: the raw skip list was cut to ten before reasons were
deduplicated, so ten identical path/reason skips used the whole limit and
hid later reasons. Path/reason pairs are now deduplicated first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#71)

PR #89 review: the component spec still described planner_empty_lenses
for any decision, including a valid skip, which keeps its own reason.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… inputs (#59)

PR #89 review: replaces the dropped already-deep case. A deep test-file
packet whose deleted tests leave surviving references gets no
coverageEscalation and no coverage_escalated event. The
applyCoverageEscalations test now compares against a snapshot taken
before the call, so an in-place mutation would fail it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…line

# Conflicts:
#	tests/pipeline-phase5.test.ts
@pkieltyka
pkieltyka merged commit 4f21784 into master Nov 19, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment