Skip to content

fix: keep webhook events when NATS is briefly unavailable - #88

Merged
horakihor merged 5 commits into
mainfrom
fix/hooks-publish-failures
Nov 11, 2026
Merged

horakihor merged 5 commits into
mainfrom
fix/hooks-publish-failures

Conversation

@horakihor

@horakihor horakihor commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Why

Dropped webhook events. When a webhook event can't be published to the HOOKS stream, for example while a NATS server restarts, TFBuddy still answers 200 and drops the event. Neither GitLab nor GitHub resends a delivery on its own, so the MR is never planned and the tfc comment is never answered. On 5 November, 7 GitLab MR and note events were lost this way while Karpenter was evicting NATS pods.

Startup crash. NewStream ignored the errors from setting up its streams and KV buckets. A pod that started while NATS was restarting got a nil polling bucket and crashed with a nil pointer in the polling dispatcher (run_polling.go:186). One pod did that twice during the v0.8.5 rollout. The bug has been there since 2022.

What

Webhooks

  • Both handlers publish through the new hooks_stream.PublishWithRetry. It tries once in the request, and if that fails it keeps retrying in the background with backoff for up to 2 minutes, long enough for a NATS restart.
  • The webhook is still answered with 200 straight away. GitLab.com gives a webhook 10 seconds, doesn't retry failed deliveries, and disables a webhook after 4 failed deliveries in a row (docs). An error response would lose more events, not fewer.
  • Retrying is safe: each publish carries the same Nats-Msg-Id, and the HOOKS stream drops duplicates inside its window. That window is now 5 minutes, up from the NATS default of 2, so it outlasts a whole retry (first attempt, 2 minutes of retries, last attempt), and it is set on existing streams at startup.
  • A note event's Nats-Msg-Id now includes the note ID. With the discussion ID alone, a second comment in the same thread inside the window was dropped as a duplicate of the first.
  • A merge request event's Nats-Msg-Id now includes the MR's updated_at. Without it, reopening an MR at the same commit again inside the window was dropped as a duplicate of the earlier reopen. Retries, and the same event arriving through a group and a project webhook, still share one ID.
  • Outcomes use the reasons and series feat: add metrics for queue-based TFBuddy SLOs #87 added, so its webhook SLI keeps working:
    • processed and processed-after-retry in tfbuddy_gitlab_webhook_success
    • publish-error in tfbuddy_gitlab_webhook_failed, without the project, once the retries give up, with an error log telling you to resend the delivery from the webhook's recent events
    • decode-error as before.
  • Rebased onto feat: add metrics for queue-based TFBuddy SLOs #87, which already fixed the shadowed err that counted failed publishes as processed. docs/architecture.md now says publish-error is recorded only after the retries, up to 2 minutes after the webhook arrived.

Startup

  • NewStream waits up to 2 minutes for NATS while it sets up the RUN_EVENTS and RUN_POLLING streams and the metadata, polling and auto-merge buckets, then exits with a clear error. All steps share that one window, so startup stays inside the 5 minutes the pod's probes allow; a step that starts after the window gets one attempt.
  • The dispatch claim bucket keeps its fail-open behaviour.
  • Over the last 7 days every stream setup error in Graylog was nats: stream is offline (9 in all, all during NATS restarts). That's exactly what the wait covers, so this doesn't turn a recurring error into a crash loop.

Not covered

An event that is still being retried when its pod is killed is lost, as today. The error log names it so it can be resent from GitLab.

Test plan

  • go test -race ./... and go vet ./...
  • PublishWithRetry: publishes right away; keeps the event until NATS is back; gives up after the policy window.
  • GitLab handler, against a real NATS server:
    • with the HOOKS stream missing, the webhook gets 200 straight away, and the event lands once the stream exists, counted as processed-after-retry;
    • for merge request and note events, an event that never lands is counted as publish-error once the retries give up, not as processed; the tests wait for that so no retry outlives its test.
  • setUpWithRetry waits for NATS and gives up after its window with the setup error; steps after a used-up window get one attempt.
  • The HOOKS Duplicates window outlasts the retries, and configureHooksStream sets it on a new or an existing stream.
  • Two comments in one discussion get different message IDs, and so do two reopens of one MR at the same commit.
  • The publish tests pass with go test -count=3.
  • Mutation checks: without the background retry, with a failure counted as processed, or without the setup retry, the matching test fails.
  • The NATS-backed tests in pkg/runstream and pkg/gitlab_hooks pass against a local nats-server v2.9.8 (production's version).

🤖 Generated with Claude Code

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Publish retries can outlast NATS’s deduplication window, allowing a webhook to be enqueued twice if the initial acknowledgment is lost.

3 open issues
What changed in this PR

This PR improves TFBuddy’s resilience to brief NATS outages by retrying webhook publishes and requiring essential streams and buckets to initialize successfully.

Changes:

  • Add background publish retries for GitLab and GitHub webhooks.
  • Correct webhook outcome counters and report exhausted retries.
  • Retry startup resource setup, with tests for recovery and timeout behavior.
File Description
pkg/​vcs/​github/​hooks/​github_hooks_handler.go Retry comment-event publishes and count final outcomes.
pkg/​runstream/​stream.go Require successful initialization of essential NATS resources.
pkg/​runstream/​setup.go Add setup retry helpers and fatal-error handling.
pkg/​runstream/​setup_test.go Test setup recovery and retry exhaustion.
pkg/​runstream/​run_polling.go Return polling-stream setup errors.
pkg/​runstream/​run_event.go Return run-event stream setup errors.
pkg/​hooks_stream/​publish.go Introduce shared background publish retries.
pkg/​hooks_stream/​publish_test.go Test immediate success, recovery, and retry exhaustion.
pkg/​gitlab_hooks/​handler.go Retry webhook publishes and correct outcome metrics.
pkg/​gitlab_hooks/​handler_publish_test.go Test publish recovery and failure accounting against NATS.

Review effort: Balanced

Comment thread pkg/hooks_stream/publish.go Resolved
Comment thread pkg/runstream/stream.go Resolved

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The retry lifetime can exceed NATS’ default two-minute deduplication window, allowing an initially stored message with a lost acknowledgement to be accepted again and trigger duplicate webhook processing.

3 open issues

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The five-minute deduplication window can suppress distinct GitLab merge-request events with the same MR ID, action, and commit SHA, causing valid reopen events to skip their plans.

1 open issue
3 resolved since last review

Review effort: Balanced

Comment thread pkg/hooks_stream/hooks_stream.go Resolved
horakihor and others added 3 commits November 9, 2026 12:29
When a webhook event can't be published to the HOOKS stream, for
example while a NATS server restarts, TFBuddy answers GitLab with 200
and drops the event. Neither GitLab nor GitHub resends a delivery on
its own, so the MR is never planned and the comment never answered.
On 5 November 7 GitLab MR and note events were lost this way during
NATS pod evictions.

The failure was invisible too. In the GitLab handler, `event, err :=`
inside each case declares a new err, so the err checked at the end is
always nil and a failed publish is counted as processed. The GitHub
handler counted a failed publish as both failed and processed.

Both handlers now publish through hooks_stream.PublishWithRetry: one
attempt in the request, then, if it fails, retries in the background
with backoff for up to 2 minutes, long enough for a NATS restart. The
webhook is still answered with 200 straight away, because GitLab gives
a webhook 10 seconds and disables it after four failed deliveries in a
row, so an error response would lose more events, not fewer. Retries
are safe because each publish carries the same Nats-Msg-Id and the
HOOKS stream drops duplicates within its window.

Outcomes are counted once: processed, processed-after-retry,
publish-failed (with an error log saying to resend the delivery from
the webhook's recent events) and invalid-payload for a body that can't
be decoded, which used to be counted as processed.

Separately, NewStream ignored the errors from setting up its streams
and KV buckets. A pod that started while NATS was restarting got a nil
polling bucket and crashed with a nil pointer in the polling dispatcher,
as one pod did twice during the v0.8.5 rollout. Setup now waits up to 2
minutes for NATS, then exits with a clear error. The streams and the
metadata, polling and auto-merge buckets are required. The dispatch
claim bucket keeps its fail-open behaviour.

Signed-off-by: Ihor Horak <ihor.horak@zapier.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… window

- The HOOKS stream's Duplicates window is now 5 minutes, up from the NATS
  default of 2, so it outlasts a whole PublishWithRetry (first attempt,
  2 minutes of retries, last attempt). A retry of an event whose first
  publish was stored but not acked is dropped as a duplicate instead of
  being stored twice. The window is set on startup for existing streams too.
- A note event's Nats-Msg-Id now includes the note ID. With the discussion
  ID alone, a second comment in the same thread inside the window was
  dropped as a duplicate of the first.
- NewStream's setup steps share one 2-minute window instead of 2 minutes
  each, so startup stays inside the pod's liveness allowance. A step that
  starts after the window gets one attempt, and giving up reports the
  setup error rather than the context's.
- Webhook outcomes are logged before they are counted, so a test that waits
  for the counter no longer races the log line into the next test.
- The publish tests compare counters with their values before the event,
  so they pass with go test -count=N.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Ihor Horak <ihor.horak@zapier.com>
Since #87, a failed publish is counted as publish-error only once its background retries give up, so the metric tests wait for that and use short retries. The never-published test duplicated TestHandlerCountsPublishFailures, which now covers it for merge request and note events. The docs say when publish-error is recorded and what processed-after-retry means.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Ihor Horak <ihor.horak@zapier.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The extended deduplication window can incorrectly acknowledge distinct merge-request reopen events as duplicates, skipping the plan trigger because message IDs omit the upstream DeliveryID.

1 open issue

Review effort: Balanced

A merge request event's Nats-Msg-Id was its MR ID, action and last commit, so reopening an MR at the same commit again inside the HOOKS Duplicates window was dropped as a duplicate of the earlier reopen, and the MR was not planned. The 5 minute window made that more likely. The ID now includes the MR's updated_at, which changes with every state change and stays the same across retries of one delivery.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Ihor Horak <ihor.horak@zapier.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are identified for the webhook retry and startup resource-setup changes.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

Drops the GitHub handler's workspaceStream field, which #89 removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Ihor Horak <ihor.horak@zapier.com>
@horakihor
horakihor merged commit f540282 into main Nov 11, 2026
8 checks passed
@horakihor
horakihor deleted the fix/hooks-publish-failures branch November 11, 2026 09:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants