Repository navigation
fix: keep webhook events when NATS is briefly unavailable - #88
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Publish retries can outlast NATS’s deduplication window, allowing a webhook to be enqueued twice if the initial acknowledgment is lost.
3 open issues
What changed in this PR
This PR improves TFBuddy’s resilience to brief NATS outages by retrying webhook publishes and requiring essential streams and buckets to initialize successfully.
Changes:
- Add background publish retries for GitLab and GitHub webhooks.
- Correct webhook outcome counters and report exhausted retries.
- Retry startup resource setup, with tests for recovery and timeout behavior.
| File | Description |
|---|---|
pkg/vcs/github/hooks/github_hooks_handler.go |
Retry comment-event publishes and count final outcomes. |
pkg/runstream/stream.go |
Require successful initialization of essential NATS resources. |
pkg/runstream/setup.go |
Add setup retry helpers and fatal-error handling. |
pkg/runstream/setup_test.go |
Test setup recovery and retry exhaustion. |
pkg/runstream/run_polling.go |
Return polling-stream setup errors. |
pkg/runstream/run_event.go |
Return run-event stream setup errors. |
pkg/hooks_stream/publish.go |
Introduce shared background publish retries. |
pkg/hooks_stream/publish_test.go |
Test immediate success, recovery, and retry exhaustion. |
pkg/gitlab_hooks/handler.go |
Retry webhook publishes and correct outcome metrics. |
pkg/gitlab_hooks/handler_publish_test.go |
Test publish recovery and failure accounting against NATS. |
Review effort: Balanced
There was a problem hiding this comment.
🔵 Needs a closer look
The retry lifetime can exceed NATS’ default two-minute deduplication window, allowing an initially stored message with a lost acknowledgement to be accepted again and trigger duplicate webhook processing.
3 open issues
Review effort: Balanced
There was a problem hiding this comment.
🟡 Changes recommended
The five-minute deduplication window can suppress distinct GitLab merge-request events with the same MR ID, action, and commit SHA, causing valid reopen events to skip their plans.
1 open issue
3 resolved since last review
Review effort: Balanced
When a webhook event can't be published to the HOOKS stream, for example while a NATS server restarts, TFBuddy answers GitLab with 200 and drops the event. Neither GitLab nor GitHub resends a delivery on its own, so the MR is never planned and the comment never answered. On 5 November 7 GitLab MR and note events were lost this way during NATS pod evictions. The failure was invisible too. In the GitLab handler, `event, err :=` inside each case declares a new err, so the err checked at the end is always nil and a failed publish is counted as processed. The GitHub handler counted a failed publish as both failed and processed. Both handlers now publish through hooks_stream.PublishWithRetry: one attempt in the request, then, if it fails, retries in the background with backoff for up to 2 minutes, long enough for a NATS restart. The webhook is still answered with 200 straight away, because GitLab gives a webhook 10 seconds and disables it after four failed deliveries in a row, so an error response would lose more events, not fewer. Retries are safe because each publish carries the same Nats-Msg-Id and the HOOKS stream drops duplicates within its window. Outcomes are counted once: processed, processed-after-retry, publish-failed (with an error log saying to resend the delivery from the webhook's recent events) and invalid-payload for a body that can't be decoded, which used to be counted as processed. Separately, NewStream ignored the errors from setting up its streams and KV buckets. A pod that started while NATS was restarting got a nil polling bucket and crashed with a nil pointer in the polling dispatcher, as one pod did twice during the v0.8.5 rollout. Setup now waits up to 2 minutes for NATS, then exits with a clear error. The streams and the metadata, polling and auto-merge buckets are required. The dispatch claim bucket keeps its fail-open behaviour. Signed-off-by: Ihor Horak <ihor.horak@zapier.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… window - The HOOKS stream's Duplicates window is now 5 minutes, up from the NATS default of 2, so it outlasts a whole PublishWithRetry (first attempt, 2 minutes of retries, last attempt). A retry of an event whose first publish was stored but not acked is dropped as a duplicate instead of being stored twice. The window is set on startup for existing streams too. - A note event's Nats-Msg-Id now includes the note ID. With the discussion ID alone, a second comment in the same thread inside the window was dropped as a duplicate of the first. - NewStream's setup steps share one 2-minute window instead of 2 minutes each, so startup stays inside the pod's liveness allowance. A step that starts after the window gets one attempt, and giving up reports the setup error rather than the context's. - Webhook outcomes are logged before they are counted, so a test that waits for the counter no longer races the log line into the next test. - The publish tests compare counters with their values before the event, so they pass with go test -count=N. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Ihor Horak <ihor.horak@zapier.com>
Since #87, a failed publish is counted as publish-error only once its background retries give up, so the metric tests wait for that and use short retries. The never-published test duplicated TestHandlerCountsPublishFailures, which now covers it for merge request and note events. The docs say when publish-error is recorded and what processed-after-retry means. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Ihor Horak <ihor.horak@zapier.com>
ea0146f to
f615e06
Compare
A merge request event's Nats-Msg-Id was its MR ID, action and last commit, so reopening an MR at the same commit again inside the HOOKS Duplicates window was dropped as a duplicate of the earlier reopen, and the MR was not planned. The 5 minute window made that more likely. The ID now includes the MR's updated_at, which changes with every state change and stays the same across retries of one delivery. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Ihor Horak <ihor.horak@zapier.com>
Drops the GitHub handler's workspaceStream field, which #89 removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Ihor Horak <ihor.horak@zapier.com>


Why
Dropped webhook events. When a webhook event can't be published to the
HOOKSstream, for example while a NATS server restarts, TFBuddy still answers 200 and drops the event. Neither GitLab nor GitHub resends a delivery on its own, so the MR is never planned and thetfccomment is never answered. On 5 November, 7 GitLab MR and note events were lost this way while Karpenter was evicting NATS pods.Startup crash.
NewStreamignored the errors from setting up its streams and KV buckets. A pod that started while NATS was restarting got a nil polling bucket and crashed with a nil pointer in the polling dispatcher (run_polling.go:186). One pod did that twice during the v0.8.5 rollout. The bug has been there since 2022.What
Webhooks
hooks_stream.PublishWithRetry. It tries once in the request, and if that fails it keeps retrying in the background with backoff for up to 2 minutes, long enough for a NATS restart.Nats-Msg-Id, and theHOOKSstream drops duplicates inside its window. That window is now 5 minutes, up from the NATS default of 2, so it outlasts a whole retry (first attempt, 2 minutes of retries, last attempt), and it is set on existing streams at startup.Nats-Msg-Idnow includes the note ID. With the discussion ID alone, a second comment in the same thread inside the window was dropped as a duplicate of the first.Nats-Msg-Idnow includes the MR'supdated_at. Without it, reopening an MR at the same commit again inside the window was dropped as a duplicate of the earlier reopen. Retries, and the same event arriving through a group and a project webhook, still share one ID.processedandprocessed-after-retryintfbuddy_gitlab_webhook_successpublish-errorintfbuddy_gitlab_webhook_failed, without the project, once the retries give up, with an error log telling you to resend the delivery from the webhook's recent eventsdecode-erroras before.errthat counted failed publishes as processed.docs/architecture.mdnow sayspublish-erroris recorded only after the retries, up to 2 minutes after the webhook arrived.Startup
NewStreamwaits up to 2 minutes for NATS while it sets up theRUN_EVENTSandRUN_POLLINGstreams and the metadata, polling and auto-merge buckets, then exits with a clear error. All steps share that one window, so startup stays inside the 5 minutes the pod's probes allow; a step that starts after the window gets one attempt.nats: stream is offline(9 in all, all during NATS restarts). That's exactly what the wait covers, so this doesn't turn a recurring error into a crash loop.Not covered
An event that is still being retried when its pod is killed is lost, as today. The error log names it so it can be resent from GitLab.
Test plan
go test -race ./...andgo vet ./...PublishWithRetry: publishes right away; keeps the event until NATS is back; gives up after the policy window.HOOKSstream missing, the webhook gets 200 straight away, and the event lands once the stream exists, counted asprocessed-after-retry;publish-erroronce the retries give up, not as processed; the tests wait for that so no retry outlives its test.setUpWithRetrywaits for NATS and gives up after its window with the setup error; steps after a used-up window get one attempt.configureHooksStreamsets it on a new or an existing stream.go test -count=3.pkg/runstreamandpkg/gitlab_hookspass against a local nats-server v2.9.8 (production's version).🤖 Generated with Claude Code