You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat: add metrics for queue-based TFBuddy SLOs - #87
TFBuddy's SLOs measure the /hooks HTTP status, latency and pod up. TFBuddy answers every webhook with 200 OK and does the work later through NATS, so those SLOs can't see the failures that matter:
On 5 November, 7 GitLab MR and note events couldn't be published to HOOKS while Karpenter was evicting the NATS pods. GitLab got 200 OK for each one, so the events were lost and no one saw it.
The failure counter didn't catch them either. In gitlab_hooks/handler.go, event, err := ... inside each case declared a new err, so the err checked after the switch was always nil and a failed publish was counted as tfbuddy_gitlab_webhook_success.
Manasseh proposed three SLIs in the review thread: whether TFBuddy accepts the event, how long a queued job waits to be picked up, and how many jobs finish. This PR adds the metrics for them.
What
Accepted:tfbuddy_gitlab_webhook_failed now counts events TFBuddy couldn't queue, with reasondecode-error or publish-error. Those two are recorded without the project (it's in the error log) so their series exist from startup. The old error reason was never emitted. GitLab still gets 200 OK. New tfbuddy_gitlab_webhook_enqueue_attempts{eventType} counts the authorized merge request and note events TFBuddy tries to queue, and is the acceptance SLI's denominator; tfbuddy_gitlab_webhook_received also counts rejected requests and ignored event types.
Queue freshness: new histogram tfbuddy_jetstream_message_wait_seconds{queue}, the time from when a message was stored in the stream to its first delivery, for every jsqueue queue (hooks, RUN_EVENTS, RUN_POLLING). Redeliveries aren't observed; tfbuddy_jetstream_messages_redelivered counts them.
Completion: new counters tfbuddy_tfc_runs_created{provider,runType} and tfbuddy_tfc_runs_finished{provider,runType,status}. A run counts as finished when a run event worker has handled its final status event. That's the handler running to the end; failed comment and commit status writes are still only logged. Stale and retried events aren't counted.
Each run counts once: JetStream redelivers a final event when its ack is lost. The first worker records the run in a new RUN_COMPLETIONS KV bucket (TTL 7h, an hour longer than RUN_EVENTS keeps messages), and later deliveries aren't counted. If the record can't be written, the run is still counted.
Series exist from startup:rate() and increase() ignore the value a series starts with, so a series that first appears at 1 hides its first event. On 6 November, 90 of 96 tfbuddy_tfc_runs_started series were still at 1, so increase() saw about 14 of 110 runs. Every series the SLIs read is now created at zero: the run counters for each provider, plan/apply and final status at startup, the two webhook failure reasons at startup, and each queue's jsqueue series when it starts consuming. That's why the completion SLI uses the new runs_created rather than runs_started (which keeps its workspace label).
GitHub hooks: a failed publish was counted as both failed and success, and onError used an event-type label the counter doesn't have, so With would panic. Both fixed.
docs/architecture.md has a "Metrics for SLOs" section with an example query for each SLI.
New series: about 2 × 2 × 7 for the run counters, 4 webhook failure series, and a handful per queue.
Rollout
The first pod on this version creates the RUN_COMPLETIONS bucket (R1, file storage). Rolling back leaves an unused bucket behind, which is harmless.
After this ships
The SLO and alert changes go in argo-cd-configs (apps/tfbuddy-slo), in a separate MR.
Test plan
go test -race ./..., go vet ./..., and golangci-lint run --new-from-rev=origin/main (0 issues)
GitLab MR and note hooks: with the HOOKS stream missing, a publish counts as publish-error and not as success. With the stream in place it counts as success. A non-JSON body counts as decode-error. These fail on main.
A message that waited about 1s before its consumer started gets one wait sample of at least 1s. Its redelivery adds none.
A final run event records the finished run; in-progress, stale and retried events don't.
The same final event recorded twice on one replica and once on another counts once (fails without the record). Without the KV the run is still counted. The bucket's TTL is longer than RUN_EVENTS retention.
Authorized MR and note events count as enqueue attempts whether they are queued, fail to publish or fail to decode; ignored event types, a wrong token and a missing event header don't.
The run counters, the webhook failure and enqueue-attempt series, and a queue's series are exported before their first event (each test fails without the setup).
TFBuddy answers every webhook with 200 OK and does the work later, so the
HTTP-based SLOs could not see lost events, slow queues or runs that never
got a final status. This adds the metrics for the three SLIs:
- tfbuddy_gitlab_webhook_failed now counts events that could not be decoded
or published. The handler declared its own err inside each case, so the
error checked after the switch was always nil and failed publishes were
counted as success. Reasons are decode-error and publish-error.
- tfbuddy_jetstream_message_wait_seconds{queue}: how long a message waited
in the stream before its first delivery.
- tfbuddy_tfc_runs_finished{provider,runType,status}: runs whose final status
a run event worker handled and acked, to compare with
tfbuddy_tfc_runs_started.
Also fixes the GitHub hook metrics: a failed publish was counted as both
failed and success, and onError used a label name the counter does not have.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Ihor Horak <ihor.horak@zapier.com>
Adds Prometheus metrics and tests to support end-to-end SLOs that track a webhook from ingestion, through JetStream delivery latency, to a run’s final status being reported back to the VCS provider.
Changes:
Add tfbuddy_tfc_runs_finished{provider,runType,status} and record it when a final TFC run status is handled and the event is acked (GitLab + GitHub workers).
Add tfbuddy_jetstream_message_wait_seconds{queue} histogram to measure time-to-first-delivery (excluding redeliveries) and tests around observation behavior.
Improve webhook failure accounting/labeling and add GitLab hook metric tests; document suggested SLIs in architecture docs.
File
Description
pkg/vcs/gitlab/run_status_updater_test.go
Adds tests/helpers to assert tfbuddy_tfc_runs_finished increments only when appropriate.
pkg/vcs/gitlab/run_status_updater.go
Records “run finished” metric when the run reaches a final status and the event is acked.
pkg/vcs/github/run_events_worker.go
Records “run finished” metric for GitHub worker on final statuses.
pkg/vcs/github/hooks/github_hooks_handler.go
Fixes webhook failure labeling and avoids counting publish failures as success.
pkg/runstream/metrics.go
Introduces and registers tfbuddy_tfc_runs_finished counter + helper API.
pkg/jsqueue/jsqueue_test.go
Adds tests for the new JetStream message wait histogram behavior.
pkg/jsqueue/jsqueue.go
Adds and records tfbuddy_jetstream_message_wait_seconds on first delivery only.
pkg/gitlab_hooks/metrics.go
Adds standardized failure reason constants and clarifies failed metric meaning.
pkg/gitlab_hooks/handler_metrics_test.go
Adds tests validating publish/decode failure metrics for GitLab hooks.
pkg/gitlab_hooks/handler.go
Ensures handler error is not shadowed so failure metrics reflect real outcomes/reasons.
go.mod
Promotes client_model to direct dependency (used by tests) and updates indirect deps.
mkdocs keeps the backslash in `\|` inside a code span in a table, so the
webhook SLI rendered as a regex that matches a literal pipe. The queries
now sit in a code block under the table. Also rewords a comment on
tfbuddy_tfc_runs_finished.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Ihor Horak <ihor.horak@zapier.com>
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The SLI queries omit results when counters are absent, and redelivered final events can double-count completions and mask missing runs.
No open issues 🎉
2 previously missed issues
In code that hasn't changed since last review
Default absent Prometheus aggregates to zero
docs/architecture.md:88
These counters create label sets lazily, so the acceptance query returns no data when either successes or failures have never occurred, rather than 100% or 0%. The completion query at line 88 likewise disappears when runs have started but none has finished. Default absent acceptance aggregates and the completion numerator to zero.
Deduplicate redelivered completion events
pkg/runstream/metrics.go:24
A final event increments this counter again if JetStream redelivers it after a lost acknowledgement; publish deduplication does not prevent consumer redelivery, so the completion SLI can hide missing runs. Pass the run ID into the recorder and deduplicate completions durably across replicas, with a regression test that delivers the same final event twice. Both provider callers are affected (pkg/vcs/gitlab/run_status_updater.go:100 and pkg/vcs/github/run_events_worker.go:74).
JetStream delivers a run's final event again when its ack is lost, for
example while a NATS pod restarts, and the worker counted the run a second
time. Publish-side dedup does not cover that. RecordRunFinished is now a
StreamClient method: the first worker to handle the final event claims the
run in the WORKSPACE_DISPATCHES bucket (2h TTL), and later deliveries are
not counted. If the claim can't be recorded the run is still counted.
The example SLI queries now default absent counters to zero. A labelled
counter has no series until its first increment, so the webhook query
returned no data until the first failure.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Ihor Horak <ihor.horak@zapier.com>
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Completion reporting can double-count redelivered runs, miss first completions in documented queries, and imply successful status delivery despite posting failures.
No open issues 🎉
3 previously missed issues
In code that hasn't changed since last review
Metric counts handler completion, not successful final-status delivery
docs/architecture.md:74
This measures final-event handler completion, not successful delivery of the final status: both workers can count completion after comment-posting failures, and GitHub also counts canceled events for which the formatter produces no comment. Calling it status delivery can therefore make reporting failures appear healthy. Describe it as handler completion in this table and query explanation, and in the comment and Help text in pkg/runstream/metrics.go:17–24.
Uninitialized label combinations cause first completions to be missed
pkg/runstream/metrics.go:28
Registration alone leaves each label combination absent until its first completion. If a combination completes once and is then idle, Prometheus first scrapes 1 and keeps seeing 1, so the documented increase(...) reports zero and misses that first completion on each pod. Initialize the bounded provider/run-type/final-status combinations at startup so a scrape can establish a zero baseline before completions occur.
Completion claims expire before retained run events, allowing double counts
pkg/runstream/metrics.go:80
The completion claim expires after 2 hours (workspaceDispatchTTL), but RUN_EVENTS retains unacked messages for 6 hours (run_event.go:155). If the first delivery is counted but its ack is lost, and consumers resume after a 2–6 hour outage, this Create succeeds again and the same run increments the completion counter twice. Retain completion claims for at least the run-event retention period, either in a separate bucket or by extending and migrating the existing bucket's TTL; the current configuration helper leaves existing buckets unchanged.
rate() and increase() ignore the value a series starts with, so a series
that first appears at 1 hides its first event. On 6 November 90 of the 96
tfbuddy_tfc_runs_started series were still at 1, so increase() saw about
14 of 110 runs.
- New tfbuddy_tfc_runs_created{provider,runType}, the completion SLI's
denominator. tfbuddy_tfc_runs_started keeps its workspace label.
- tfbuddy_tfc_runs_finished and tfbuddy_tfc_runs_created are created at
zero for every provider, run type and final status on startup.
- decode-error and publish-error failures are recorded without the project
and created at zero, and the acceptance SLI divides by
tfbuddy_gitlab_webhook_received.
- jsqueue creates each queue's series when it starts consuming.
- Completion records move to their own RUN_COMPLETIONS bucket, kept for an
hour longer than RUN_EVENTS keeps messages, instead of the 2h
WORKSPACE_DISPATCHES bucket.
- tfbuddy_tfc_runs_finished is described as the final status being
handled: failed comment and status writes are logged, not counted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Ihor Horak <ihor.horak@zapier.com>
tfbuddy_gitlab_webhook_received goes up before the token and event header
checks and for event types TFBuddy ignores, so using it as the acceptance
SLI's denominator let ignored and rejected traffic dilute queueing
failures. New tfbuddy_gitlab_webhook_enqueue_attempts{eventType} counts
the authorized merge request and note events TFBuddy tries to queue, the
population decode-error and publish-error come from. Its series are
created at zero on startup.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Ihor Horak <ihor.horak@zapier.com>
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Human assessment is needed because creation-counter values and GitHub completion recording lack test coverage, while the documented ratio misrepresents boundary behavior.
Adds a trigger test that counts each created run in
tfbuddy_tfc_runs_created, and GitHub run event worker tests that record a
finished run for a final status and nothing for a status in progress. The
docs note that the completion ratio can move a little either side of 1
over a short window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Ihor Horak <ihor.horak@zapier.com>
Since #87, a failed publish is counted as publish-error only once its background retries give up, so the metric tests wait for that and use short retries. The never-published test duplicated TestHandlerCountsPublishFailures, which now covers it for merge request and note events. The docs say when publish-error is recorded and what processed-after-retry means.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Ihor Horak <ihor.horak@zapier.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
TFBuddy's SLOs measure the
/hooksHTTP status, latency and podup. TFBuddy answers every webhook with200 OKand does the work later through NATS, so those SLOs can't see the failures that matter:/hooksreturned zero 5xx. On 4 November RUN_EVENTS had about 1,800 redeliveries (the replan bug fixed in fix(hooks): stop JetStream redeliveries from replanning merge requests #83), and no SLO burned.HOOKSwhile Karpenter was evicting the NATS pods. GitLab got200 OKfor each one, so the events were lost and no one saw it.gitlab_hooks/handler.go,event, err := ...inside eachcasedeclared a newerr, so theerrchecked after theswitchwas always nil and a failed publish was counted astfbuddy_gitlab_webhook_success.Manasseh proposed three SLIs in the review thread: whether TFBuddy accepts the event, how long a queued job waits to be picked up, and how many jobs finish. This PR adds the metrics for them.
What
tfbuddy_gitlab_webhook_failednow counts events TFBuddy couldn't queue, withreasondecode-errororpublish-error. Those two are recorded without the project (it's in the error log) so their series exist from startup. The olderrorreason was never emitted. GitLab still gets200 OK. Newtfbuddy_gitlab_webhook_enqueue_attempts{eventType}counts the authorized merge request and note events TFBuddy tries to queue, and is the acceptance SLI's denominator;tfbuddy_gitlab_webhook_receivedalso counts rejected requests and ignored event types.tfbuddy_jetstream_message_wait_seconds{queue}, the time from when a message was stored in the stream to its first delivery, for everyjsqueuequeue (hooks,RUN_EVENTS,RUN_POLLING). Redeliveries aren't observed;tfbuddy_jetstream_messages_redeliveredcounts them.tfbuddy_tfc_runs_created{provider,runType}andtfbuddy_tfc_runs_finished{provider,runType,status}. A run counts as finished when a run event worker has handled its final status event. That's the handler running to the end; failed comment and commit status writes are still only logged. Stale and retried events aren't counted.RUN_COMPLETIONSKV bucket (TTL 7h, an hour longer thanRUN_EVENTSkeeps messages), and later deliveries aren't counted. If the record can't be written, the run is still counted.rate()andincrease()ignore the value a series starts with, so a series that first appears at 1 hides its first event. On 6 November, 90 of 96tfbuddy_tfc_runs_startedseries were still at 1, soincrease()saw about 14 of 110 runs. Every series the SLIs read is now created at zero: the run counters for each provider, plan/apply and final status at startup, the two webhook failure reasons at startup, and each queue'sjsqueueseries when it starts consuming. That's why the completion SLI uses the newruns_createdrather thanruns_started(which keeps its workspace label).onErrorused anevent-typelabel the counter doesn't have, soWithwould panic. Both fixed.docs/architecture.mdhas a "Metrics for SLOs" section with an example query for each SLI.New series: about 2 × 2 × 7 for the run counters, 4 webhook failure series, and a handful per queue.
Rollout
The first pod on this version creates the
RUN_COMPLETIONSbucket (R1, file storage). Rolling back leaves an unused bucket behind, which is harmless.After this ships
The SLO and alert changes go in
argo-cd-configs(apps/tfbuddy-slo), in a separate MR.Test plan
go test -race ./...,go vet ./..., andgolangci-lint run --new-from-rev=origin/main(0 issues)HOOKSstream missing, a publish counts aspublish-errorand not as success. With the stream in place it counts as success. A non-JSON body counts asdecode-error. These fail onmain.RUN_EVENTSretention.🤖 Generated with Claude Code