nflog gossip merge drops firing alerts, causing duplicate notifications from healthy cluster peers #5685

What did you do?

In a 3 instance cluster, one instance knows about fewer alerts in a group than its peers, then notifies for an alert the peers do not have. The peers afterwards re-notify for the alerts that instance never saw.

No restart and no clock skew required. It needs only one instance writing an nflog entry that is newer and less complete than its peers'.

Failing tests in #5686, one unit and one against the existing 3 instance acceptance harness.

To reproduce:

Run three instances pointed at each other, with a short peer timeout so instances at non-zero cluster positions get a turn to notify quickly:

alertmanager --config.file=am.yml --storage.path=/tmp/am1 \
  --web.listen-address=:9093 --cluster.listen-address=127.0.0.1:9094 \
  --cluster.peer=127.0.0.1:9096 --cluster.peer=127.0.0.1:9098 \
  --cluster.peer-timeout=1s
# repeat for :9095/:9096 and :9097/:9098

am.yml

route:
  receiver: default
  group_by: []
  group_wait: 1s
  group_interval: 1s
  repeat_interval: 1h
receivers:
- name: default
  webhook_configs:
  - url: 'http://127.0.0.1:5001/'

Then, with any webhook listener on :5001:

1. Post alert1 and alert2 to all three instances. One notification arrives containing both. Alerts are not gossiped between instances, so each instance must be posted to directly, the same way Prometheus fans out.
2. Wait for the notification, then post alert3 to instances 1 and 2 only. A notification arrives containing alert1, alert2, alert3. Instance 3 does not know about alert3.
3. Wait again, then post alert4 to instance 3 only. Instance 3 sees alert4 as new, notifies with alert1, alert2, alert4, and records an entry whose firing set omits alert3.
4. Watch the listener. alert3 is notified a second time.

Keep repeat_interval well above the test duration. Entry TTL is min(data.retention, 2 * repeat_interval), so a short repeat_interval will expire entries and confound the result.

What did you expect to see?

Three notifications: {alert1, alert2}, then {alert1, alert2, alert3}, then {alert1, alert2, alert4}.

An instance that knows about fewer alerts should not be able to remove alerts from the cluster's shared dedup state.

What did you see instead? Under which circumstances?

Five notifications. alert3 is notified a second time, by both peers that knew about it:

@ 2.0 {alert1, alert2} expected
@ 4.0 {alert1, alert2, alert3} expected
@ 10.0 {alert1, alert2, alert4} expected
@ 11.0155 {alert1, alert2, alert3} duplicate
@ 11.0159 {alert1, alert2, alert3} duplicate

Stable across repeated runs.

The cause

prev, ok := s[k]
if !ok || prev.Entry.Timestamp.AsTime().Before(e.Entry.Timestamp.AsTime()) {
    s[k] = e
    return true
}

So the instance that notified for alert4 writes an entry whose FiringAlerts omits alert3, and because that entry is newer it replaces the complete entry on both peers. On their next flush, DedupStage.needsUpdate finds alert3 missing from the entry, returns ReasonNewAlertsInGroup, and notifies again.

The local write path in nflog.Log already guards against a comparable problem:

if prevle, ok := l.st[key]; ok {
    // Entry already exists, only overwrite if timestamp is newer.
    // This may happen with raciness or clock-drift across AM nodes.
    if prevle.Entry.Timestamp.AsTime().After(now) {
        return nil
    }
}

The guard exists locally and is absent on the merge path.

System information

Linux 6.12.43+deb13-amd64 x86_64

Alertmanager version

Observed on 0.33.0. Both tests also fail on main at 294c544d (post v0.34.0). The merge logic dates to fd49dbb4 (2018-02-07, move to memberlist); the local write guard to 3d8e60de (2016-08-11). The asymmetry has existed since clustering was introduced.

Alertmanager configuration file

See test above.

Prometheus version

3.12.0

Prometheus configuration file

standard, config with list of targets.

Logs

Gossiped entries showing the regression:
firing_alerts: ...9885, ...1614, ...9876    {alert1, alert2, alert3}
firing_alerts: ...9885, ...1614, ...3183    {alert1, alert2, alert4}   newer, wins merge

CPU profiles

No response

Memory profiles

No response

Number of alerts

No response

Number of silences

No response

prometheus-operator checklist

• I have confirmed this issue is not caused by a missing feature in prometheus-operator.
• I have reproduced this issue when running Alertmanager directly (without prometheus-operator).