Prometheus heartbeat example
Monitor the liveness of your Prometheus and Alertmanager stack with an always-firing alert that pings an ilert heartbeat monitor.
This recipe monitors the health of your Prometheus + Alertmanager stack itself. The idea is a dead man's switch: you define a single Prometheus alert that is always firing and route it through Alertmanager to an ilert heartbeat monitor. As long as the whole pipeline is healthy, Alertmanager keeps pinging the heartbeat endpoint at a regular interval. The moment Prometheus stops evaluating rules, Alertmanager goes down, or the network to ilert breaks, the pings stop — and ilert raises an alert because the heartbeat is overdue.
The flow looks like this:
Prometheus (always-firing rule) → Alertmanager (webhook receiver) → ilert heartbeat endpointPrerequisite: Create a heartbeat monitor in ilert
Follow the steps in Heartbeat monitoring to create a heartbeat monitor. When choosing the interval, pick a value that is several times longer than the rate at which Alertmanager actually pings. The route in Step 3 below produces a ping every 60 seconds, so a 5-minute heartbeat interval gives Alertmanager about five chances to ping before ilert considers the heartbeat overdue.
After creating the monitor, click its row in Alerting → Heartbeat monitors and copy the Integration URL from the panel that opens. It has the following form:
https://beat.ilert.com/api/pings/YOUR_INTEGRATION_KEYStep 1: Define an always-firing alert in Prometheus
Prometheus does not send anything to Alertmanager unless an alert is firing. To generate a continuous signal, add an alerting rule whose expression is always true. Create a rule file, e.g. ilert_heartbeat_rules.yml:
groups:
- name: ilert-heartbeat
rules:
- alert: ilert
expr: vector(1)
labels:
severity: none
annotations:
summary: "Prometheus/Alertmanager liveness heartbeat for ilert"The expression vector(1) always evaluates to 1, so this alert fires continuously and never resolves. The alert name ilert is what Alertmanager matches on in Step 3.
Step 2: Load the rule and point Prometheus at Alertmanager
In your prometheus.yml, register the rule file and make sure Prometheus is configured to send alerts to your Alertmanager instance. These are the sections to add to your existing prometheus.yml — merge them in alongside your current global, scrape_configs, and other settings rather than replacing the whole file.
Reload Prometheus (SIGHUP or POST /-/reload) and confirm the ilert alert appears as FIRING under Status → Alerts in the Prometheus web UI.
Step 3: Route the alert to ilert in Alertmanager
In your AM.yml, add a route that matches the ilert alert and forwards it to the heartbeat endpoint as a webhook receiver. If you already have routes and receivers, merge the ilert entries into your existing route.routes and receivers lists rather than replacing the file. Set group_wait, group_interval, and repeat_interval on the route — all three together determine how often ilert is pinged, and leaving any of them at its default breaks the heartbeat. See Timing below for why.
Reload Alertmanager to apply the configuration.
Timing: why repeat_interval alone is not enough
Alertmanager's ping rate is governed by three settings, and the defaults are far too slow for a heartbeat:
group_wait
30s
How long Alertmanager holds a newly created group before sending the first notification
group_interval
5m
How often the dispatcher wakes the group up and pushes it into the notification pipeline
repeat_interval
4h
The minimum time that must have passed since the last successful send before the same notification is sent again
repeat_interval is a permission check, not a timer. Nothing is sent unless the dispatcher wakes the group up first, and that happens only every group_interval. On each wake-up the deduplication stage asks whether at least repeat_interval has elapsed since the last successful send; if not, the wake-up passes without a ping.
The effective ping rate is therefore the first multiple of group_interval that is strictly greater than repeat_interval. Setting repeat_interval: 50s while leaving group_interval at its 5m default gives you a ping every 5 minutes, not every 50 seconds — the dispatcher simply never wakes up more often than that. With group_interval: 30s and repeat_interval: 50s, the wake-up at 30 seconds is skipped (only 30 seconds have elapsed) and the one at 60 seconds sends, so pings land exactly once a minute.
How it works
The
ilertalert fires permanently, so Alertmanager's dispatcher wakes the group everygroup_interval(30 seconds above) and re-sends the notification on the first wake-up afterrepeat_interval(50 seconds) has elapsed — one ping every 60 seconds.Each notification is an HTTP POST to
https://beat.ilert.com/api/pings/YOUR_INTEGRATION_KEY, which counts as a ping to your ilert heartbeat monitor.As long as pings keep arriving within the monitor's interval, the heartbeat stays healthy and no alert is created.
If Prometheus, Alertmanager, or the connection to ilert fails, the pings stop. Once the heartbeat interval elapses without a ping, ilert raises an alert through the alert source and escalation policy you assigned to the monitor.
Troubleshooting: pings arrive exactly every 5 minutes
Symptom: the heartbeat occasionally goes overdue and ilert creates an alert, even though Alertmanager is healthy and repeat_interval is set to well under a minute.
Confirm: open the heartbeat monitor in ilert and look at the time between pings. If they are 300 seconds apart, the route is still using the default group_interval of 5m. A 5-minute heartbeat interval fed by 5-minute pings has no margin at all, so any delay in evaluation, delivery, or the network pushes a ping past the deadline.
Fix: add group_interval (and group_wait: 0s) to the heartbeat route as shown in Step 3. repeat_interval alone cannot make the dispatcher wake up more often.
Related articles
Heartbeat monitoringPrometheus IntegrationLast updated
Was this helpful?