> For the complete documentation index, see [llms.txt](https://docs.ilert.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.ilert.com/alerting/heartbeat-monitoring/prometheus-heartbeat-example.md). # Prometheus Heartbeat Example This recipe monitors the health of your Prometheus + Alertmanager stack itself. The idea is a **dead man's switch**: you define a single Prometheus alert that is *always* firing and route it through Alertmanager to an ilert heartbeat monitor. As long as the whole pipeline is healthy, Alertmanager keeps pinging the heartbeat endpoint at a regular interval. The moment Prometheus stops evaluating rules, Alertmanager goes down, or the network to ilert breaks, the pings stop — and ilert raises an alert because the heartbeat is overdue. The flow looks like this: ``` Prometheus (always-firing rule) → Alertmanager (webhook receiver) → ilert heartbeat endpoint ``` ## Prerequisite: Create a heartbeat monitor in ilert Follow the steps in [Heartbeat monitoring](/alerting/heartbeat-monitoring.md) to create a heartbeat monitor. When choosing the interval, pick a value that is several times **longer** than the rate at which Alertmanager actually pings. The route in Step 3 below produces a ping every 60 seconds, so a 5-minute heartbeat interval gives Alertmanager about five chances to ping before ilert considers the heartbeat overdue. After creating the monitor, copy its **integration URL**. It has the following form: ``` https://beat.ilert.com/api/pings/${YOUR-APIKEY} ``` ## Step 1: Define an always-firing alert in Prometheus Prometheus does not send anything to Alertmanager unless an alert is firing. To generate a continuous signal, add an alerting rule whose expression is always true. Create a rule file, e.g. **ilert\_heartbeat\_rules.yml**: ```yaml groups: - name: ilert-heartbeat rules: - alert: ilert expr: vector(1) labels: severity: none annotations: summary: "Prometheus/Alertmanager liveness heartbeat for ilert" ``` The expression `vector(1)` always evaluates to `1`, so this alert fires continuously and never resolves. The alert name `ilert` is what Alertmanager matches on in Step 3. ## Step 2: Load the rule and point Prometheus at Alertmanager In your **prometheus.yml**, register the rule file and make sure Prometheus is configured to send alerts to your Alertmanager instance. These are the sections to **add to your existing `prometheus.yml`** — merge them in alongside your current `global`, `scrape_configs`, and other settings rather than replacing the whole file. ```yaml rule_files: - "ilert_heartbeat_rules.yml" alerting: alertmanagers: - static_configs: - targets: - "alertmanager:9093" ``` {% hint style="warning" %} The target `alertmanager:9093` is a **placeholder** — replace it with wherever your Alertmanager is actually reachable. `9093` is Alertmanager's default HTTP port, but the hostname depends on how you run it. {% endhint %} Reload Prometheus (`SIGHUP` or `POST /-/reload`) and confirm the `ilert` alert appears as **FIRING** under **Status → Alerts** in the Prometheus web UI. ## Step 3: Route the alert to ilert in Alertmanager In your **AM.yml**, add a route that matches the `ilert` alert and forwards it to the heartbeat endpoint as a webhook receiver. If you already have routes and receivers, merge the `ilert` entries into your existing `route.routes` and `receivers` lists rather than replacing the file. Set `group_wait`, `group_interval`, and `repeat_interval` on the route — all three together determine how often ilert is pinged, and leaving any of them at its default breaks the heartbeat. See **Timing** below for why. ```yaml route: receiver: default group_by: - job routes: - receiver: ilert matchers: - alertname="ilert" group_wait: 0s group_interval: 30s repeat_interval: 50s receivers: - name: ilert webhook_configs: - url: 'https://beat.ilert.com/api/pings/${YOUR-APIKEY}' send_resolved: false ``` Reload Alertmanager to apply the configuration. ### Timing: why `repeat_interval` alone is not enough Alertmanager's ping rate is governed by three settings, and the defaults are far too slow for a heartbeat: | Setting | Default | What it does | | ----------------- | ------- | ---------------------------------------------------------------------------------------------------------------- | | `group_wait` | `30s` | How long Alertmanager holds a newly created group before sending the first notification | | `group_interval` | `5m` | How often the dispatcher wakes the group up and pushes it into the notification pipeline | | `repeat_interval` | `4h` | The minimum time that must have passed since the last successful send before the same notification is sent again | `repeat_interval` is a permission check, not a timer. Nothing is sent unless the dispatcher wakes the group up first, and that happens only every `group_interval`. On each wake-up the deduplication stage asks whether at least `repeat_interval` has elapsed since the last successful send; if not, the wake-up passes without a ping. The effective ping rate is therefore the first multiple of `group_interval` that is strictly greater than `repeat_interval`. Setting `repeat_interval: 50s` while leaving `group_interval` at its `5m` default gives you a ping every 5 minutes, not every 50 seconds — the dispatcher simply never wakes up more often than that. With `group_interval: 30s` and `repeat_interval: 50s`, the wake-up at 30 seconds is skipped (only 30 seconds have elapsed) and the one at 60 seconds sends, so pings land exactly once a minute. {% hint style="warning" %} Do not set `group_interval` equal to `repeat_interval`. The elapsed time measured at each wake-up is a fraction of a second short of `repeat_interval`, so the send is skipped and the ping arrives a full `group_interval` late. `group_interval: 30s` with `repeat_interval: 30s` pings every 60 seconds, not every 30. Keep `repeat_interval` between one and two times `group_interval`, as in the values above. {% endhint %} ## How it works * The `ilert` alert fires permanently, so Alertmanager's dispatcher wakes the group every `group_interval` (30 seconds above) and re-sends the notification on the first wake-up after `repeat_interval` (50 seconds) has elapsed — one ping every 60 seconds. * Each notification is an HTTP POST to `https://beat.ilert.com/api/pings/${YOUR-APIKEY}`, which counts as a ping to your ilert heartbeat monitor. * As long as pings keep arriving within the monitor's interval, the heartbeat stays healthy and no alert is created. * If Prometheus, Alertmanager, or the connection to ilert fails, the pings stop. Once the heartbeat interval elapses without a ping, ilert raises an alert through the alert source and escalation policy you assigned to the monitor. ## Troubleshooting: pings arrive exactly every 5 minutes **Symptom:** the heartbeat occasionally goes overdue and ilert creates an alert, even though Alertmanager is healthy and `repeat_interval` is set to well under a minute. **Confirm:** open the heartbeat monitor in ilert and look at the time between pings. If they are 300 seconds apart, the route is still using the default `group_interval` of `5m`. A 5-minute heartbeat interval fed by 5-minute pings has no margin at all, so any delay in evaluation, delivery, or the network pushes a ping past the deadline. **Fix:** add `group_interval` (and `group_wait: 0s`) to the heartbeat route as shown in Step 3. `repeat_interval` alone cannot make the dispatcher wake up more often. ## Related articles {% content-ref url="/pages/-M9bd2vgpr1x\_yIPXlIE" %} [Heartbeat monitoring](/alerting/heartbeat-monitoring.md) {% endcontent-ref %} {% content-ref url="/pages/-M9ScNiNyfedtovre9wO" %} [Prometheus Integration](/integrations/inbound-integrations/prometheus.md) {% endcontent-ref %} --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.ilert.com/alerting/heartbeat-monitoring/prometheus-heartbeat-example.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.