> For the complete documentation index, see [llms.txt](https://docs.ilert.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.ilert.com/integrations/inbound-integrations/prometheus.md).

# Prometheus Integration

Send Prometheus Alertmanager alerts to ilert: each alert group opens one alert, and ilert resolves it when the whole group has resolved.

[Prometheus](https://prometheus.io/) evaluates alert rules and hands firing alerts to [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/), which groups them and posts each group to its receivers. A Prometheus alert source in ilert gives you a URL to use as a webhook receiver. Each group opens one alert in ilert, and ilert resolves that alert when the group has resolved.

## Requirements <a href="#requirements" id="requirements"></a>

The configuration samples on this page need Alertmanager 0.22 or later. They were last run against Alertmanager 0.34 and Prometheus 3.15.

## In ilert: Create a Prometheus alert source <a href="#create-alarm-source" id="create-alarm-source"></a>

1. Go to **Alerting** → **Alert sources** and click **Create new alert source**.
2. Search for **Prometheus**, click its tile, and click **Next**.
3. Give your alert source a name, optionally assign teams, and click **Next**.
4. Choose who gets notified: select **Use existing escalation policy** and pick a policy, or **Generate new escalation policy** to start with a basic one that notifies you first. Click **Next**.

   <figure><img src="https://3394882078-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M76ygPnS4HUcFSX8ulm%2Fuploads%2Fgit-blob-8a281bf3931f5a1a531951ce97984d50989dd875%2Falert-source-escalation-step.png?alt=media" alt="The escalation step of the ilert alert source wizard, headed Who should be notified about alerts, with an escalation policy selected and a preview of its escalation rules below."><figcaption><p>Pick an existing escalation policy, or generate a basic one without leaving the wizard.</p></figcaption></figure>
5. Select **Default grouping** and click **Continue setup**. It groups events by the alert key the integration sets; for Prometheus, that key comes from the Alertmanager group. The other [grouping options](/alerting/configure-alerting/alert-sources.md#event-grouping) are easier to judge once events are arriving.
6. The next page shows more settings, such as notification priority and alert templates. Click **Finish setup** for now; you can change them later.
7. On the final page, ilert shows the **Prometheus URL** for this alert source. Copy it; you will paste it into Alertmanager in the next section.

## In Alertmanager: Add a webhook receiver <a href="#add-webhook" id="add-webhook"></a>

{% stepper %}
{% step %}

### Add the receiver and a route

In `alertmanager.yml`, add a receiver with the **Prometheus URL** you copied, and route alerts to it. This example sends every alert that no other route matches to ilert:

```yaml
route:
  receiver: ilert
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

receivers:
  - name: ilert
    webhook_configs:
      - url: 'YOUR_PROMETHEUS_URL'
```

The URL contains the alert source's integration key. To keep it out of the configuration file, put the URL in a file and replace `url` with `url_file: /path/to/file`. This needs Alertmanager 0.28 or later.
{% endstep %}

{% step %}

### Check and load the configuration

Check the file, then tell Alertmanager to reload it. Restarting Alertmanager works too.

```bash
amtool check-config alertmanager.yml
curl -X POST http://localhost:9093/-/reload
```

{% endstep %}

{% step %}

### Send a test alert

{% hint style="warning" %}
This opens a real alert in ilert, and the alert source's escalation policy notifies whoever it names.
{% endhint %}

```bash
curl -H 'Content-Type: application/json' \
  -d '[{"labels":{"alertname":"ilert test"},"annotations":{"summary":"Test alert from Alertmanager"}}]' \
  http://localhost:9093/api/v2/alerts
```

Alertmanager answers with HTTP 200 and an empty body.
{% endstep %}

{% step %}

### Check the alert in ilert

After `group_wait`, 30 seconds in this example, the alert appears under **Alerts** with the summary `1 FIRING: Test alert from Alertmanager`.

You do not need to clean up. Alertmanager resolves the test alert by itself after `resolve_timeout`, 5 minutes by default, and ilert resolves its alert when the next `group_interval` passes: about 6 minutes after you sent it, with these values.

If no alert appears, see [Troubleshooting](#troubleshooting).
{% endstep %}
{% endstepper %}

## How Alertmanager groups become alerts

Alertmanager posts groups, not single alerts, and ilert receives each post as one event. So one ilert alert can stand for several Prometheus alerts. With **Default grouping**, ilert handles each post like this:

| When Alertmanager                                              | ilert                                                                                      |
| -------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| Posts a firing group that has no open alert                    | Opens one alert and starts its escalation policy                                           |
| Adds a Prometheus alert to the group, or resolves some of them | Adds an entry to the alert's timeline. The summary stays as it was, and no new alert opens |
| Sends the group again after `repeat_interval`                  | Adds an entry to the timeline. An accepted alert stays accepted                            |
| Reports that the whole group has resolved                      | Resolves the alert                                                                         |

A later post can still change two things on an open alert. Its priority stays as it was when the alert opened.

* **Severity.** A severity in the post replaces the alert's severity, with no entry in the timeline.
* **Escalation policy.** A [routing key](#route-alerts-to-different-escalation-policies) that points to a different policy moves the alert to that policy, and the timeline records the move.

Three settings on the route decide how quickly this happens:

| Setting           | What it means for the alert in ilert                                                                                                                  |
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| `group_wait`      | How long a new group waits before its first post, and so how long before the alert opens                                                              |
| `group_interval`  | The shortest gap between posts for a group that has changed. A resolution waits for it, so the alert can stay open this long after the problem clears |
| `repeat_interval` | How often an unchanged, still-firing group is sent again                                                                                              |

Two consequences are worth planning for:

* **Resolving an alert in ilert does not stop it in Prometheus.** If the group is still firing, Alertmanager's next post opens a new alert. That post comes after `repeat_interval` at the latest. To hold an alert while you work on it, accept it. To quiet a known problem, [silence it in Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/#silences).
* **`group_by` decides how many alerts you get.** Grouping by `alertname` opens one alert for all instances of a rule. To open one alert for each Prometheus alert, use `group_by: ['...']`. Alertmanager recommends this only for low alert volumes, and every group then posts on its own, which counts against the alert source's [limit of 50 events per minute](/developer-docs/rest-api/alertsource-throttle.md).

## What ilert takes from a post

| In the ilert alert | Comes from                                                                                                                                                                                                                                   |
| ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Summary            | `N FIRING` and, once some have resolved, `M RESOLVED`, followed by the `summary` annotations of the firing Prometheus alerts, joined by `;` with duplicates removed. The value of a `gcp_project` or `cluster` label is added in parentheses |
| Details            | The alert name, summary and description, the `service`, `env`, `job` and `severity` labels, an `instance` label that is not a URL, the webhook version, the number of truncated alerts, and all labels of the firing Prometheus alerts       |
| Labels             | The labels the group's Prometheus alerts share, plus those of the first firing one                                                                                                                                                           |
| Priority           | The alert source's **Default priority**, unless you [map it](#set-priority-and-severity-from-the-severity-label). A `severity` label alone does not change it                                                                                |
| Severity           | A `severity` label set to a number from 1 to 5. Without one, the alert source's **Default severity**. A word such as `critical` counts as no severity unless you [map it](#set-priority-and-severity-from-the-severity-label)                |
| Links              | See [Add links to the alert](#add-links-to-the-alert)                                                                                                                                                                                        |

Give every alert rule a `summary` annotation. Without one, the ilert alert's summary shows the raw annotations, for example `1 FIRING: {}`.

## Set priority and severity from the severity label

Most alert rules label their severity with a word such as `critical` or `warning`. To turn that word into a priority and a severity in ilert, edit the alert source. Both mappings need a plan that includes alert templates, **Pro** or higher.

1. Under **Notification priority**, tick **Enable dynamic priority mapping**. ilert fills in the template `{{ commonLabels.severity }}` and maps the **Extracted value** `critical` to the **ilert priority** High, and `warning` to Low. Change these if they do not suit your labels.
2. Under **Alert severity**, tick **Enable dynamic severity mapping** and enter the same template. Click **Add mapping** for each value: map `critical` to SEV1 and `warning` to SEV3, for example.
3. Click **Save**.

A value with no mapping gets the alert source's **Default priority** and **Default severity**. See [Dynamic priority mapping](/alerting/configure-alerting/alert-sources.md#dynamic-priority-mapping) for the settings.

`commonLabels` only holds labels that every Prometheus alert in the group shares. If a group mixes `critical` and `warning` alerts, the template finds nothing and the ilert alert gets the defaults. Add `severity` to `group_by` to keep severities apart:

```yaml
route:
  receiver: ilert
  group_by: ['alertname', 'severity']
```

## Route alerts to different escalation policies

A Prometheus alert can choose its own escalation policy through a routing key. You do not need to turn on **Dynamic routing** on the alert source for this.

1. Give each target policy a [routing key](/on-call-management-and-escalations/escalation-policies.md#routing-key-optional).
2. In your Prometheus alert rule, add the label `ilert_routingkey` with that key as its value:

```yaml
groups:
  - name: checkout
    rules:
      - alert: CheckoutDown
        expr: up{job="checkout"} == 0
        for: 5m
        labels:
          ilert_routingkey: payments
        annotations:
          summary: 'Checkout instance {{ $labels.instance }} is down'
```

ilert reads the label from the first firing Prometheus alert in each post, so alerts with different routing keys must not share a group: a later post could move the open alert to another policy. Add the label to `group_by`:

```yaml
route:
  receiver: ilert
  group_by: ['alertname', 'ilert_routingkey']
```

If the key matches no policy, or the label is missing, the alert goes to the alert source's escalation policy.

## Add links to the alert

ilert adds these links to a new alert:

| Link text                                   | Comes from                                                                  | Notes                                                                                                                              |
| ------------------------------------------- | --------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| The value of the `urlLabel` label, or `Url` | The `url` label of the first firing alert                                   | The value must start with `http`. Use it for a runbook or a dashboard                                                              |
| `Prometheus External Url`                   | The URL Alertmanager reports for itself                                     | Start Alertmanager with `--web.external-url`. Otherwise this link names the machine's host name, which readers usually cannot open |
| `Prometheus Generator Url`                  | The alert's `generatorURL`, which opens the rule's expression in Prometheus | Start Prometheus with `--web.external-url`, for the same reason                                                                    |
| `Prometheus Instance Url`                   | The `instance` label                                                        | Only if the value starts with `http`                                                                                               |

For example, to link a runbook:

```yaml
groups:
  - name: checkout
    rules:
      - alert: CheckoutDown
        expr: up{job="checkout"} == 0
        for: 5m
        labels:
          url: https://runbooks.example.com/checkout-down
          urlLabel: Runbook
        annotations:
          summary: 'Checkout instance {{ $labels.instance }} is down'
```

## Send alerts to more than one alert source

Create one alert source in ilert for each destination, and one receiver for each in Alertmanager.

To send **different alerts** to different alert sources, match them in child routes:

```yaml
route:
  receiver: alert-source-1
  group_by: ['alertname', 'cluster', 'region']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - receiver: alert-source-1
      matchers:
        - severity="warning"
    - receiver: alert-source-2
      matchers:
        - severity="critical"

receivers:
  - name: alert-source-1
    webhook_configs:
      - url: 'PROMETHEUS_URL_OF_ALERT_SOURCE_1'
  - name: alert-source-2
    webhook_configs:
      - url: 'PROMETHEUS_URL_OF_ALERT_SOURCE_2'
```

To send **the same alert** to both, add `continue: true` to the first route. Each alert source then opens its own alert:

```yaml
route:
  receiver: alert-source-1
  group_by: ['alertname', 'cluster', 'region']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - receiver: alert-source-1
      continue: true
    - receiver: alert-source-2

receivers:
  - name: alert-source-1
    webhook_configs:
      - url: 'PROMETHEUS_URL_OF_ALERT_SOURCE_1'
  - name: alert-source-2
    webhook_configs:
      - url: 'PROMETHEUS_URL_OF_ALERT_SOURCE_2'
```

## Troubleshooting

Start with the Alertmanager log. If the entries below do not explain what you see, contact <support@ilert.com>.

### Alertmanager logs `invalid API key`

The Alertmanager log shows `Notify for alerts failed` with `unexpected status code 400` and `HTTP 400 invalid API key`. The key in the URL is wrong, or the alert source's key has been rotated. Copy the **Prometheus URL** from the alert source again and reload Alertmanager.

### Alertmanager logs status code 429

The Alertmanager log shows `Notify for alerts failed` with `unexpected status code 429`. The alert source received more than its [limit of 50 events per minute](/developer-docs/rest-api/alertsource-throttle.md). Alertmanager does not retry this response. It tries again at the next `group_interval`, so alerts and resolutions arrive late.

To send fewer posts, group by fewer labels or raise `group_interval`. To raise the limit itself, contact us through the page linked above.

### No alert appears and Alertmanager logs no error

Open the alert source in ilert and look at its [**Alert source logs**](/alerting/working-with-alerts/alert-source-logs.md). New entries can take a short time to appear.

**If the event is listed,** ilert received it and opened no new alert. Check these causes:

* The alert source is disabled or in a maintenance window. ilert accepts events in both states and opens no alert.
* The alert source's [event filter](/alerting/configure-alerting/alert-sources.md#event-filter) dropped the event.
* The event went to an alert that is still open. Look for that alert under **Alerts**.

**If the event is not listed,** Alertmanager did not send it. Check where Alertmanager routes the alert. This command prints the receiver for a given set of labels:

```bash
amtool config routes test --config.file=alertmanager.yml 'alertname="ilert test"'
```

If it names your ilert receiver, check that no silence or inhibition rule in Alertmanager covers the alert, and that `group_wait` has passed.

### An alert in ilert does not resolve

* **Part of the group is still firing.** The alert resolves only when every alert in the group has. Its timeline shows the count, for example `2 FIRING, 1 RESOLVED`.
* **`group_interval` has not passed.** The resolution is sent when it next passes.
* **The receiver sets `send_resolved: false`.** Webhook receivers send resolutions by default. Remove the setting.

### A new alert opens after I resolve one

The group is still firing in Prometheus, and Alertmanager's next post opened a new alert. See [How Alertmanager groups become alerts](#how-alertmanager-groups-become-alerts).

## FAQ <a href="#faq" id="faq"></a>

### Will alerts in ilert be resolved automatically?

Yes. Webhook receivers send resolutions by default, and ilert resolves the alert when the whole group has resolved.

### What happens to alerts if Alertmanager cannot reach ilert?

Alertmanager keeps retrying while the group is active, so a short outage delays the alert rather than losing it. A failed post is logged as `Notify attempt failed, will retry later`. In a test where the receiver answered HTTP 503 for more than 3 minutes, the alert arrived at the first retry after it recovered.

If Alertmanager itself stops, nothing is sent. To be alerted about that, use a [Prometheus heartbeat](/alerting/heartbeat-monitoring/prometheus-heartbeat-example.md).

## Further references

* [Prometheus heartbeat example](/alerting/heartbeat-monitoring/prometheus-heartbeat-example.md): get alerted when your Prometheus or Alertmanager stops running.
* [Import metrics from Prometheus](/incidents-and-status-pages/metrics/import-metrics-from-prometheus.md): show Prometheus metrics on your ilert status page.
* Alertmanager's [`webhook_config`](https://prometheus.io/docs/alerting/latest/configuration/#webhook_config) and [`route`](https://prometheus.io/docs/alerting/latest/configuration/#route) reference: every option for the configuration on this page.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.ilert.com/integrations/inbound-integrations/prometheus.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
