Cortex Integration

Connect Cortex Metrics with ilert to route performance and observability alerts to on-call responders and escalate issues from Prometheus-based systems.

Cortex is a CNCF incubation project used in several production systems including Amazon Managed Service for Prometheus (AMP) and it provides horizontally scalable, highly available, multi-tenant, long term storage for Prometheus.

You can use our example prometheus setup to test the Cortex integration

In ilert

Create a Cortex alert source

1. Go to Alert sources -> Alert sources and click on Create new alert source

1. Search for Cortex in the search field, click on the Cortex tile, and click on Next.

   
2. Give your alert source a name, optionally assign teams, and click Next.

3. Select an escalation policy by creating a new one or assigning an existing one.

   
4. Select your Alert grouping preference and click Continue setup. You may click Do not group alerts for now and change it later.

   
5. The next page shows additional settings such as customer alert templates or notification prioritiy. Click on Finish setup for now.

6. On the final page, a Cortex URL will be generated that you will need later in this guide.

In Prometheus Alertmanager

In order to be able to use cortex alerts and receive notifications, you need first configure and start alertmanager

1. Install Prometheus Alertmanager in any way that suits your needs. For more information about the alertmanager installation process please visit https://prometheus.io/docs/alerting/latest/alertmanager/

2. Configure Alertmanager receivers in order to inform ilert every time there's an alert. In the example below replace the previously created Cortex URL:

alertmanager.yaml

receivers:
  - name: "ilert"
    webhook_configs:
      - url: "<your alert source url here>"
        send_resolved: true

You could also split alert to high and low priority by creating two alert sources accordingly

alertmanager.yaml

receivers:
  - name: "high-priority"
    webhook_configs:
      # high priority alert source url
      - url: "<your high priority alert source url here>"
        send_resolved: true
  - name: "low-priority"
    webhook_configs:
      # low priority alert source url
      - url: "<your low priority alert source url here>"
        send_resolved: true

In Cortex

1. Install Cortex in any way that suits your needs.

2. Configure the Cortex alert rules in order to trigger alerts regarding the rule expression. For example:

cortex-alert-rules.yaml

groups:
  - name: cortex-critical
    rules:
      - alert: PrometheusDown
        expr: sum(up{instance=~"prometheus.*"}) < 1
        for: 15s
        labels:
          severity: critical
        annotations:
          summary: Prometheus is down alert
          description: "Prometheus is down \n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

1. Configure the Mimir ruler to send alerts to an external alertmanager and point the alert rules folder:

cortex-config.yaml

ruler:
  alertmanager_url: http://alertmanager:9093
  enable_api: true

ruler_storage:
  backend: local
  local:
    directory: /etc/alertmanager

FAQ

Will alerts in ilert be resolved automatically?

Yes, as soon as the Alertmanager sends a "RESOLVE" event, the associated alert is automatically resolved in ilert.