VictoriaMetrics Integration

VictoriaMetrics is a high-performance, scalable time series database that is designed for monitoring large-scale environments and efficiently storing and querying large volumes of metrics data.

In ilert: Create a VictoriaMetrics alert source

1. Go to Alert sources -> Alert sources and click Create new alert source.

   
2. Search for VictoriaMetrics in the search field, click the VictoriaMetrics tile, and then Next.

   
3. Give your alert source a name, optionally assign teams, and click Next.

4. Select an escalation policy by creating a new one or assigning an existing one.

   
5. Select your Alert grouping preference and click Continue setup. You may click Do not group alerts for now and change it later.

   
6. The next page shows additional settings, such as customer alert templates or notification priority. Click Finish setup for now.

7. On the final page, an API key and/or webhook URL will be generated. You will need it later.

In Prometheus Alertmanager

In order to be able to use VictoriaMetrics alerts and receive notifications, you need first configure and start alertmanager

1. Install Prometheus Alertmanager in any way that suits your needs. For more information about the alertmanager installation process please visit https://prometheus.io/docs/alerting/latest/alertmanager/

2. Configure Alertmanager receivers in order to inform ilert every time there's an alert. In the example below replace the previously created VictoriaMetrics URL:

alertmanager.yaml

route:
  receiver: ilert

receivers:
  - name: "ilert"
    webhook_configs:
      - url: "<your alert source url here>"
        send_resolved: true

You could also split alert to high and low priority by creating two alert sources accordingly

alertmanager.yaml

receivers:
  - name: "high-priority"
    webhook_configs:
      # high priority alert source url
      - url: "<your high priority alert source url here>"
        send_resolved: true
  - name: "low-priority"
    webhook_configs:
      # low priority alert source url
      - url: "<your low priority alert source url here>"
        send_resolved: true

In VictoriaMetrics

1. Configure the vmalert alert rules in order to trigger alerts regarding the rule expression. For example:

groups:
  - name: vm-health
    rules:
      - alert: TooManyRestarts
        expr: changes(process_start_time_seconds{job=~".*(victoriametrics|vmselect|vminsert|vmstorage|vmagent|vmalert|vmsingle|vmalertmanager|vmauth).*"}[2m]) > 2
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.job }} too many restarts (instance {{ $labels.instance }})"
          description: "Job {{ $labels.job }} (instance {{ $labels.instance }}) has restarted more than twice in the last 15 minutes.
            It might be crashlooping."

2. Configure vmalert:

./bin/vmalert -rule=alert.rules \            # Path to the file with rules configuration. Supports wildcard
    -datasource.url=http://localhost:8428 \  # Prometheus HTTP API compatible datasource
    -notifier.url=http://localhost:9093 \    # AlertManager URL (required if alerting rules are used)
    -notifier.url=http://127.0.0.1:9093 \    # AlertManager replica URL
    -remoteWrite.url=http://localhost:8428 \ # Remote write compatible storage to persist rules and alerts state info (required if recording rules are used)
    -remoteRead.url=http://localhost:8428 \  # Prometheus HTTP API compatible datasource to restore alerts state from
    -external.label=cluster=east-1 \         # External label to be applied for each rule
    -external.label=replica=a                # Multiple external labels may be set

FAQ

Will alerts in ilert be resolved automatically?

Yes, as soon as the Alertmanager sends a "RESOLVE" event, the associated alert is automatically resolved in ilert.