# VictoriaMetrics Integration [VictoriaMetrics](https://victoriametrics.com/) is a high-performance, scalable time series database that is designed for monitoring large-scale environments and efficiently storing and querying large volumes of metrics data. With the VictoriaMetrics integration, metric-triggered alerts – such as high latency or error rate breaches – from Prometheus-compatible systems are sent to on-call engineers via ilert. These alerts follow escalation policies for rapid triage and resolution of performance issues. ## In ilert: Create a VictoriaMetrics alert source 1. Go to **Alert sources** -> **Alert sources** and click **Create new alert source**.

2. Search for **VictoriaMetrics** in the search field, click the VictoriaMetrics tile, and then **Next**.

3. Give your alert source a name, optionally assign teams, and click **Next**. 4. Select an **escalation policy** by creating a new one or assigning an existing one.

5. Select your [Alert grouping](https://docs.ilert.com/alerting/configure-alerting/alert-sources#alert-grouping) preference and click **Continue setup**. You may click **Do not group alerts** for now and change it later.

6. The next page shows additional settings, such as customer alert templates or notification priority. Click **Finish setup** for now. 7. On the final page, an API key and/or webhook URL will be generated. You will need it later.

## In Prometheus Alertmanager > In order to be able to use VictoriaMetrics alerts and receive notifications, you need first configure and start alertmanager 1. Install Prometheus Alertmanager in any way that suits your needs. For more information about the alertmanager installation process please visit 2. Configure Alertmanager receivers in order to inform ilert every time there's an alert. In the example below replace the previously created VictoriaMetrics URL: {% code title="alertmanager.yaml" lineNumbers="true" %} ```yaml route: receiver: ilert receivers: - name: "ilert" webhook_configs: - url: "" send_resolved: true ``` {% endcode %} {% hint style="info" %} You could also split alert to high and low priority by creating two alert sources accordingly {% endhint %}

receivers:
  - name: "high-priority"
    webhook_configs:
      # high priority alert source url
      - url: "<your high priority alert source url here>"
        send_resolved: true
  - name: "low-priority"
    webhook_configs:
      # low priority alert source url
      - url: "<your low priority alert source url here>"
        send_resolved: true

## In VictoriaMetrics 1. Configure the `vmalert` alert rules in order to trigger alerts regarding the rule expression. For example: ```yaml groups: - name: vm-health rules: - alert: TooManyRestarts expr: changes(process_start_time_seconds{job=~".*(victoriametrics|vmselect|vminsert|vmstorage|vmagent|vmalert|vmsingle|vmalertmanager|vmauth).*"}[2m]) > 2 labels: severity: critical annotations: summary: "{{ $labels.job }} too many restarts (instance {{ $labels.instance }})" description: "Job {{ $labels.job }} (instance {{ $labels.instance }}) has restarted more than twice in the last 15 minutes. It might be crashlooping." ``` 2. Configure `vmalert`: ```sh ./bin/vmalert -rule=alert.rules \ # Path to the file with rules configuration. Supports wildcard -datasource.url=http://localhost:8428 \ # Prometheus HTTP API compatible datasource -notifier.url=http://localhost:9093 \ # AlertManager URL (required if alerting rules are used) -notifier.url=http://127.0.0.1:9093 \ # AlertManager replica URL -remoteWrite.url=http://localhost:8428 \ # Remote write compatible storage to persist rules and alerts state info (required if recording rules are used) -remoteRead.url=http://localhost:8428 \ # Prometheus HTTP API compatible datasource to restore alerts state from -external.label=cluster=east-1 \ # External label to be applied for each rule -external.label=replica=a # Multiple external labels may be set ``` ## FAQ **Will alerts in ilert be resolved automatically?** Yes, as soon as the Alertmanager sends a "RESOLVE" event, the associated alert is automatically resolved in ilert.