VictoriaMetrics Integration
VictoriaMetrics is a high-performance, scalable time series database that is designed for monitoring large-scale environments and efficiently storing and querying large volumes of metrics data.
In ilert: Create a VictoriaMetrics alert source
Go to Alert sources -> Alert sources and click Create new alert source.
Search for VictoriaMetrics in the search field, click the VictoriaMetrics tile, and then Next.
Give your alert source a name, optionally assign teams, and click Next.
Select an escalation policy by creating a new one or assigning an existing one.
Select your Alert grouping preference and click Continue setup. You may click Do not group alerts for now and change it later.
The next page shows additional settings, such as customer alert templates or notification priority. Click Finish setup for now.
On the final page, an API key and/or webhook URL will be generated. You will need it later.

In Prometheus Alertmanager
In order to be able to use VictoriaMetrics alerts and receive notifications, you need first configure and start alertmanager
Install Prometheus Alertmanager in any way that suits your needs. For more information about the alertmanager installation process please visit https://prometheus.io/docs/alerting/latest/alertmanager/
Configure Alertmanager receivers in order to inform ilert every time there's an alert. In the example below replace the previously created VictoriaMetrics URL:
route:
receiver: ilert
receivers:
- name: "ilert"
webhook_configs:
- url: "<your alert source url here>"
send_resolved: true
receivers:
- name: "high-priority"
webhook_configs:
# high priority alert source url
- url: "<your high priority alert source url here>"
send_resolved: true
- name: "low-priority"
webhook_configs:
# low priority alert source url
- url: "<your low priority alert source url here>"
send_resolved: true
In VictoriaMetrics
Configure the
vmalert
alert rules in order to trigger alerts regarding the rule expression. For example:
groups:
- name: vm-health
rules:
- alert: TooManyRestarts
expr: changes(process_start_time_seconds{job=~".*(victoriametrics|vmselect|vminsert|vmstorage|vmagent|vmalert|vmsingle|vmalertmanager|vmauth).*"}[2m]) > 2
labels:
severity: critical
annotations:
summary: "{{ $labels.job }} too many restarts (instance {{ $labels.instance }})"
description: "Job {{ $labels.job }} (instance {{ $labels.instance }}) has restarted more than twice in the last 15 minutes.
It might be crashlooping."
Configure
vmalert
:
./bin/vmalert -rule=alert.rules \ # Path to the file with rules configuration. Supports wildcard
-datasource.url=http://localhost:8428 \ # Prometheus HTTP API compatible datasource
-notifier.url=http://localhost:9093 \ # AlertManager URL (required if alerting rules are used)
-notifier.url=http://127.0.0.1:9093 \ # AlertManager replica URL
-remoteWrite.url=http://localhost:8428 \ # Remote write compatible storage to persist rules and alerts state info (required if recording rules are used)
-remoteRead.url=http://localhost:8428 \ # Prometheus HTTP API compatible datasource to restore alerts state from
-external.label=cluster=east-1 \ # External label to be applied for each rule
-external.label=replica=a # Multiple external labels may be set
FAQ
Will alerts in ilert be resolved automatically?
Yes, as soon as the Alertmanager sends a "RESOLVE" event, the associated alert is automatically resolved in ilert.
Last updated
Was this helpful?