How ilert processes events & alerts
What happens to an event between your tool sending it and someone being notified, in the order ilert applies each check, and where every setting takes effect.
Every alert starts as an event, and every event goes through the same checks in the same order on its way to becoming a notification. Each check either passes the event on or stops it, and nearly every stop is recorded. When an alert paged nobody, or paged more than you expected, the order on this page tells you where to look.
The diagram follows an event that opens an alert. Events that accept or resolve an alert take a shorter path, described in Events that accept or resolve.
The event arrives
Your tool sends the event to an alert source, over HTTP to its integration URL or by email to its address. Over HTTP, ilert checks the integration key and the rate limit, queues the event, and answers straight away. The alert is created after the response, so the response cannot tell you whether one was.
202 Accepted
The event is queued. The response carries no alert ID. A few integrations receive 200 instead.
400 Bad Request
The integration key is missing or unknown, or ilert cannot parse the body.
429 Too Many Requests
The integration key has already sent 50 events this minute, the default limit. See API rate limits.
A rejected event is not recorded anywhere in ilert. The response in your tool's own logs is its only trace.
A disabled alert source still answers 202, and so does one in a maintenance window. The event is stopped at a later stage, and the reason is recorded only in the alert source logs.
An event flow can route it
If your tool sends to an event flow instead of an alert source, the flow runs first and decides which alert source receives the event. A path that never reaches a Route to alert source node creates nothing, and the event is recorded only in the event flow logs.
A routed event then goes through every stage below on its target alert source, as if your tool had sent it there directly.
The alert source decides whether it counts
The alert source applies its checks in this order:
Enabled. An event for a disabled alert source is recorded as Alert source disabled and goes no further.
Maintenance. While a maintenance window covers the alert source, every event is recorded as Under maintenance and goes no further, including events that would accept or resolve an open alert.
Custom processing rules. If the alert source has a Create alert rule, Accept alert rule or Resolve alert rule, the first rule the payload meets decides what the event does, whatever type it arrived as. An event that meets none of them is dropped when a rule exists for its own type. See Custom processing rules.
Templates. The alert summary, details, dynamic routing, alert key, links, priority mapping, severity mapping and affected services templates rewrite the event. Every later stage, event grouping included, works with the rewritten event.
Event filter. An event that does not meet Accept events that meet the following conditions is recorded as Dropped. See Event filter.
Events that accept or resolve
Of the templates, only the alert key template applies to an event that accepts or resolves. The event then looks for an open alert on the same alert source with the same alert key:
If there is none, the event is recorded as No open alert with key and changes nothing. It never creates an alert.
If there is one, the event filter still applies, and then the alert is accepted or resolved.
It joins an open alert or opens a new one
An event that passed every check becomes part of an alert, decided in this order:
Same alert key. If the alert source has an open alert, pending or accepted, with the same alert key, the event is added to it and recorded as Event appended to alert. No new escalation starts. A resolved alert is never reopened: an event carrying its key opens a new alert.
Event grouping. If no alert key matched, the alert source's event grouping can still add the event to an open alert. See the table below.
New alert. Otherwise the event opens a new alert, recorded as Alert created.
An event that carries no alert key is given a random one, so it never matches an existing alert by key. Integrations with rich payloads, such as Prometheus or Grafana, set the alert key for you.
Default grouping
Nothing. Only the alert key groups events.
Time-based, with a duration
The open alert, until that duration has passed since the alert was created
Time-based, Until alert is accepted
A pending alert
Time-based, Until alert is resolved
An open alert, pending or accepted
Group by content similarity (using ilert AI)
The most recently reported unresolved alert that meets the similarity threshold, until the grouping window has passed since that alert's latest event. See Intelligent alert grouping with ilert AI.
A new alert starts escalating
When an alert is created:
Its priority comes from the alert source's Notification priority. With High in hours, low priority out of hours, or the reverse, support hours choose between high and low. Support hours never hold an alert back.
A priority sent in the event itself is used only when the alert source's notification priority is High (with escalation), or when the event comes from ServiceNow or Grafana. Dynamic priority mapping applies with every setting.
Its escalation policy is the alert source's, unless a routing key selects another one.
Automatic alert actions for the alert source run, if their trigger and conditions match.
Escalation starts. With delayed escalation, the first escalation rule waits out the delay. Otherwise the responders of the first rule are notified at once. A low-priority alert stops at that first rule and never escalates further.
People are notified
Each person the escalation reaches is notified according to their own notification rules: their high-priority rules for a high-priority alert, their low-priority rules for a low-priority one, each with its own channel and delay. The notification limits below then apply.
Messages that ilert posts to Slack, Microsoft Teams or Google Chat channels are not notifications, and none of these limits apply to them.
Notification limits
These limits keep a burst of alerts from turning into a burst of calls to one phone. A notification that a limit stops is not lost without a trace: the alert's timeline records what happened to it.
Voice call
One call to the same phone number every 15 seconds, across all ilert accounts. A call due inside that gap is dropped, not delayed.
was dropped as the temporary rate limit for this target was reached
Voice call, SMS, email
Three notifications of the same kind, then the rest are held back and summarized. See Aggregation.
was suppressed and a bulk update notification was sent instead
Push, WhatsApp, Telegram
A notification with the same content as one sent to the same device or account in the previous 4 minutes is dropped.
was dropped as it was identified as duplicate in its related time period
Push, WhatsApp, Telegram
20 per device or account per clock minute. The rest are dropped. These channels are never aggregated.
was dropped as the temporary rate limit for this target was reached
Voice call and SMS about an incident
Three per person per channel in each 10-minute interval
was dropped as the temporary rate limit for this target was reached
Aggregation
Voice calls, SMS and email are counted in streams. Each stream belongs to one alert source, one person, one channel, one phone number or email address, and one kind of notification: a new alert, an escalation, an acceptance or a resolution.
The first three notifications in a stream go out as normal.
Every further notification is held back. Three minutes after the first one was held back, ilert sends a single summary in their place, for example There were at least 4 suppressed notifications in the last minutes regarding 2 new alert(s) in Acme Checkout API.
A stream starts counting from zero after 200 seconds without a notification, or once its summary is sent.
When only one notification was held back, the summary is skipped if it no longer matters: a new-alert notification whose alert has since been accepted or resolved, or an acceptance notification whose alert has since been resolved. The summary itself is exempt from the limits above, and does not appear in the alert's timeline.
On-call reminder notifications and test notifications are never aggregated.
FAQ
My tool got a 202 response, but no alert appeared
Open the alert source and check its alert source logs. Every event has an outcome there, such as Alert source disabled, Under maintenance, Dropped, No open alert with key, or Event appended to alert if it joined an alert that was already open. If your tool sends to an event flow, check the event flow logs first.
Why did I get a summary instead of a call for each alert?
ilert had already sent three notifications of the same kind from one alert source to the same phone number or email address of yours within a few minutes, so it held the rest back and summarized them. The alert's timeline marks each held-back notification. To get fewer notifications in the first place, group related events on the alert source.
Why did a second call not come through?
It was due within 15 seconds of an earlier call to the same phone number, so ilert dropped it. Only an aggregation summary is exempt from that gap. See Why are my ilert calls being discarded due to a temporary limit?
Does sending the same event twice create two alerts?
Not if the event carries an alert key and the first alert is still open: the second event is added to it. Without an alert key, each event gets a random key of its own, so each opens its own alert unless event grouping adds it to an open one.
Last updated
Was this helpful?