OpenNMS Integration
OpenNMS is an open-source network monitoring and management platform that discovers your infrastructure, collects performance data, and turns events into alarms. With the ilert OpenNMS plugin, alarms are forwarded to ilert as alerts, so your on-call team is notified through SMS, phone calls, and push notifications, while acknowledging or clearing the alarm in OpenNMS keeps the ilert alert in sync.
System requirements
OpenNMS Horizon 35 (or higher)
Shell access to the OpenNMS server and access to the Karaf console over SSH on port
8101, default passwordadmin. From the OpenNMS host, or from anywhere the port is published, usessh admin@localhost -p 8101. On the Docker image, rundocker exec -it <container> ssh admin@localhost -p 8101unless you publish8101on the hostJDK 17 and Maven, only if you build the plugin from source instead of downloading a release
The plugin is built against OpenNMS Integration API 2.0.0, which Horizon ships since version 35. It does not load on anything older.
Locate your OpenNMS install root
This guide writes paths as $OPENNMS_HOME, which is the OpenNMS install root. The packages do not export that variable, so it is empty in a normal login shell: a command pasted as-is writes to /deploy/ at the filesystem root instead of into OpenNMS, and the plugin is never picked up. Look up your root and copy it before you continue.
RPM or Debian package
/usr/share/opennms
Docker image or tarball
/opt/opennms
With the Docker image, the install root is a path inside the container, so $OPENNMS_HOME on the host points at nothing. Every path below that starts with $OPENNMS_HOME is a path in the container: read and write it with docker exec or docker cp and spell it out as /opt/opennms. The deploy step and the upgrade FAQ give the Docker form explicitly.
In ilert: Create an OpenNMS alert source
Go to Alert sources -> Alert sources and click Create new alert source.
Search for OpenNMS in the search field, click the OpenNMS tile, and then Next.
Give your alert source a name, optionally assign teams, and click Next.
Select an escalation policy by creating a new one or assigning an existing one.
Select your Alert grouping preference and click Continue setup. You may click Do not group alerts for now and change it later.
The next page shows additional settings, such as custom alert templates or notification priority. Click Finish setup for now.
On the final page, copy the generated events URL verbatim. You will use it to configure the plugin on the OpenNMS server. It looks like this:
The integration key is part of the events URL, so the URL is both the address and the credential. There is no separate API key to configure. Treat the URL like a password, since anyone who has it can create alerts in your account. For this reason, the plugin never writes it to its log, not even at DEBUG level.
If different classes of alarm should reach different teams, create one alert source per team. Each one gets its own events URL, and you repeat the alert source settings once per alert source with a different filter.
In OpenNMS: Install the plugin
Get the plugin
Download opennms-ilert-plugin.tar.gz from the releases page of the ilert OpenNMS plugin and unpack it. Besides the .kar file, the archive contains the license and shasum256.txt, so you can verify the download:
Alternatively, build the plugin yourself, which requires JDK 17 and Maven:
The kar is created in assembly/kar/target/. Two kars land there: take opennms-ilert-plugin-<version>.kar, which is around 6 MB. The 3 KB org.opennms.plugins.ilert.assembly.kar-<version>.kar next to it is a Maven artifact, not the plugin, and deploying it installs nothing.
Deploy the plugin
Either way you end up with a file named opennms-ilert-plugin-<version>.kar. Copy it into the OpenNMS deploy directory, then tell Karaf to install the feature on every boot.
sudo cp and sudo tee write to the host filesystem, not into the container, so neither works here. Move both files in with docker cp instead, against the literal /opt/opennms root:
docker cp writes to the container's own layer, so both files are gone the next time the container is recreated. Keep deploy/ and etc/ on volumes, or bake the kar and the boot file into your image, if the plugin has to survive an image update.
The wait-for-kar value must match the filename of your .kar file without the .kar extension, including the version. kar:list in the Karaf console prints the name that Karaf actually registered. If the two do not match, the feature does not come up on boot and you have to install it manually after every restart.
Restart OpenNMS to pick up the plugin, or docker restart <container> on the Docker image. To avoid the restart, install the feature from the Karaf console instead. The boot file still makes sure that it survives future restarts:
Confirm the installation
Confirm that the feature is Started and the bundle is Active:
Nothing is forwarded yet, since the plugin has no alert source configured. That is the next step.
In OpenNMS: Configure the plugin
Alert source settings
Each ilert alert source is one plugin service with its own URL, filter, and hold-down delay. Create it from the Karaf console:
The --alias is only a label for this service and becomes part of the filename. The command above writes $OPENNMS_HOME/etc/org.opennms.plugins.ilert.services-production.cfg, and you can create that file yourself instead:
Either way, the change takes effect within seconds and does not require a restart. Repeat this step once per ilert alert source, giving each a different --alias.
url
yes
The events URL you copied in ilert. A service without a url does not start and logs an error.
jexlFilter
in practice, yes
Which alarms this service forwards. Without a filter, nothing is forwarded: the plugin logs a warning and stays idle.
holdDownDelay
no
An ISO-8601 duration such as PT5M, which delays alerts so that alarms clearing inside the window never page. Absent means send immediately.
Global settings
Every ilert alert carries a link back to the alarm in OpenNMS. The link defaults to 127.0.0.1, which is of no use on a responder's phone, so set it to a URL your team can actually reach. These settings apply to the whole OpenNMS instance, not to a single alert source:
alarmDetailsUrlPattern
http://127.0.0.1:8980/opennms/alarm/detail.htm?id=%d
Deep link to the alarm. Must contain exactly one %d for the alarm ID; a literal percent sign has to be escaped as %%.
client
OpenNMS
The label shown on that link in ilert. Worth changing if you run more than one OpenNMS instance.
A malformed pattern does not stop alerts. The plugin logs the problem once at startup and sends alerts without the link instead of not sending them at all.
Choose which alarms are forwarded
jexlFilter is a JEXL expression that is evaluated per alarm. The alarm is bound to the variable alarm, and any getter on the OpenNMS Alarm model is reachable as a property. The expression must evaluate to a boolean. An expression that throws, or that returns something else, drops the alarm and logs a warning.
Some starting points:
The plugin ships a Karaf command that runs an expression against the alarms currently in your database, so you can see what a filter would have caught without waiting for the next outage:
-c
Report only the count, which is useful for sizing a filter against a busy database.
-p
Print the ilert payload that each matching alarm would produce, without the alarm deep link.
-a <id>
Evaluate against a single alarm by ID and report whether that alarm would be forwarded.
-a is the fastest way to find out why a specific alarm did not page. Here a minor nodeLostService alarm falls below a major-and-above filter:
Alert priority
The plugin derives the priority of the ilert alert from the severity of the alarm. The mapping is fixed and there is no plugin setting for it:
CRITICAL, MAJOR
High
INDETERMINATE, CLEARED, NORMAL, WARNING, MINOR
Low
Priority decides how the alert notifies: a high priority alert runs your escalation policy, while a low priority alert notifies through each responder's low-priority notification rules and is never escalated.
This matters as soon as you widen jexlFilter below major. A WARNING or MINOR alarm creates an alert in ilert, which is easy to read as proof that the integration pages correctly, but the escalation policy never runs for it. If those alarms should page, override the priority on the ilert side with dynamic priority mapping on the alert source. Run opennms-ilert:eval-jexl -p to see which payload fields you can map from.
Delay alerts for short outages
Service pollers sometimes report brief outages that resolve themselves. With holdDownDelay set, the plugin waits before sending the alert. If the alarm clears inside that window, the alert is cancelled and nobody is paged:
The value is an ISO-8601 duration such as PT30S, PT5M, or PT1H. An unparseable value is reported in the log and treated as no delay, so a typo does not stop alerts from being sent.
Two details are worth knowing:
The deadline does not move. While an alert is held, further updates to the same alarm replace the pending payload but keep the original deadline. An alarm that escalates during the wait pages with its latest severity, and an alarm that updates every few seconds cannot push its own deadline out forever.
Only new alerts are delayed. If the alarm is acknowledged or clears while the alert is still held, nothing is sent to ilert at all.
Verify the integration
Raise a real alarm through the OpenNMS REST API. nodeLostService is a good choice, because its reduction key varies per interface and service, so each test creates a distinct alert instead of deduplicating onto the previous one. Use a nodeid that exists on your system:
A successful POST returns 202, and within a few seconds the alert appears in your ilert alert source.
nodeLostService raises a minor alarm, so the major-and-above filter from Alert source settings drops it and nothing reaches ilert. Widen the filter to alarm.severity.id >= 5 for the duration of the test, and set it back afterwards. The alert then arrives with low priority and does not run your escalation policy, which is expected here; see Alert priority.
<service> must be a plain string. The nested form <service><name>ICMP</name></service> is rejected with an HTTP 500.
Then exercise the rest of the life cycle:
Acknowledge the alarm in the OpenNMS UI. The ilert alert is accepted. Acknowledging does not resolve the alert.
Clear the alarm by sending the matching
nodeRegainedServiceevent with the samenodeid,interface, andservice. The ilert alert is resolved.
All three steps map onto one ilert alert, because the plugin uses the reduction key of the alarm as the ilert alert key.
Watch what the plugin does
Enable debug logging from the Karaf console:
Then follow $OPENNMS_HOME/logs/karaf.log. A successful delivery looks like this:
To keep debug logging enabled across restarts, add the following to $OPENNMS_HOME/etc/org.ops4j.pax.logging.cfg:
The plugin also reports on itself through OpenNMS events, which is the better signal for a dashboard or a long-running check:
uei.opennms.org/ilert/sendEventSuccessful— the event was delivered.uei.opennms.org/ilert/sendEventFailed— the plugin gave up after the third attempt. This raises an alarm of its own, which is cleared automatically by the next success.
Neither of these two events is ever forwarded to ilert, so they cannot cause a feedback loop.
FAQ
Will alerts in ilert be resolved automatically?
Yes, as soon as the alarm is cleared or deleted in OpenNMS, the plugin sends a RESOLVE event and the corresponding alert in ilert is resolved.
Will alerts in ilert be accepted automatically?
Yes, as soon as the alarm is acknowledged in OpenNMS, the plugin sends an ACCEPT event and the corresponding alert in ilert is accepted.
What happens if ilert cannot be reached?
ALERT events are attempted up to three times, waiting two seconds and then eight seconds between the attempts, which the log counts as attempt 1 of 3. Once the attempts are exhausted, the plugin raises uei.opennms.org/ilert/sendEventFailed locally, which creates an alarm containing the error detail. That alarm is worth wiring to a secondary notification path, and it clears automatically on the next successful send for the same alarm. ACCEPT and RESOLVE events are sent once and are not retried, because they are sent from the alarm life cycle callback, which must not block.
Nothing arrives in ilert and the log is quiet. What can I do?
This is almost always a missing or non-matching jexlFilter. Look for No 'jexlFilter' configured ... No alarms will be forwarded until one is set in karaf.log, then check your expression with opennms-ilert:eval-jexl -a <alarm-id> '<your filter>'.
The service does not start. What can I do?
Look for No 'url' is configured for iLert service pid: ... in the log and confirm that the configuration landed with config:list '(service.pid=org.opennms.plugins.ilert.services*)'.
The log shows iLert answered with status 401 or 404.
The URL is wrong or the alert source was deleted. Copy the events URL from your alert source in ilert again. To check a URL without raising an alert, post an empty JSON body to it. A valid URL answers 400 missing 'alertKey' field, which means the request reached the alert source and only the payload was wrong. A 404 means the URL itself is bad.
Deliveries fail with timeouts.
The OpenNMS host has to reach api.ilert.com over outbound HTTPS. Test this from the OpenNMS host itself, and keep in mind that a container has its own view of the network.
The feature disappears after a restart.
The wait-for-kar value in $OPENNMS_HOME/etc/featuresBoot.d/ilert.boot does not match the kar name. Compare it against kar:list in the Karaf console; it must be the filename of the .kar file without the .kar extension.
The link on the alert does not open.
The alarmDetailsUrlPattern points to a host that your browser cannot reach, see Global settings.
After an upgrade, the old version keeps running.
Karaf caches kar contents under $OPENNMS_HOME/data/kar/<kar-name>/ and does not overwrite a cached kar of the same version, so everything looks Active while the old code runs. Release versions avoid this. If you hit it with a snapshot build, clear the cache explicitly. From the Karaf console:
Then on the shell:
On the Docker image, run the same two removals inside the container. The wildcards need a shell, so pass them to sh -c rather than to rm directly:
Copy the new kar in afterwards and install the feature again. Removing the file from deploy/ before copying the new one in matters: copying over a filename that Karaf still holds state for makes its redeploy fail with The KAR ... is not installed, and the kar is then never extracted. To confirm which build is live, check Bnd-LastModified in bundle:headers <bundle-id> against the jar you deployed.
Can I forward alarms to more than one alert source?
Yes, create one service configuration per alert source, each with its own --alias, url, and jexlFilter.
Last updated
Was this helpful?