Section 8 - Events & Alerts
8.1 Overview
Mutiny has a number of built-in methods for displaying Events and/or sending Alerts when a property passes a threshold, or when an agent detects a Warning or Critical condition on a node. Events can be displayed in various ways depending on custom view settings, and Alerts can be delivered by email or SMS. Alerts can also be forwarded to other systems using SNMP Traps, v2 Informs, custom email templates, or custom-built APIs to upstream systems such as ServiceNow.
When a threshold is passed, Mutiny creates an Unconfirmed Event and places it in the Event table. Mutiny then checks whether the Event is transient. If the value returns below the threshold before the Event threshold time expires, it is treated as transient and marked as closed.
If the property is still over threshold after the Event threshold time has passed, it is considered a real Event and changes to a Confirmed Event. Mutiny then checks whether a No Alerts Period applies (see 8.2.2) to determine whether the Event should be ignored at that time. If not suppressed, it is upgraded to an Open Event.
Mutiny then updates Open Events views and any wallboards that contain the node to include the node and its alerting property, and checks whether there are any Contacts configured for this Event/Alert type.
For each listed Contact, Mutiny creates an Alert queue item. If the Contact is On Shift and has a zero delay for this Event type, the Alert is processed immediately. Any Contact that is Off Shift or has a delay configured will have the Alert queued, ready to send when the delay expires or the Contact becomes On Shift. You can see processed and queued items listed in the Alert Log.
The Alert can be tracked from the Related Alerts panel at the bottom of each Event page. Clicking the ID or open time expands the Alert tracking page.
Queued items remain queued until one of the following occurs:
- User is back on shift – Alert is sent
- User's delay expires – Alert is sent
- Property returns to OK – Alert is cancelled
- Event is cancelled because the node has been placed into Maintenance Mode
- Event is older than 365 days – Alert is cancelled (but will be automatically regenerated if the condition still exists)
Once a property returns to OK, this transition is recorded in the Event Log.
8.2 Thresholds
Most properties can have thresholds set against them. When a threshold is passed, Mutiny creates an Event. During discovery, default thresholds are set for most items. To change these, open the relevant property and adjust the threshold values.
8.2.1 Event Thresholds
Event thresholds are dwell times used by Mutiny to filter out transient events. The Event Thresholds panel is at the bottom of the node's Configure panel.
Mutiny will wait for the configured threshold time before treating an Event as real. This reduces false positives so Contacts treat Alert messages with consistent importance.
8.2.2 No Alerts Periods (see also 8.9 Maintenance Mode)
In addition to Event Thresholds, Mutiny can suppress the sending of Alerts during predefined No Alerts periods. These can be used for planned maintenance windows, regularly scheduled restarts, or other known noisy periods. Two time periods can be set for each day, as well as the property affected.
8.3 Contacts
Contacts receive the Alerts generated by the methods described above. To administer Contacts, navigate to the Contacts administration screen in the Admin section.
A Contact can be an individual, an email distribution group, an API POST endpoint, or another ticketing system destination. Each Contact has a settings panel containing options to receive certain types of Alerts, editable shift patterns, and fields for an email address and/or mobile number for SMS paging.
Alerts raised while a Contact is Off Shift are queued until the next On Shift time.
Ticking either of the All Alerts options will send all Alerts raised on the system and overrides shifts and per-alert settings.
8.4 Receive All Alerts with Tracked Views and Track Events
This is the simplest method of receiving Alerts generated by your Mutiny system. By default, Receive All Alerts is set to No and the alerting methods described below apply. If you wish to receive all Critical Alerts, tick Critical only and press Update. Alternatively, select Warning and Critical to receive every Alert generated for any monitored device. This is commonly used when forwarding all Alerts to a ticketing system.
Tracked Views and Track Events provide a more selective method of receiving Alerts from groups of nodes you have arranged into custom views. Click [Track Views] to open the view selection panel.
Select the views you want to receive Alerts from. Click [Track Events] to open the events selection panel and choose which event types you wish to receive. You can configure different delays per event type, and set repeats if required.
8.5 Event Alert Copy Templates
The Event Alert Settings button at the bottom of the Contacts panel contains event settings for all properties. This panel provides a quick method for applying a template of Alert settings for each Contact.
Alert settings can be configured per property, and the list boxes at the bottom of the panel allow copying settings to nodes or to views of nodes.
8.6 Alert Actions (Deprecated)
Located at the bottom of each Event panel is the Alert Actions section.
If enabled, this allows an Event to be forwarded to another system via SNMP Trap, v2 Inform, or (if available) a custom action. Note: Contacts can be configured as API endpoints, and it is generally recommended to use an API Contact instead.
8.7 Email Alerts
Email Alerts are the main method of communicating issues to Contacts. They can be customised to include additional properties as well as the Alert message itself. Templates can be tailored for helpdesk ingestion, email-to-SMS gateways, or internal workflows. The subject line can be prefixed with a keyword in System Configuration, and templates can also be customised per Contact if required.
8.7.1 Email Alert Templates
To edit email templates, open the template editor from the bottom of the Contacts page.
To create a new template, enter the name you wish to use and select an existing template to base it on. Press Update and your new template will open in the editor.
The top half is the editor and the bottom half shows a live preview of the output. You can enter free text and embed variables using the variable drop-down at the top. Press Update to save changes. Your new template is now available for selection in the Contact record.
The default Short format template produces an Alert similar to this:
8.8 JSON Templates for API Contacts
API templates are JSON-formatted blocks that allow Mutiny to POST event data to an external system.
8.9 SMS Alerts
In addition to email Alerts, Mutiny can send SMS Alerts through a modem and PSTN line, or via an email-to-SMS gateway.
SMS templates are editable like email templates, but are restricted by message length.
Using a combination of email Alerts and SMS Alerts allows flexible alerting procedures based on time, criticality, and escalation.
8.10 Maintenance Mode for Groups and Nodes
Maintenance Mode allows an Administrator to temporarily exclude monitored devices (nodes) from the Event/Alert process.
To recap: an Event occurs when a monitored property (e.g. disk usage, memory, ping response, etc.) changes its Status from OK to Warning or Critical. Once the Event threshold has been exceeded (2 minutes by default), the Event becomes Open and remains Open until the property returns to OK or is no longer monitored. While an Event is Open, Mutiny can (if configured) send Alerts (email, SMS, or Traps) to one or more Contacts.
There are times when it is desirable to exclude a device from this process, such as during scheduled or emergency maintenance. While this has always been possible, it can be labour-intensive and there is limited visibility to indicate that a node has been excluded. Maintenance Mode addresses this by providing quick selection and clear visual indicators.
8.10.1 Group Mode
To enable (or disable) Maintenance Mode you must be logged on as an Administrator or Super-Admin, as this is managed from the Node Manager. From the menu bar, select [Nodes] => [Manager] and click the [Maintenance] button.
Click [Enable Maintenance] and select the nodes and/or Views to include. You can choose whether to exclude nodes from both Event logging and Alerting, or to suppress Alerts only. Select a duration from the drop-down selector and click [Apply].
Selected nodes are automatically placed into a Smart View named In Maintenance, and the node status icon is replaced by a spanner symbol.
Any already-open Events from nodes placed into Maintenance Mode will be automatically closed within 1 minute and removed from wallboards.
While the node is in Maintenance Mode, no Alerts will be sent from Events on that node. Depending on the selected option, the node may also be prevented from creating new Events. Note that monitoring continues and graphing will not show gaps.
Maintenance Mode automatically ends when the chosen duration expires. To remove nodes from Maintenance Mode early, return to Node Manager, click [Maintenance], choose [Exit Maintenance], select nodes, and click [Apply].
Once nodes exit Maintenance Mode, any Events will quickly open (or re-open) and reappear on wallboards. Any configured Alerts will be sent once alerting conditions are met.
8.10.2 Single Node Maintenance Mode
To put a single node into Maintenance Mode directly from a view or event screen, right-click the node and select Maintenance Mode, then choose either Alerts only or Events and Alerts.
To exit Maintenance Mode, right-click the node and select Exit Maintenance.
