Alerts and Notifications
LiveNX supports real-time monitoring of the network and generates alerts to the user when anomalous network conditions occur. LiveNX creates visual changes in the system view that affect the status icons for devices and interfaces, and keeps a running log of all alerts generated by the system. To prevent false warnings from occurring, LiveNX provides the user with the flexibility to define the thresholds that define the anomalous network condition.
LiveNX displays all alerts in a real-time fashion. Go to Tools > View Alerts.
The In-Application Alerts window retains the most recent 100 alerts and displays them with the most recent at the top of the window. Alerts no longer pertinent can be removed by selecting the alerts and either pressing the Delete key or right clicking and choosing Remove Selected Alerts. Fields are:
- Time – time of the alarm
- Severity – choices are Emergency, Alert, Critical, Error, Warning, Notice, Info and Debug. Default is Warning (severity choices are covered in the following Configure Alerts section) • Device – device name
- Alert Type – alert type definitions and thresholds are covered in the following Configure Alerts section
- Details – provides additional details about the alert including cleared status, interface name and threshold violations.
Enable the checkbox to bring the window to the front when a new alert is received or to beep when a new alert is received. Default for both is disabled.
- Clear List – clicking on this button immediately clears the In-Application Alert window
- Export List – allows the user to store the alert information in a .csv format
- Historical Search – provides the user with historical and sorting capability for the alerts
- Configure Alerts – allows user-defined thresholds and alert severity definitions
Both the View Alerts and Historical Alerts (see next section) can drill down to a time series report specific to that individual alert. In order to access the report, right-click on the alert in question and select Open Report. A time series report encompassing the previous and next thirty minutes from the time of the alert will be generated.
LiveNX supports user-defined alert filtering on the In-Application Alerts. Click on Tools > View Alerts and click on Historical search.
LiveNX supports five filter types; each is independently enabled or disabled. Default is disabled for all types except time, which is defaulted to the last hour since the dialog was opened. If no filters are selected, all results are returned.
- Filter by Time – create a time range to filter the alerts using the Start Time/ End Time dialog boxes
- Filter by Device – filter the alerts by using the dropdown menu to list only the desired device
- Filter by Alert Type – filter the alerts by selecting only a particular alert type using the dropdown menu
- Filter by Severity – filter the alerts by selecting one of the eight available severity labels. There is an additional option. Include Higher Priorities that will include all priorities above the selected severity level (i.e. If WARNING is selected, alerts of WARNING, ERROR, CRITICAL, ALERT and EMERGENCY severity will also be returned).
- Maximum Number of Results – limit the number of alerts viewed by selecting the dropdown for 100, 200, 500, 1000, 10,000 or 100,000 alerts. Default is 100.
Select Execute to return the desired historical search.
Helpful Tip: If the maximum number of results is reached for any query, narrow the scope and re-execute the query. The alerts returned from a query are not ordered and thus the list may be missing key alert values.
Use the magnifying glass and the adjacent text box to further filter the alerts via alphanumeric searches. This search bar has a range of options ranging from searching case sensitive to only searching over specific columns. Click on the magnifying glass to see all possible filters.
NOTE: This search only filters the list of alerts gathered, as a result of the desired filtering.
The current list of alerts can be exported in .csv format by right-clicking on any cell and selecting Export Data.
Alert thresholds and severity levels are user-configured with the Configure Alerts dialog box. Go to Tools > Configure Alerts or Tools
>View Alerts and click on the Configure alerts button at the bottom of the page.
LiveNX supports eight types of alerts and notifications: Emergency, Alert, Critical, Error, Warning, Notice, Info and Debug. Default is Warning.
Each alert can be enabled by clicking on the checkbox and using the dropdown to select the desired Alert type.
- The device alert is logged in the In-Application Alerts window and increments the Device Up/Down Alert count in the System Dashboard when the device SNMP polling status changes between responsive and unresponsive.
- Default for the Device Down alert trigger is disabled.
- The values in the CPU and memory thresholds are editable only if that alert is enabled (checkbox is checked).
- These alerts are logged in the In-Application Alerts window and increments the Device CPU/Memory Alert count in the System Dashboard when the Device CPU or memory usage state exceeds the defined threshold. The count is also incremented when the CPU or memory usage state falls within the defined threshold.
- The device alerts (turns red) in the System Tree View and the Topology View if either the device’s CPU or memory alert threshold is exceeded and the alert is enabled.
- Default for the CPU and Memory Device alert triggers are enabled and thresholds set at 80%.
Config Change and Access
- The config change alert is logged in the In-Application Alerts window and increments the Device Config Change Alert count in the System Dashboard when the device’s running config changed time is more recent than the startup config changed time.
- The commands sent by monitor only credentials alert is triggered whenever the system uses the monitor only credentials if that was specified for a device and these credentials are used to send commands to the device.
- The device configuration alert is logged in the In-Applications Alerts window and increments the Device Config Change Alert count in the System Dashboard when any device’s configuration is changed by LiveNX. The alert contains the device name, the username and the commands sent to the device.
- Default for the Config Change alert triggers are disabled.
- The interface unavailable alert is logged in the In-Application Alerts window and increments the Interface Up/Down Alert count in the System Dashboard when the interface SNMP polling status changes between responsive and unresponsive.
- The interface errors alert is logged in the In-Application Alerts window when the interface generates CRC, frame, overrun, ignore or abort errors.
- Default for the interface error triggers are disabled.
- The values in the Interface drop, Class drop and Class-default drop thresholds are editable only if that threshold is selected. Note that the Interface drop rate is in packets per second, while the Class drop and Class-default drop rates are in Kilobits per second (Kbps).
- Click on the Generate events only for selected interfaces checkbox to trigger the interface drop alerts only on the interfaces selected during the Add or Discover Device process. Default for the selected interfaces checkbox is disabled.
- The status icons for devices and interfaces will change only if the threshold desired is enabled (checkbox is checked).
- The QoS alert is logged in the In-Application Alerts window and increments either the Interface drop, Class drop rate or Class-default drop rate count when those respective rates exceed the user defined rates.
- Default for all QoS alert triggers are enabled. Default for Interface drop rates, Class drop rates and Class-default drop rates is 0.
To configure alerts in the Flow technology, go to Tools > Configure Alerts > Flow Triggers tab.
Each alert can be enabled by clicking on the checkbox and using the dropdown to select the desired Alert type. The values in the Medianet thresholds (min jitter, max jitter, mean jitter, bit rate, packet rate, packet loss and round trip time) and in the Applications (AVC) thresholds (network delay and retransmission count) are editable only if the alert is enabled (checkbox is checked). The Flow, Medianet, AVC, PfR and Medianet alerts are viewed in the Tools > View Alerts and in the Reporting > Flows > Dashboard. The threshold crossing alerts for PfRv3 are for delay, jitter, drop and unreachable.
IP SLA Triggers
To configure alerts in the IP SLA technology, go to Tools > Configure Alerts > IP SLA Triggers tab.
Each alert can be enabled by clicking on the checkbox and using the dropdown to select the desired Alert type. The values in the IP SLA thresholds are editable only if the alert is enabled (checkbox is checked). The IP SLA alerts are viewed in the Tools > View Alerts and in the Reporting > IP SLA > Dashboard. Default for all IP SLA Triggers is disabled.
To configure alerts in the LAN technology, go to Tools > Configure Alerts > LAN Triggers tab.
The spanning tree topology alert is enabled by clicking on the checkbox and using the dropdown to select the desired Alert type. This generates an alert for any spanning tree change across all VLANs in the system. The LAN alerts are viewed in Tools > View Alerts. The LAN alert details include the VLAN index and a description of the state change that generated the alert.
NOTE: Since many alerts can be generated, listening and learning spanning tree state changes are intentionally not reflected in the alerts. Spanning tree ports in those states will be considered to be in a blocked state.
To configure alerts in the Routing technology, go to Tools > Configure Alerts > Routing Triggers tab.
Each routing alert can be enabled by clicking on the checkbox and using the dropdown menu to select the desired Alert type. The EIGRP/OSPF/IS-IS (Enhanced Interior Gateway Routing Protocol/Open Shortest Path First/Intermediate System – Intermediate System) and the polling error alerts can be enabled or disabled independently. If a routing alert is generated, the alert displays whether it is a state change or polling error and describes the routing protocol, the IP address and the applicable state change. Please see the alert details in the Alert Notification Configuration section occurring later in this chapter. Default for both routing alerts is disabled.
Four classes of custom triggers can be configured:
- QoS Class
Custom triggers can be set to trigger notifications based on threshold exceptions such as QoS drops. A list of created triggers appears in the list box.
Tagged alerts can be created using the custom alert triggers. A QoS class, NBAR or Flow tagged alert is created by using a custom alert trigger and then filtering the alert trigger based on the user-defined attributes in the system device tree. These custom alert triggers can then be named or tagged to describe the specific alert based on the filter attributes chosen.
Alert Notification Configuration
Notification that an alert condition has been triggered can be conveyed in the following methods:
- Notification within LiveNX (in-application alert)
- Notification via e-mail
- Notification within LiveNX and via e-mail
To suppress multiple alerts of the same condition that occur within a given timeframe, select the Ignore repeated alerts within the following interval check box. For example, when CPU usage spikes beyond the set threshold, the alarm could repeat itself six times within a minute, depending on the polling cycle set for the device (e.g., 10-second polling). In this situation, a single alarm per minute would likely suffice.
NOTE: An exception is made for the NSEL flow denied event occurred alert. The NSEL flow denied event alert will trigger no matter what ignore interval is selected. This exception was made to continue to record alerts for the instance where the security device alerts on multiple flow denied event occurrences for different flows within the same ignore interval.
E- mail alert notifications are sent when the first of the following conditions has been met:
- Total number of alerts reaches 200
- Maximum send delay timer – time since last alert reached maximum delay (default is 5 minutes)
Alerts generated by LiveNX can be sent to a Syslog server. Set up the Syslog server location and message format using the Syslog tab.
Message priority/level is set in the Trigger windows for each of the alert types.
Status Bar Alerts
The status bar at the bottom of the LiveNX screen includes an alert status icon
Alerting on the Web UI
With 6.2.0 release onwards, users can configure email alerting from the WebUI. A prerequisite for alerting to setup to an email address is to setup the SMTP Server in the Email Configuration page. The Email configuration can be found on the Server Settings page. Each time an alert is triggered an email will be sent to the addresses the user configured. The information contained will vary with the type of alert sent.
Alert Management: User can configure an email address to where the alerts need to be sent at the LiveAction Configure ->Alert Management section. The following alerts can be configured in this release. Alerts are formatted to support both standard email clients and PagerDuty consumption.
WAN Interface Percent Utilization Threshold: Alert whenever any WAN interface in the system exceeds a certain percent utilization threshold (either inbound or outbound). Note the following properties are user defined within LiveNX and must be set per interface in order to receive this alert for a particular interface:
- WAN tag on an Interface capacity (inbound and outbound)
Upon delivering the alert we will provide the following information about the alert:
- Device that the WAN interface is on
- Interface Name
- Direction of the interface (Inbound or outbound)
- Interface Utilization value that caused the alert to trigger
- Interface Capacity
- Threshold (user defined threshold to describe when to trigger an interface utilization alert)
- Time detected when LiveNX first detects this alert
BGP Peer Connection Change: This alert watches the BGP Peer Table of all devices polled and will automatically respond to devices being added or removed from LiveNX. User can set which severity this alert should be tagged with (info, warning critical). User can set a list of email addresses that should receive the BGP Peer Change Alert. User must have seen the peer drop out of established first to send this alert. No alerts are sent for previously unseen connections being established. To identify that this is a reconnecting peer we look for any connection with the same peer remote address as the dropped connection to be established. This is due to the peer identifier being set to 0.0.0.0 when the connection state becomes idle. To make sure dropped connections that never recover are not held in memory forever they are dropped after a day. This means that we will end up sending an alert again if the connection remains down for over 24 hours
An alert will be sent when
• A BGP peer on any polled device leaves the established state
SD-WAN Path change
In order for a path change to occur in the network, the customer must have the network setup in the following ways: IWAN/PfR enabled network with at least 1 Master Controller (MC) and 2 Border Routers (BR)
When a path change occurs in a Cisco environment with IWAN/PfR, we will receive a particular flow to identify that a route change occurred. This could be treated as a “set” of the event/route change. Our current definition of the alert is to treat every single instance of these records as an alert. It is understood this is noisy, and potentially inaccurate, but is a first example of how to define the alert. This alert can be enabled/disabled. Upon enabling this, all devices in the system will be processed and any device which exports the particular Cisco template will trigger the alert. The details of this flow are as follows:
Upon delivering the alert (to e-mail, web ui, etc) we will provide the following information about the alert:
- Time (per Node) recognized that the Alert was first seen
- Device the alert was received from
- Time (per device) recognized for when this alert occurred
- Master Controller IP Address + LiveNX Site based on IP Mappings
- Border Router IP Address + LiveNX Site based on IP Mappings
- DSCP Marking + LiveNX PfR App Group based on Lookup
- Destination Prefix of offending traffic + LiveNX Site based on IP Mappings
- New Service Provider the traffic is going over**
- VRF ID
The following levels of severity can be configured as user alert.