The following article exclusively pertains to a Graylog Security feature or functionality. Graylog Security is a part of the Graylog centralized log management platform and requires a separate license. Contact the Graylog Sales team for more information on this product.

Anomaly detection plays a vital role in managing modern, large-scale distributed systems. Within these systems, analyzing data aggregations can be a significant pain point, and sometimes potential issues and threats slip into these systems undetected.

Anomaly Detection is a Security tool that utilizes your Graylog environment powered by Illuminate. (See Illuminate documentation on the Anomaly Detection add-on technology pack for more information.) This tool's primary purpose is to help you detect outliers in a dataset by running Artificial Intelligence / Machine Learning (AI/ML) behavioral analysis, allowing you to receive alerts whenever something deviates from its usual behavior or operates outside normal levels based on an understanding of your unique log data. As a result, you and your team become empowered to navigate cyber threats proactively, identify unusual activities, and take steps toward mitigating anomalies within your IT environment.

The process begins when Graylog Illuminate receives your log data, then normalizes and enriches it. Graylog then feeds the enriched data into the Anomaly Detection tool, which breaks the data up into time slices and looks for data points outside of the expected range based on your historical data. When anomalous data points are detected, these messages are logged into a special anomaly index in your Graylog instance. You can then create alerts regarding these anomalies based on your configuration settings. Additionally, these anomalies are represented on Security dashboards with various customizable widgets, offering interactive and actionable analytics.

This user guide helps you prepare for and set up your Anomaly Detection tool.

Prerequisites

Warning: For Graylog versions prior to 5.x, do not upgrade to OpenSearch 2.0+ as this will break your Graylog instance!
  • A valid Graylog Security license is also required. Contact the Graylog Sales team for more information on purchasing and downloading this license. (See the Anomaly Detection Illuminate technology pack for more details.)
  • Illuminate 2.2.2+ is also required to run the detection tool. New users can install the product or upgrade an existing instance as needed.
  • You will need live data constantly flowing into the server for the AI/ML behavioral analysis to function properly. It is recommended that you have at least a day's worth of ingested data on the server before enabling any anomaly detectors.

Enable Anomaly Detectors

Using Anomaly Detection requires no prior experience in machine learning as the user interface is intuitive and provides numerous descriptors. Configuring the tool begins by switching the toggles to enable any of the available detectors offered.

  1. After you have purchased the Graylog Security license and installed or updated your Illuminate product, you can then install the Anomaly Detection technology packs.

  2. Navigate to Enterprise > Illuminate then search for "Anomaly Detection."

  3. Select and enable both the Anomaly Detection Add-on and the Anomaly Detection Spotlight content packs.

  1. In Graylog, select the dropdown menu on the Graylog logo and click Security. You have now changed your view to the Security layout where all your security-related content is located.

  1. Now, select Anomalies > Anomaly Detectors to see available anomaly detectors. Depending on your organization's needs and preferences, enable the required detectors by clicking on the toggle so that running state is shown.

Hint: When multiple detectors are turned on, initialization is queued and performed sequentially.

A full list of all the available detectors and their functions may be reviewed in the anomaly detector user guide.

Note that starting multiple detectors simultaneously can be resource intensive on your OpenSearch nodes. The number of detectors allowed to be initializing at a time can be configured with the graylog config value opensearch_anomaly_max_concurrent_initializations=X, with the default value being 1. Hence, we recommend that you change the concurrent initializations value with caution and be sure that your clusters are adequately resourced to handle it gracefully.

Interpret the Data

When the anomaly tool detects anomalies in the log data, it pulls that data into the Anomaly Detection processor. The data is then deconstructed into human-readable and decluttered messages known as Graylog anomaly event messages, which will flow through the standard event message processing flow. These anomaly event messages summarily give you precise details of detected anomalous events.

Hint: For a complete list of these event message types and their definitions, review the anomaly event message index.

All anomaly event messages have common fields and additional fields depending on which detector the messages originate from. These additional fields tell you which entity (user, host, etc.) demonstrated anomalous behavior. Different detectors, like anomaly_detector_name, will add different fields.

Use Case

Let's look at an example.

Below is an example of an anomaly event message with both standard fields and anomaly fields populated. Think of an anomaly event message simply as how the Graylog anomaly detection tool represents data ingested behind the scenes.

In this example, an anomaly was detected by the Windows Event Log File Permissions Change Spike detector with an anomaly confidence value of 0.99, meaning that the detector is quite certain that the detected event is anomalous. The full event message provides further details:

Copy 
anomaly score: 3.99 (Indicates relative severity of an anomaly. The higher the score, the more anomalous a data point is. With an anomaly score of 3.99, this represents a relatively high anomalous data point)

anomaly grade: 1 (This is a severity level scale, ranging from 0 to 1. 0 is ‘not anomalous’ and a non-zero value represents the relative severity of the anomaly. With a value of 1 in this case, this is a high severity anomaly.

anomaly_file_perm_change_count: 3  (This is an additional field value which is specific to the Windows Event Log File Permissions Change Spike detector, and this value represents the number of times a file was permanently changed.

user_name: (This is an additional field value that is specific to the Windows Event Log File Permissions Change Spike detector and this value provides information on the username where the detected anomaly originated from. 

Here, the tool provides filtered data in the anomaly event message, further empowering you to locate the needle in the haystack within your IT infrastructure.

Create Anomaly Events and Alerts

Now that you know what Graylog Anomaly Detection is and how to enable it, the next step is to utilize the anomaly event messages to create events and alerts.

Using the example of the Windows Event Log File Permissions Change Spike above, let's assume you want to be alerted any time there are changes to file permissions in a given environment’s Windows hosts. You could begin by creating the search query:

Copy 
anomaly_detector_name:wineventlog_file_permissions_change  AND anomaly_file_perm_change_count:>0 AND anomaly_confidence:>.01 

This query would then search log data within the set parameters. Now, you can create an alert that triggers a specific alert, like an email or Slack message, to a specific group of users. For full instructions on this process, see the events and alerts user guide.

Generate Events from Anomaly Detectors

By default when you enable an anomaly detector, Graylog will generate an event definition. You have the option to disable event definitions for enabled anomaly detectors by toggling the Enable event definition capsule button found in the Anomaly Detectors tab on the Anomalies page.

Use Case

In this example, the anomaly detector for the Fortigate - Unusual Data Transfer is enabled, while its corresponding event definition is disabled.

When a system-generated event definition is created by enabling an anomaly detector, this appears on the Events Definitions page. You can then add alerts to event definitions for when an anomalous event definition is triggered:

  1. Click the More button for a selected event definition.

  2. Select Edit from the drop-down options.

  3. Navigate to the Notifications section in the resulting screen and click the Add Notification button.

  4. Select the required alert type and configure the applicable parameters.