Anomaly Detection

The following article exclusively pertains to a Graylog Security feature or functionality. Graylog Security is a part of the Graylog centralized log management platform and requires a separate license. Contact the Graylog Sales team for more information on this product.

Warning: Anomaly detection has been updated. Legacy detectors are no longer supported—contact Graylog Support for migration assistance. Custom anomaly detectors are disabled by default; contact Support to enable custom detection logic.

Anomaly Detection is a Graylog Security feature powered by Illuminate that identifies unusual patterns in log data. It analyzes historical data to establish a baseline of normal behavior and generates events when deviations exceed expected thresholds. Using this baseline, Graylog continuously evaluates incoming data and applies statistical and machine learning models to detect activity outside expected ranges.

Graylog provides anomaly detection capabilities in two ways:

Illuminate detectors
Configurable anomaly detectors (Anomaly Events)

Understanding how these detector types differ is key to configuring anomaly detection effectively. The following article explains how Illuminate detectors are installed and managed, and how to configure user-defined detectors for specific use cases.

Prerequisites

Before proceeding, ensure that the following prerequisites are met:

A valid Graylog Security license.
Install the following Illuminate Content Packs:
- Illuminate:Core:Anomaly Detection Add-on
- Illuminate:Core:Anomaly Detection Spotlight
Install Illuminate version 7.1 or later.
Confirm that relevant log data is flowing into Graylog.

Highlights

The following highlights provide a summary of the key takeaways from this article:

Use Illuminate detectors for quick deployment of common detection scenarios with minimal configuration.
Use configurable Anomaly Events detectors (Impossible Travel, Log Volume Anomalies) when the specific use case is required.
Manage all detectors from the Event Definitions page.

Install Illuminate Anomaly Detection Packs

Graylog provides predefined anomaly detectors through Illuminate content packs. These detectors are product-agnostic and provide broad coverage for common detection scenarios such as authentication anomalies, network activity, and data exfiltration.

To install the Illuminate Anomaly Detection packs:

Navigate to Enterprise > Illuminate.
Search for "Anomaly Detection" in the search box.
Select and enable both the Anomaly Detection Add-on and the Anomaly Detection Spotlight content packs.
From the Events Definitions page, select and enable the required detectors.

When you install the Illuminate Anomaly Spotlight pack, a set of built-in detectors are installed as Event Definitions and listed on the Event Definitions page. Manage these detectors through the Events Definition workflow.

Hint: Anomaly results may require training data to improve accuracy. You may see some false positives initially as the detector begins training on a new data set. For optimal results, ingest sufficient historical data to allow anomaly detection logic evaluate log patterns effectively.

Anomaly Events Configuration

Graylog also provides anomaly detection through configurable anomaly detectors, available within the Anomaly Events definition configuration menu.

These detectors are implemented as detection types that you select when creating an Event Definition.

Currently supported detection types include:

Unlike Illuminate detectors, these detectors are not installed through a content pack. They are built-in detection types available in the Event Definition workflow, where you can define thresholds and parameters.

Configure an Anomaly Event Definition

To configure an Anomaly Events definition:

Navigate to Alerts > Event Definitions.
Click the Create event definition button in the upper right corner.
Follow the configuration wizard and in the Event Details section, enter a title, description, and priority for the event, and optionally add new or existing event procedures.
Select Anomaly Events as the Condition Type.
Choose a Detection Type.
Configure detection thresholds and parameters.
Add notifications as needed.
Enable the event definition.

Troubleshooting and Common Issues

The following section outlines troubleshooting steps for common issues to assist you in resolving potential challenges you may encounter.

Issue: No Anomalies Detected

After configuring Anomaly Detection, no anomaly events are generated, even though data is actively being ingested.

Anomaly Detection requires sufficient historical data to establish a behavioral baseline. If the model has not observed enough data over time, it cannot accurately identify deviations. Additionally, missing or incomplete asset correlation can prevent certain detectors from evaluating data.

Solution: Ensure Sufficient Data and Verify Detector Settings

To resolve this issue, ensure that the anomaly detection model has enough data to establish a baseline and that your configuration is correct:

Verify that sufficient historical data has been ingested for the relevant streams or entities.
Confirm that data is consistently flowing into Graylog without gaps.
Check that assets are properly populated and correlated, especially for detectors that rely on asset context.
Review detector configuration to ensure the correct fields, streams, and categories are selected.
Allow additional time for the model to learn baseline behavior before expecting results.