Anomaly Detection

The following article exclusively pertains to a Graylog Security feature or functionality. Graylog Security is a part of the Graylog centralized log management platform and requires a separate license. Contact the Graylog Sales team for more information on this product.

Warning: Anomaly detection has changed. Legacy detectors are no longer supported, and creating or editing custom RCF detectors is now disabled by default. Contact Graylog Support for migration assistance or to enable custom RCF detection logic.

Anomaly Detection is a Graylog Security feature that identifies unusual patterns in log data. Instead of matching a fixed rule, it compares current activity against a reference of normal behavior, such as a user's previous login, historical log volume, or a learned baseline, and generates events when deviations exceed expected thresholds.

Graylog provides anomaly detection in two ways:

Built-in detection types that you configure in your anomaly event definitions: Impossible Travel Anomalies and Log Volume Anomalies.
Random Cut Forest (RCF) detectorsare prebuilt machine learning anomaly event definitions delivered through Graylog Illuminate.

Understanding how these approaches differ is key to configuring anomaly detection effectively. This topic explains how each approach works and how they compare.

Prerequisites

Before proceeding, ensure that the following prerequisites are met:

You must have a valid Graylog Security license.
For RCF detectors: Illuminate version 7.1 or later is installed and the Anomaly Detection content pack is enabled.

Highlights

The following highlights provide a summary of the key takeaways from this article:

Use RCF detectors for quick deployment of common detection scenarios with minimal configuration.
Use built-in detection types, such as Impossible Travel and Log Volume, when your monitoring needs align with these specific use cases and you want full control over thresholds.
Manage all detectors from the Event Definitions page.

Detection Types

Graylog provides two kinds of anomaly detection. Both create standard Graylog events and are managed from the Event Definitions page, but they detect anomalies in different ways.

Built-in detection types are rule-based detectors available in the event definition wizard: Impossible Travel Anomalies and Log Volume Anomalies. You create the event definition, select a detection type, and set the thresholds. On each run, the detector compares current activity against a defined reference. Impossible Travel Anomalies compares each user login with that user's previous login and calculates whether the implied travel speed is realistic. Log Volume Anomalies compares the current message count with a baseline calculated from historical data. Graylog creates an event when the result exceeds the configured thresholds. See Anomaly Events.

RCF detectors are prebuilt anomaly event definitions that use the Random Cut Forest (RCF) detection type and are delivered with the Illuminate Anomaly Detection content pack. The RCF algorithm learns the normal pattern of a metric over time, such as authentication counts per application or network traffic per host. Each detector maintains a separate model for each category value, scores every new observation by how far it departs from the learned pattern, and creates an event when the anomaly grade and score exceed their thresholds. Because the models learn from your data, they need a training period before their results are reliable. See Random Cut Forest (RCF) Anomaly Detection.

The following table compares the two approaches.

Comparison	Built-In Detection Types	RCF Detectors
Available detections	Impossible Travel Anomalies Log Volume Anomalies	Anomaly Detection: Unusual Network Activity for Monitored System Anomaly Detection: Authentication Success and Failure by Application Anomaly Detection: File Activity Anomaly Detection: Windows Object Permissions Change
How they detect	Compare activity against user-defined thresholds based on the selected detection type, such as a user's previous login or a statistical baseline (average or percentile) of historical log volume.	An RCF model learns the normal shape of a metric over time for each category value, such as each application or host, and scores new activity by how far it departs from that pattern.
How you get them	Available in the event definition wizard. Select Anomaly Events as the condition type, then choose a detection type.	Installed with the Illuminate Anomaly Detection content pack as event definitions prefixed with Anomaly Detection:.
What you configure	Streams, search query, schedule, and the options specific to the selected detection type, including all detection thresholds.	Detectors are preconfigured. You enable them and can tune a limited set of settings, such as Window Delay, Grade Threshold, and Score Threshold. Creating or editing custom RCF detectors is disabled by default.
Requirements	Graylog Security license and log data that meets the detection type's requirements.	Graylog Security license, Illuminate 7.1 or later, the Anomaly Detection content pack, and the technology packs for your log sources. Some detectors also require asset categories.
Ramp-up time	Impossible Travel evaluates as soon as users have prior logins within the lookback period. Log Volume needs a full historical time period of data.	Models need time to learn a baseline. Expect some false positives while a detector trains on new data.
Documentation	Anomaly Events.	Random Cut Forest (RCF) Anomaly Detection.

Hint: RCF detectors also use the Anomaly Events condition type, so they appear alongside anomaly event definitions on the Event Definitions page. Look for the Anomaly Detection: prefix in the title to tell them apart.

Next Steps

To configure Impossible Travel or Log Volume detection, see Anomaly Events.
To install, enable, and tune the predefined RCF detectors, see Random Cut Forest (RCF) Anomaly Detection.