Graylog Illuminate is a collection of content comprising pipelines, parsing rules, lookup tables, and more. This content enables various event logs to be processed using a standard methodology, leveraging the Graylog Information Model (GIM) schema, to make searching and analyzing common log sources more efficient.
By enriching and normalizing your log data so that the username or IP address is always in the same field, searching for logs becomes much easier and faster. Additionally, you can create more universal dashboards that will work across any data type (as they are mapped to the schema) and regardless of which firewall connection(s) you may have.
To accomplish this, Illuminate works by ingesting logs, sorting them, and processing them. The sorting process occurs on the original log message as it comes into Graylog, so how the log data is sent affects whether Illuminate will pick up and process the message correctly. For example, some devices can send logs in multiple formats, like syslog-compliant messages, BSD-compliant messages, and free-form messages, but a specific form is still required to make parsing rules work. For specifics on system versions, specific formats, or settings, please refer to the individual content pack documentation.
Illuminate Bundles and Graylog License Models
Illuminate provides content for a variety of Graylog license models.
Graylog Open
Graylog Open users may access select content packs for use with Illuminate. These packs provide parsing for specific logs based on the GIM schema. The following content packs are available for use with Graylog Open:
Graylog Enterprise and Security
A variety of additional content packs are available for Graylog Enterprise and Security users. These packs further enhance Graylog's log management and analysis capabilities by providing pre-built content, including customized dashboards, enhanced parsing rules, and more. For more information on content packs available with Illuminate, see the content pack documentation.
You may also contact the Graylog Sales team for more information on purchasing a license for use with Graylog Enterprise or Security.
Illuminate Architecture
Illuminate is designed with a processing hierarchy that breaks up processing into three key areas.
Processing Packs
Individual packs for parsing and processing logs:
- Identify logs from the collection of all logs received by a Graylog instance.
- Perform parsing and/or normalization and apply the Graylog schema.
- Identify specific event message types and assign type codes.
- Enrich event messages.
Illuminate Core
The Illuminate core processor:
- Provides common processing logic to event log messages.
- Identifies common private or reserved IP addresses.
- Enriches event messages that have been assigned event type codes with category, subcategory, and event type data.
- Optionally provides Geolocation and ASN enrichment to eligible messages using either MaxMind or IPinfo databases.
- Optionally provides GIM enforcement, which will ensure events have required fields for categories and subcategories, and identifies potential event categorization issues.
Spotlight Packs
Individual content packs that operate on parsed and processed logs and provide:
-
Dashboards to visualize processed logs
-
Sigma Rules and Event Definitions to detect unusual activity within logs
Performance Impact of Illuminate
Illuminate log processing allows for items like alert rules, anomaly detectors, and dashboards to work across various log sources. With Illuminate processing log data, you do not have to create separate rules like "Windows Logon Brute Force" and "Linux Logon Brute Force." You only need to create one rule to cover them both.
As with all processing in Graylog, there will be performance implications as each log message goes through the process described above. Gates or sorting rules are the first set evaluated to limit logs to be processed further, shortening the number of rules each message touches.
Processing rules can range from simple key-value extractors, which perform very quickly, to complex regex statements or GROK patterns. Each rule can have a different performance impact, and each rule can perform differently based on the log type, so finding an actual cost per rule is subjective to an environment.
Indexes and Shards
Graylog Illuminate does not use unique values for index and shard settings; instead, it currently takes the system's default for those settings. After the indexes and streams are created, you can adjust the default settings if a replica is needed or for more or fewer shards.
Illuminate sets up indexes with a retention time based on common practices and standards. These settings allow the dashboards, anomaly rules, and alert rules to have enough online data to operate. Adjustments to these settings can be made, but note that any previously saved settings can be affected.
