Illuminate is a Graylog-provided collection of content comprising pipelines, parsing rules, lookup tables, and more. This content enables various event logs to be processed using a standard methodology, leveraging the Graylog schema and Graylog Information Model (GIM) to make searching and analyzing common log sources more efficient.
By enriching and normalizing your log data so that the username or IP address is always in the same field, searching for logs becomes much easier and faster. Also, you can make more universal dashboards (Dashboards that would work across any data type because it is mapped to the schema) as it will not matter which firewall connection you may have, the details all look the same.
To accomplish this, Illuminate works by ingesting the logs, sorting them, and processing them. The sorting process occurs on the original log message as it comes into Graylog, so how the log data is sent affects whether Illuminate will pick up and process the message correctly. For example, some devices can send logs in multiple formats like Syslog compliant messages, BSD compliant messages, and free-form messages. But we have to have it in a specific form to make parsing rules work. Please refer to the Getting in log data page on the Graylog documentation site.
For specifics on system versions, specific formats, or settings, please refer to the Security Content Packs page on the Graylog documentation site.
Illuminate Architecture
Illuminate is designed with a processing hierarchy that breaks up processing into roughly two steps:
Technology Packs
- Identifies logs from the collection of all logs received by a Graylog instance.
- Performs parsing/normalization and applies the Graylog schema.
- Identifies specific event message types and assigns type codes.
- Enriches event messages.
Illuminate Core
- Provides common processing logic to event log messages.
- Identifies common private/reserved IP addresses.
- Enriches event messages that have been assigned event type codes with category, subcategory, and event type data.
- Optionally provides Geolocation and ASN enrichment to eligible messages using either MaxMind or IPinfo databases.
- Optionally provides GIM enforcement which will ensure events have required fields for categories and subcategories and identifies potential event categorization issues.
Performance Impact of Illuminate
Illuminate log processing allows for items like alert rules, anomaly detectors, and dashboards to work across various log sources. With Illuminate processing log data, you do not have to create separate rules like "Windows Logon Brute Force" and "Linux Logon Brute Force." You only need to create one rule to cover them both.
As with all processing in Graylog, there will be performance implications as each log message goes through the process described above. Gates or sorting rules are the first set evaluated to limit logs to be processed further, shortening the number of rules each message touches.
Processing rules can range from simple key-value extractors, which perform very quickly, to complex regex statements or GROK patterns. Each rule can have a different performance impact, and each rule can perform differently based on the log type, so finding an actual cost per rule is subjective to an environment.
Indexes and Shards
Graylog Illuminate does not use unique values for index and shard settings; instead, it currently takes the system's default for those settings. After the indexes and streams are created, you can adjust the default settings if a replica is needed or for more or fewer shards.
Illuminate sets up indexes with a retention time based on common practices and standards. These settings allow the dashboards, anomaly rules, and alert rules to have enough online data to operate. Adjustments to these settings can be made, but note that any previously saved settings can be affected.
License Cost
Graylog Operations and Security Licenses are calculated based on the amount of data written to disk and not the processing of data it ingests.
Illuminate adds data to enrich the message, like Geo Coordinates or a GIM tag like "logon." While these add to the total amount of data stored on disk, additional data in the log can be truncated or removed. Illuminate balances the benefits of enriching log data with the cost of storage to give you the best bang for your buck!
Contact the Graylog Sales team for more information on license purchase.
