After logs are ingested into Graylog, they undergo a customizable routing process that offers granular control over how data is processed, where it is sent, and the specific actions taken with it. This flexible architecture allows you to define rules and conditions that determine how logs are filtered, transformed, and forwarded to different destinations based on various factors, ensuring that each piece of log data is directed appropriately for search, analysis, or storage.
Inputs
Logs are ingested from a source into Graylog via inputs, which essentially define where and how Graylog pulls in log data. For more information on inputs and log sources, see Getting in Log Data.
Data Routing
Once logs have been ingested into Graylog, the process by which they are filtered, enriched, and routed to a destination is referred to as Data Routing. This process is fundamentally applied at the level of a stream and involves the application of various rules and filters to move data where you want it to go.
Streams
Once logs have been ingested into Graylog, you can assign them to specific streams, which are specific processes used to filter and route log data through and outside of Graylog. You can additionally establish stream rules to determine which log messages are routed to which stream. For example, you could create a stream that only accepts logs from a particular application or server, or you could route logs that contain only certain keywords in the message. You can also have multiple streams that receive the same log data, allowing you to have different views and rules for the same subset of information.
Pipelines
Additionally, pipelines provide a flexible way to transform and enrich messages after they are routed into streams. A pipeline is a sequence of processing stages through which messages pass. Each stage can apply one or more pipeline rules consisting of various functions to perform specific operations on the log messages, such as filtering, transforming, or routing. Pipeline rules can be built using the rule builder function in the Graylog interface.
Destinations
Once logs have been channeled into streams and pipeline rules are applied, you can then route logs into three specific destinations:
Using specific filter rules, Data Routing allows you to route logs into one or multiple destinations based on the filters applied to the log data.
Data Warehouses
This is a Graylog Enterprise feature. A valid Graylog Enterprise license is required.
Data Warehouses are centralized repositories that allow you to store large amounts of log data without the need to write this data to your search backend, like OpenSearch. This log data is compressed for storage and may be retrieved in future so that the data can be used for search and analytics. Additionally, Data Warehouses may use either an Amazon S3 bucket or network storage as a Data Warehouse backend.
Index Sets
Data from a stream can also be directly written to one or more indices. The Graylog index model allows you to apply configurations to index sets, which are predetermined collections of indices. This process allows you to manage the lifecycle of indices, including rotation and retention, what storage backend is utilized, and when indices are archived (if desired).
You can also apply index set templates with predefined configuration settings that meet your desired performance and maintenance costs. These templates and all configuration settings for index sets are based on the different data tiers offered by Graylog depending on your specific performance parameters.
Outputs
An output is a mechanism that allows Graylog to send logs to external systems or destinations, like a specific database or another Graylog instance, after they have been collected. Graylog instances with an Enterprise license may also make use of the Enterprise Output Framework, which enables you to forward messages to external systems via a structured approach that can make use of additional pipeline rules to filter and enrich the log data.
