Pipelines are an essential part of log message processing in Graylog, forming the backbone that ties together the processing steps applied to your data. They serve as a structured framework that enables you to define how incoming log data is evaluated, modified, and routed through Graylog.
Each pipeline is composed of rules and can be linked to one or more streams, enabling detailed and powerful control over how messages are processed. This capability makes pipelines indispensable for creating efficient, customized workflows and ensures that your log data is handled with precision and flexibility.
Key Concepts
The following key concepts are essential to understanding and using pipelines in Graylog.
Pipelines
Pipelines are a collection of pipeline rules that are applied in stages, which can then be attached to a stream. Messages that are sent to a stream are first subject to any attached pipelines.
Pipeline Rules
Pipeline rules are programmatic actions applied to messages. They can be used to direct messages to streams or for enrichment tasks like extracting fields from messages, renaming fields, performing transforms on messages, and much more. A series of pipeline rules enacted on a specific stream is referred to as a pipeline.
Functions
Functions are the building blocks of pipeline rules. Essentially, they are pre-defined methods to perform specific actions on the log messages being processed. Each function can take various parameters and return outputs that can be used to influence the processing of messages.
Stages
Pipelines are composed of stages, each of which can contain one or more processing rules. Stages are executed sequentially in numerical order. Think of stages as groups of conditions and actions that need to run in order. All stages with the same priority run at the same time across all connected pipelines.
Streams
Pipelines must be attached to a stream, which is the primary mechanism used to route and organize logs. See Streams for more information.
Pipeline Documentation
Before creating your first pipeline, we recommend that you read through the following documentation to understand how pipeline rules work, learn when and how to utilize functions in building these rules, and explore examples of pipeline rules that can be created:
-
Pipeline Rule Logic reviews the basic structure of how rules are created and the syntax used to compose them.
-
Functions provides an explanation of how functions behave and a reference of all the specific functions you can employ when creating rules, including the actions they perform and examples.
-
Build Pipeline Rules walks you through the process of creating and testing pipeline rules in the rule builder interface.
-
Rule Builder Use Cases identifies many common uses of pipeline rules and provides examples of these rule types.
