Once you have reviewed pipelines, rules, and stream connections, you can start creating your own rules and processing pipelines. This article provides you with the information you need to get started with the user interface.
Configuration
Configure the Message Processor
Before you start using processing pipelines, you need to ensure the Pipeline Processor (message processor) is enabled and correctly configured. You can do so by going to System > Configurations and clicking Message Processors.
On the Configurations page, enable the Pipeline Processor. If you want your pipelines to have access to static fields set on inputs and/or fields set by extractors, set the Pipeline Processor after the Message Filter Chain and click Update configuration.
Create and Manage Rules
Rules can be used to enrich, modify, or route your messages according to your specifications. A rule consists of a condition and a list of actions. Graylog evaluates a condition against a message and executes the actions if the condition is satisfied. For example, you might want to add a source_ip field to a message that does not currently display this information. If your incoming messages do not have a source_ip field, you would create a rule such as the following:
You can create, edit, and delete your pipeline rules on the System > Pipelines page under Manage rules. Existing rules are displayed on this page.
You can write new rules by clicking Create Rule or Edit (found to the right of each rule row). In the next section, we go into detail about how you can create your own pipeline rules.
The pipeline rule UI is a user-friendly tool for creating and managing rules within Graylog. You may choose to use either the rule builder or the source code editor to create your own rules.
Create a Rule Using the Rule Builder
The rule builder was developed to provide a more visual method for creating rules directly in the Graylog UI. It is the default tool for building pipeline rules. The rule builder follows a structured when-then statement format. When statements are built with conditions, and then statements are built with actions.
Both statement boxes have searchable drop-down menus. If you enter the first few letters of a function name the drop-down will provide you with suggestions. The suggested functions will also include a short description of what they do. You may also refer to our Functions Descriptions documentation for a full list of available function descriptions.
To create a rule, first enter a title and description of the rule being created. Then enter a when statement. For example, let's say that we want to decrease the size of our ingested data and so we have decided to remove the Fortigate pattern field from messages that come through the Gelf input. The when statement is Check if message arrived on input Gelf. So, the condition is that the message has arrived via a Gelf input.
The then statement or action is to remove the field and this is done by selecting the Remove field function and entering the field name.
You may choose to add one or more statements to your rule. When statements may be linked by and or or options, which are found in the top right corner of the when box. In order to edit or delete a when statement, hover over it. The edit/delete buttons will appear at the right side of the row.
Then statements can be edited via the drop-down menu which also provides you with selections such as duplicate and insert above or insert below.
If a then statement returns a value, then these values will be displayed as output variables. Note that these variables may be inserted into other statements as needed. All available output variables will be displayed in a drop-down menu. The following use case displays this feature.
Use Case
For this example we want to use a lookup table for message enrichment. We would like to see more geographical information about any source IPs. First, we navigate to Manage Rules, and then click on the Create Rule button. to reach the rule builder. We want to find any messages with the field source_ip. . We use the has_field function to accomplish this.
The then statement is to retrieve the value for the source IP from our lookup table. $output_1 which is the output of the prior step is looked up against the lookup table. Then we use the function set fields to enrich our messages with the result. We also chose to add a "geo_" prefix to the field names during this step.
The rule builder will take data types into account and only provide matching output variables for each function parameter. If you have more than one action, the output variables will be highlighted when hovered over. This feature helps you track which variable you have used and where you have used it. You can see any condition outputs or action outputs (also highlighted) in the area under the rule simulation box.
Rule Simulation
Rule simulation is a way to test your rule before actually creating it. You can test a rule by providing a normal string to simulate the message field. You may choose to only simulate a field and not the whole message. In this case, enter a key/value pair or JSON in the rule simulation box. For more details on how this works, see the Rule Simulation documentation.
In the rule simulation box, you can see the values that the output variables are assigned to in each step. When you save a rule (either in the rule builder or the source code editor), the last used message will be stored along with the rule.So, you will always have a message to work with.
Create a Rule with the Source Code Editor
You can build a pipeline rule by writing when and then statements using source code. The Rules page explains how to write a rule in detail.
The source code editor also lists available functions and their details in the rules quick reference to make the task more manageable.
The optional rule simulation box can be used to test rules in isolation from the pipeline itself. Messages may be tested and reset multiple times.
Manage Pipelines
Once there are some rules in Graylog, you can create pipelines that use them to modify and enrich your messages.
To manage your pipelines, go to System > Pipelines and select the Manage Pipelines tab. Here you may create, edit, and delete pipelines by clicking the Edit or Delete buttons found at the end of each row.
In order to create or edit pipelines, you need to add your rules to a stage, which has a certain priority. The web interface will let you add rules to the default stage (priority 0) and to create new stages with potentially different priorities. For further details, please refer to Pipelines.
Pipeline Stages
A pipeline can have more than one stage, and when you create or edit a stage you need to select how to proceed to the next stage in the pipeline:
-
All rules in this stage match the message
This option will only consider further stages in the pipeline when all conditions in rules evaluated in this stage are true. This is equivalent to match all in the Pipelines section. -
At least one of the rules in this stage matches the message
Selecting this option will continue to further stages in the pipeline when one or more of the conditions in rules evaluated in this stage are true. This is equivalent to match either in the Pipelines section.
Connect Pipelines to Streams
You can decide which streams are connected to a pipeline from the pipeline details page. Under System > Pipelines, click on the title of the pipeline you want to connect to a stream, and then click on the Edit connections button.
You can assign many pipelines to the same stream, in which case all connected pipelines will process messages routed into that stream based upon the overall order of stage priorities.
Remember, as mentioned in the Stream connections documentation, All messages (the stream) is where all messages are initially routed, and is therefore a good place to apply pipelines applicable to all of your messages. Such pipelines might be responsible for stream routing, blacklisting, field manipulation, etc.
Simulate Your Changes
After performing some changes in a processing pipeline, you most likely want to see how they are applied to incoming messages. This is the job of the simulator.
Click the button under System > Pipelines and click on a specific pipeline under Manage rules to access the pipeline simulator.
In order to test the message processing you need to provide a raw message that will be routed into the stream you want to simulate. The raw message should use the same format Graylog will receive. For example: you can type a GELF message, in the same format your GELF library would send, in the field. Don’t forget to select the correct codec for the message you provide.
After specifying the message and codec, click to start the simulation and display the results.
The simulation provides the following results:
-
Changes summary
Provides a summary of modified fields in the original message, as well as a list of added and dropped messages. -
Results preview
Shows all fields in the processed message. -
Simulation trace
Displays a trace of the processing, indicating which rules were evaluated and which were executed. It also includes a timeline, in microseconds, to allow you to see which rules and pipelines are taking up the most time during message processing.
