Slow Pipeline Processing and High CPU Usage

After you add or change pipeline rules, you might notice that message throughput drops, CPU usage on your Graylog nodes increases, the process buffer fills, or the journal grows because Graylog cannot process messages as fast as they arrive. In environments with many streams, pipelines, and rules, it can be difficult to tell which rule is responsible.

This article explains how to identify the pipelines and rules that use the most processing time and how to resolve the problem.

Possible Causes

The cost of a pipeline rule varies widely. Routing a message to a stream or removing a field costs very little, while a single inefficient rule connected to a busy stream can affect the CPU usage of the entire cluster. The following conditions commonly cause this issue:

A rule uses a resource-intensive function, such as a complex regular expression or Grok pattern, in its condition, so the function runs against every message in the stream.
A pipeline is connected to a high-volume stream, such as the default stream (All messages), even though its rules apply to only a small subset of those messages.
A rule calls a lookup table whose data adapter responds slowly or does not use a cache.
Several pipelines connected to the same stream repeat the same resource-intensive work.

Identify the Affected Stream

Start by finding the streams with the longest processing times. These streams point you to the pipelines that are most likely responsible.

To identify the affected stream:

Navigate to Streams.
Select the Performance view.
Sort by Avg Processing Time (Last 15min) or Max Processing Time (Last 15min).

The streams at the top of the list take the longest to process. Note these streams and the pipelines connected to them.

For a description of these columns, see Streams.

Identify the Pipeline and Rule Causing the Load

The processing load metric shows each pipeline's and rule's share of the total pipeline execution time. Graylog samples one in every 100 messages to calculate it, so the values are approximate and are intended for comparing pipelines and rules with each other.

To identify the pipeline and rule:

Navigate to System > Pipelines.
Click Enable debug metrics. A Debug metrics are enabled banner appears at the top of the page.
Wait a few minutes for the processing load values to stabilize. Values are smoothed over a 15-minute window.
Sort the pipelines by the Pipeline Load column.
Click the pipeline with the highest load to open its details page.
Review the processing load of each rule to find the rules that use the most processing time.

If a rule is used in more than one pipeline, select the Manage rules tab to view the rule's combined processing load across all pipelines.

Important: Debug metrics add overhead to message processing and increase memory usage. Disable them when you finish troubleshooting. See Disable Debug Metrics.

Optimize the Rule

After you identify the rule, reduce the amount of work it does for each message. Test each change before you apply it to production traffic.

To optimize the rule:

Navigate to System > Pipelines and select the Manage rules tab.
Click Edit next to the rule.
Narrow the rule's condition so the resource-intensive check runs only on the messages it applies to. Put a lightweight check, such as has_field or from_input, first, and join it to the resource-intensive check with &&. For example: when from_input(name: "firewall-input") && regex("...", to_string($message.message)).matches == true.
Replace complex regular expressions with simpler functions, such as contains or starts_with, where they produce the same result.
Test the updated rule with rule simulation.
Click Update rule.

If the pipeline is connected to a high-volume stream but processes only a subset of its messages, route those messages to a dedicated stream and connect the pipeline to that stream instead. See Build Pipeline Rules for how to edit pipeline connections.

After you make changes, review the Pipeline Load column again to confirm that the rule's share of processing time has decreased.

Disable Debug Metrics

To disable debug metrics:

Navigate to System > Pipelines.
From the Manage rules tab, click the Disable debug metrics button in the banner at the top of the page.

Share Processing Load Data with Graylog Support

If you cannot resolve the issue, generate a Cluster Support Bundle and attach it to your support ticket. The bundle includes pipeline processing load metrics in the metrics.json file, so Graylog Support can review how processing time is distributed across your pipelines and rules. Enable debug metrics before you generate the bundle so that these metrics are included. See Cluster Support Bundle.