Rule Builder Use Cases

Graylog's rule builder interface is a powerful tool designed to simplify the process of creating, managing, and applying rules for processing log messages. As organizations collect large volumes of data from various sources, the ability to efficiently parse, filter, and route this data becomes crucial for effective log management and real-time insights. The rule builder serves as an intuitive, visual way to construct these rules without needing to write complex code.

This article covers practical use cases for pipeline rules, including filtering logs, data enrichment, and routing messages to streams and alerting systems. We encourage you to use these scenarios as a reference to build efficient pipeline rules of your own.

Anonymization Rules

Pipeline rules can be used to modify or redact parts of log messages that contain sensitive data. This data can be modified before messages are stored or forwarded. Anonymization helps ensure compliance with privacy regulations by masking sensitive information in log messages.

Example Rule

In this rule, the following logic is applied: 

• When (condition)

  • The rule checks if the field source_ip exists in the log message. This ensures the rule is applied only to messages that actually contain the source_ip field.

• Then (action)

  • If the condition is true (i.e. the source_ip field exists), the rule removes the source_ip field from the message. This action effectively deletes the field and ensures that the sensitive data (e.g. an IP address) is no longer part of the log message.

Example Rule Syntax

rule "Mask sensitive information."
when
    has_field("source_ip")
then
    remove_single_field("source_ip");
end

Breadcrumb Rules

A breadcrumb rule helps to track or trace the flow of log messages through different stages or systems, serving as a breadcrumb that can be followed or traced. These rules could be used to add metadata, tag messages, or track processing.

Example Rule

In this rule, the following logic is applied: 

• When (condition)

  • Because the when statement is true by default, this rule will always execute for every log message processed by the pipeline regardless of the message's content.

• Then (action)

  • The set_field function creates or updates a field named rule_demo in the log message. The field is set to the value "Every cloud has a silver lining.", meaning each message that passes through this pipeline will have a new field named rule_demo added with the value "Every cloud has a silver lining.".

Example Rule Syntax

rule "Set demo field"
when
    true
then
    set_field("rule_demo", "Every cloud has a silver lining.");
end

Filter Rules

Filter rules can be used to drop messages and reduce unnecessary ingestion and license usage.

Example Rule

In this rule, the following logic is applied: 

• When (condition)

  • The condition checks whether the log message contains a field named testing_only. If the field exists in the log message, the condition evaluates to true.

• Then (action)

  • If the condition is true, the message is dropped via the drop_message() function. This action removes the log message entirely, preventing it from being stored, processed further, or routed to any streams.

Example Rule Syntax

rule "Drop Test Messages"
when
has_field("testing_only")
then
drop_message();
End

Modification Rules

Modification rules are rules applied to log messages to alter or enrich them by adding, updating, or removing fields; changing values; or performing other transformations.

Example Rule

In this rule, the following logic is applied: 

• When (condition)

  • The rule checks if the log message contains a field named event_time. If the field exists, the rule proceeds to process the message.

• Then (action)

  • The rule takes the value of the event_time field (converted to a string) and parses it into a date object using the provided format and the timezone UTC.

  • It then takes the parsed date object (event_time_date) and formats it into a string using the UK timezone.

  • Finally, it adds the new field event_time_uk to the log message with the converted time value.

Example Rule Syntax

rule "convert event_time to UK timezone"
when
    has_field("event_time")
then
    let event_time_date = parse_date(
    value: to_string($message.event_time),
    pattern: "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'", // Adjust this pattern as needed
    timezone: "UTC"
    );
    let event_time_uk = format_date(
    value: event_time_date,
    date_format: "yyyy-MM-dd'T'HH:mm:ss.SSSZ",
    timezone: "Europe/London"
    );
    set_field("event_time_uk", event_time_uk
    );
end

Enrichment Rules

Enrichment rules are used to enhance log messages by adding additional information or context to them.

Example Rule

In this rule, the following logic is applied: 

• When (condition)

  • The rule checks if the field Src_ip_geo_country exists in the message and has the value of "US".

• Then (action)

  • The action updates the value of the field Src_ip_geo_country to "UniSt".

Example Rule Syntax

rule "SrcCountryUnitedStates"
when
    (has_field("Src_ip_geo_country") &&
     to_string($message."Src_ip_geo_country") == "US"
     )
then
    set_field(
      field : "Src_ip_geo_country",
      value : "UniSt",
      clean_field : false
    );
end

Routing Rules

Routing rules allow you to route certain messages to a stream and remove them from the current stream.

Example Rule

In this rule, the following logic is applied: 

• When (condition)

  • The rule applies to messages that have a field named gl2_remote_ip, and the value of the gl2_remote_ip field must match the string "66914166ac1d1568bad817f3".

• Then (action)

  • The message is routed to the stream named "My First Stream".

Example Rule Syntax

rule "Route Message to Stream"
when
  has_field("gl2_remote_ip") && to_string($message.gl2_remote_ip) == "66914166ac1d1568bad817f3"
then
    route_to_stream(
    name: "My First Stream",
    remove_from_default: true
    );
end