Pipeline Functions

Functions are the means of interacting with the messages Graylog processes.

Functions are written in Java and are pluggable, allowing Graylog’s pipeline processing capabilities to be easily extended.

Conceptually a function receives parameters, the current message context, and (potentially) returns a value. The data types of its return value and parameters determine where it can be used in a rule. Graylog ensures the rules are sound from a data type perspective.

Function parameters can either be passed as named pairs or positions, as long as optional parameters are declared as coming last. (The functions' documentation indicates which parameters are optional by wrapping them in square brackets.)

Let’s look at a small example to illustrate these properties:

Text
rule "function howto"
when    
    has_field("transaction_date")
then    
    // the following date format assumes there's no time zone in the string    
    let new_date =parse_date(to_string($message.transaction_date), "yyyy-MM-dd HH:mm:ss");    
    set_field("transaction_year", new_date.year);
 end

In this example, we check if the current message contains the field transaction_date and then, after converting it to a string, try to parse it according to the format string yyyy-MM-dd HH:mm:ss, so for example the string 2016-03-05 14:45:02 would match. The parse_date function returns a DateTime object from the Java Joda-Time library, allowing easier access to the date’s components.

We then add the transaction’s year as a new field, transaction_year to the message.

You’ll note that we didn’t specify a time zone for our date, but Graylog still had to pick one. Graylog never relies on the local time of your server, as that makes it nearly impossible to figure out why date handling came up with its result.

The reason Graylog knows which timezone to use is because parse_date actually takes four parameters, rather than the two we’ve given it in this example. The other two parameters are a String called timezone (default value: "UTC") and a String called locale (default value: the default locale of the system running Graylog) which both are optional.

Let’s assume we have another message field called transaction_timezone, which is sent by the application and contains the time zone ID the transaction was done in (hopefully no application in the world sends its data like this, though):

Text
rule "function howto"
when    
    has_field("transaction_date") && has_field("transaction_timezone")
then    
    // the following date format assumes there's no time zone in the string    
    let new_date = parse_date(  
            to_string($message.transaction_date),  
            "yyyy-MM-dd HH:mm:ss",  
            to_string($message.transaction_timezone) 
       );    
       set_field("transaction_year", new_date.year);
end

Now we’re passing the parse_date function its timezone parameter the string value of the message’s transaction_timezone field.

In this case we only have a single optional parameter, which makes it easy to simply omit it from the end of the function call. However, if there are multiple optional parameters, or if there are so many parameters that it gets difficult to keep track of which positions correspond to which parameters, you can also use the named parameter variant of function calls. In this mode the order of the parameters does not matter, but all required ones still need to be there.

In our case the alternative version of calling parse_date would look like this:

Text
rule "function howto"
when    
    has_field("transaction_date") && has_field("transaction_timezone")
then    
    // the following date format assumes there's no time zone in the string    
    let new_date = parse_date(                        
            value: to_string($message.transaction_date), 
            pattern: "yyyy-MM-dd HH:mm:ss",
            timezone: to_string($message.transaction_timezone) 
         );    
      set_field("transaction_year", new_date.year);
 end

Java Data Types

Pipeline rules can theoretically be built using some Java data types when creating your query. This is limited to those types that are queried using the get function.

For example, the function .millis can potentially be used in Graylog pipeline rules for DateTime and Period objects.

rule "time diff calculator millis"
when true
then
let time_diff = to_long((parse_date(
value: to_string(now(timezone:"Europe/Berlin")),
pattern: "yyyy-MM-dd'T'HH:mm:ss.SSSZ",
locale: "de_DE").millis)) -
to_long(parse_date(
value: to_string($message.timestamp),
pattern: "yyyy-MM-dd'T'HH:mm:ss.SSSZ",
locale: "de_DE").millis);
set_field("scan_age_millis", time_diff);
end