GELF Inputs

The Graylog Extended Log Format (GELF) is a log format that avoids the shortcomings of classic plain Syslog and is perfect for logging from your application layer. It comes with optional compression, chunking, and, most importantly, a clearly defined structure. The Input of GELF messages can be UDP, TCP, or HTTP. Additionally, a queue is possible.

Some applications like Docker can send GELF messages native. Also, fluentd speaks GELF.

There are dozens of GELF libraries for many frameworks and programming languages to get you started. Read more about GELF in the specification.

GELF HTTP

You can send in all GELF types via HTTP, including uncompressed GELF, which is simply a plain JSON string. This input supports the configuration of authorization headers, adding password-like protection. When configured, any client making a request must provide the correct authorization header name and value with each request for it to be accepted.

After launching your new input, configure the following fields based on your preferences: 

• Global

  • Select this check box to enable the input on all Graylog nodes, or keep it unchecked to enable the input on a specific node.

• Title  

  • Assign a unique title to the input. Example: “GELF TCP Input for XYZ Source”.

• Bind address

  • Enter an IP address on which this input listens. The source system/data sends logs to this IP/input.

• Port

  • Enter a port to use in conjunction with the IP address.

• Receive Buffer Size (optional)

  • Depending on the amount of traffic being ingested by the input, this value should be large enough to ensure proper flow of data but small enough to prevent the system from spending resources trying to process the buffered data.

• No. of worker threads (optional)

  • This setting controls how many concurrent threads are used to process incoming data. Increasing the number of threads can enhance data processing speed, resulting in improved throughput. The ideal number of threads to configure depends on the available CPU cores on your Graylog server. A common starting point is to align the number of worker threads with the number of CPU cores. However, it is crucial to strike a balance with other server demands.
    
    

• TLS cert file (optional)

  • The certificate file that is stored on a Graylog system. The value of this field is a path (/path/to/file) that Graylog should have access to.

• TLS private key file (optional)

  • The certificate private key file that is stored on a Graylog system. The value of this field is a path (/path/to/file) that Graylog should have access to.

• Enable TLS

  • Select if this input should use TLS.

• TLS key password (optional)

  • The private key password.

• TLS client authentication (optional)

  • If you want to require the source of the messages sending logs to this input to authenticate themselves, set to optional or required.

• TLS Client Auth Trusted Certs (optional)

  • The path where the client (source) certificates are located on a Graylog system. The value of this field is a path (/path/to/file) which Graylog should have access to.

• TCP keepalive

  • Enable this option if you want the input to support TCP keep-alive packets to prevent idle connections.

• Enable Bulk Receiving

  • Enable this option to receive bulk message separated by newlines (\n or \r\n).

• Enable CORS

  • Enable Cross-Origin Resource Sharing (CORS) to configure your server to send specific headers in the HTTP response that instruct the browser to allow cross-origin requests.

• Max. HTTP chunk size (optional)

  • For large data, it is common practice to chunk smaller blocks (e.g., 8KB or 64KB chunks) to prevent overwhelming buffers. The maximum HTTP chunk size is 65536 bytes.

• Idle writer timeout (optional)

  • The maximum amount of time the server will wait for a client to send data when writing to an output stream before closing the connection due to inactivity.

• Authorization Header Name (optional)

  • Specify a custom authorization header name to optionally enforce authentication for all received messages. This is a way to add password-like security for this input.

• Authorization Header Value (optional)

  • Specify authorization header value to optionally enforce authentication for all received messages.

• Encoding (optional)

  • All messages need to support the encoding configured for the input. For example, UTF-8 encoded messages should not be sent to an input configured to support UTF-16.

• Override source (optional)

  • By default, messages parse the source field as the provided hostname in the log message. However, if you want to override this setting for devices that output non-standard or unconfigurable hostnames, you can set an alternate source name here.

• Decompressed size limit

  • The maximum size of the message after being decompressed.

After launching a GELF HTTP input you can use the following endpoints to send messages:

http://graylog.example.org:[port]/gelf (POST)

Try sending an example message using curl:

curl -XPOST http://graylog.example.org:12202/gelf -p0 -d '{"short_message":"Hello there", "host":"example.org", "facility":"test", "_foo":"bar"}'

Both keep-alive and compression are supported via the common HTTP headers. The server will return a  202 Accepted  when the message is accepted for processing.

Enable Bulk Receiving Option for HTTP GELF Input

Graylog provides users with an option to enable bulk receiving of messages via HTTP GELF input, which allows bulk receiving of messages separated by newline characters.

The input will automatically separate multiple GELF messages, which are newline-delimited (\n or \r\n) when this option is enabled.

Example curl request:

curl -XPOST -v http://127.0.0.1:12202/gelf -p0 \
-d $'{"short_message":"Bulk message 1", "host":"example.org", "facility":"test", "_foo":"bar"}\r\n\
{"short_message":"Bulk message 2", "host":"example.org", "facility":"test", "_foo":"bar"}\r\n\
{"short_message":"Bulk message 3", "host":"example.org", "facility":"test", "_foo":"bar"}\r\n\
{"short_message":"Bulk message 4", "host":"example.org", "facility":"test", "_foo":"bar"}\r\n\
{"short_message":"Bulk message 5", "host":"example.org", "facility":"test", "_foo":"bar"}'

GELF TCP

After launching your new input, configure the following fields based on your preferences: 

• Global

  • Select this check box to enable the input on all Graylog nodes, or keep it unchecked to enable the input on a specific node.

• Title  

  • Assign a unique title to the input. Example: “GELF TCP Input for XYZ Source”.

• Bind address

  • Enter an IP address on which this input listens. The source system/data sends logs to this IP/input.

• Port

  • Enter a port to use in conjunction with the IP address.

• Receive Buffer Size (optional)

  • Depending on the amount of traffic being ingested by the input, this value should be large enough to ensure proper flow of data but small enough to prevent the system from spending resources trying to process the buffered data.

• No. of worker threads

  • This setting controls how many concurrent threads are used to process incoming data. Increasing the number of threads can enhance data processing speed, resulting in improved throughput. The ideal number of threads to configure depends on the available CPU cores on your Graylog server. A common starting point is to align the number of worker threads with the number of CPU cores. However, it is crucial to strike a balance with other server demands.
    

• TLS cert file (optional)

  • The certificate file that is stored on a Graylog system. The value of this field is a path (/path/to/file) that Graylog should have access to.

• TLS private key file (optional)

  • The certificate private key file that is stored on a Graylog system. The value of this field is a path (/path/to/file) that Graylog should have access to.

• Enable TLS

  • Select if this input should use TLS.

• TLS key password (optional)

  • The private key password.

• TLS client authentication (optional)

  • If you want to require the source of the messages sending logs to this input to authenticate themselves, set to optional or required.

• TLS Client Auth Trusted Certs (optional)

  • The path where the client (source) certificates are located on a Graylog system. The value of this field is a path (/path/to/file) which Graylog should have access to.

• TCP keepalive

  • Enable this option if you want the input to support TCP keep-alive packets to prevent idle connections.

• Null frame delimeter

  • This option is typically left unchecked. New line is the delimiter for each message.

• Maximum message size

  • The maximum message size of the message. A default value should suffice but can be modified depending on message length. Each input type usually has specifications that note the maximum length of a message.

• Override Source

  • By default, messages parse the source field as the provided hostname in the log message. However, if you want to override this setting for devices that output non-standard or unconfigurable hostnames, you can set an alternate source name here.

• Encoding

  • All messages would need to support the encoding configured for the input. UTF-8 encoded messages shouldn’t be sent to an input configured to support UTF-16.

• Decompressed size limit

  • The maximum size of the message after being decompressed.

GELF UDP

After launching your new input, configure the following fields based on your preferences: 

• Global

  • Select this check box to enable the input on all Graylog nodes, or keep it unchecked to enable the input on a specific node.

• Title  

  • Assign a unique title to the input. Example: “GELF UDP Input for XYZ Source”

• Bind address

  • Enter an IP address that this input will listen on. The source system/data sends logs to this IP/input.

• Port

  • Enter a port to use in conjunction with the IP address.

• Receive Buffer Size (optional)

  • Depending on the amount of traffic being ingested by the input, this value should be large enough to ensure proper flow of data but small enough to prevent the system from spending resources trying to process the buffered data.

• No. of worker threads

  • This setting controls how many concurrent threads are used to process incoming data. Increasing the number of threads can enhance data processing speed, resulting in improved throughput. The ideal number of threads to configure depends on the available CPU cores on your Graylog server. A common starting point is to align the number of worker threads with the number of CPU cores. However, it is crucial to strike a balance with other server demands.
    

• Override Source

  • By default, messages parse the source field as the provided hostname in the log message. However, if you want to override this setting for devices that output non-standard or unconfigurable hostnames, you can set an alternate source name here.

• Encoding

  • All messages would need to support the encoding configured for the input. UTF-8 encoded messages shouldn’t be sent to an input configured to support UTF-16.

• Decompressed size limit

  • The maximum size of the message after being decompressed.