GELF Inputs

The Graylog Extended Log Format (GELF) is a structured logging format designed to overcome the limitations of traditional plain Syslog, making it ideal for application-layer logging. GELF supports optional compression, message chunking, and most importantly, a clearly defined, parseable structure.

Graylog can receive GELF messages over UDP, TCP, or HTTP, and also supports queued delivery through messaging systems. This flexibility allows you to choose the most appropriate transport method based on your performance, reliability, and scalability needs.

Several platforms, such as Docker and Fluentd, can natively output logs in GELF format. In addition, a wide range of GELF libraries are available for various programming languages and logging frameworks, making it easy to integrate GELF into your existing applications.

Input Type

This Input is a listener input type. See Inputs to learn about Input types.

Input Configuration

Follow the input setup instructions. During setup of this input, you can configure the following options:

The GELF AMQP Input allows Graylog to receive GELF (Graylog Extended Log Format) messages through an AMQP-compatible message broker, such as RabbitMQ. This input is useful in environments that rely on message queuing for decoupled, asynchronous communication. It enables structured log data to be published to a queue or exchange, which Graylog can consume reliably and at scale.

After launching your new input, configure the following fields based on your preferences: 

Configuration Option Description

Global

Select this check box to enable the input on all Graylog nodes, or keep it unchecked to enable the input on a specific node.
Node Select the node on which the input should start.
Title Assign a title to the input. Example: GELF AMQP Input for XYZ Source.

Broker hostname

The hostname or IP address of the message broker (e.g., RabbitMQ server).

Broker virtual host

The logical namespace within the broker used to separate applications or tenants.

Prefetch count

The maximum number of messages a consumer can receive before acknowledging them.

Queue

The name of the queue from which messages will be consumed.

Exchange

The exchange to which the input should bind in order to receive messages.

Routing key

The key used by the exchange to route messages into the specified queue.

Number of consumers/queues

Defines how many consumers or queues should be created for message processing.

Allow throttling this input (Checkbox)

Enables message rate limiting to control ingestion speed and prevent system overload.
Timezone Select the timestamp configured on the system that is sending GELF messages. If the sender does not include the timezone information, you can configure the timezone applied to the messages on arrival. That configuration does not overwrite the timezone included in the timestamp; however, it is the assumed timezone for messages that do not include timezone information.

Allow throttling this input

Enables message rate limiting to control ingestion speed and prevent system overload.

Broker port (optional)

The port number used by the message broker (default for AMQP is usually ----).

Username (optional)

The username for authenticating against the message broker, if required.

Password (optional)

The password associated with the broker username for authentication.

Passive queue declaration

Declares the queue in passive mode, ensuring it exists without modifying or creating it.

Bind to exchange

Binds the input to a specific exchange in the broker to receive messages from it.

Heartbeat timeout (optional)

Interval (in seconds) for client–broker heartbeat checks to detect lost connections.

Enable TLS

Select this option if this input should use TLS.

Re-queue invalid messages

Determines whether invalid/unprocessable messages are placed back into the queue.

Connection recovery interval (optional)

Time interval (in seconds) to wait before retrying a broker connection after failure.

Locale (optional)

This setting is used to determine the language of the message.

Use full field names

The GELF key name is usually used as the field name. Select this option if the full field name should be used.

The Graylog Extended Log Format (GELF) HTTP Input allows Graylog to receive GELF messages via standard HTTP POST requests. It supports all GELF message types, including uncompressed GELF, which is simply a valid JSON object. This input is ideal for applications and services that communicate over HTTP and prefer a stateless, firewall-friendly transport mechanism.

GELF HTTP also supports optional authentication via custom authorization headers, allowing you to protect the input with a shared secret. When this feature is enabled, clients must include the correct header name and value with each request to successfully submit messages.

Hint: This input listens for HTTP posts on the /gelf path.

After launching your new input, configure the following fields based on your preferences: 

Configuration Option Description

Global

Select this check box to enable the input on all Graylog nodes, or keep it unchecked to enable the input on a specific node.

Title  

Assign a unique title to the input. Example: GELF HTTP Input for XYZ Source.
Bind address

Enter an IP address on which this input listens. The source system/data sends logs to this IP/input.
Port Enter a port to use in conjunction with the IP address.
Receive Buffer Size (optional) Depending on the amount of traffic being ingested by the input, this value should be large enough to ensure proper flow of data but small enough to prevent the system from spending resources trying to process the buffered data.
No. of worker threads (optional) This setting controls how many concurrent threads are used to process incoming data. Increasing the number of threads can enhance data processing speed, resulting in improved throughput. The ideal number of threads to configure depends on the available CPU cores on your Graylog server. A common starting point is to align the number of worker threads with the number of CPU cores. However, it is crucial to strike a balance with other server demands.

TLS cert file (optional)

The certificate file that is stored on a Graylog system. The value of this field is a path (/path/to/file) that Graylog should have access to.

TLS private key file (optional)

The certificate private key file that is stored on a Graylog system. The value of this field is a path (/path/to/file) that Graylog should have access to.

Enable TLS

 Select if this input should use TLS.

TLS key password (optional)

The private key password.

TLS client authentication (optional)

If you want to require the source of the messages sending logs to this input to authenticate themselves, set to optional or required.

TLS Client Auth Trusted Certs (optional)

The path where the client (source) certificates are located on a Graylog system. The value of this field is a path (/path/to/file) which Graylog should have access to.

TCP keepalive

Enable this option if you want the input to support TCP keep-alive packets to prevent idle connections.

Enable Bulk Receiving

Enable this option to receive bulk message separated by newlines (\n or \r\n).

Enable CORS

Enable Cross-Origin Resource Sharing (CORS) to configure your server to send specific headers in the HTTP response that instruct the browser to allow cross-origin requests.

Max. HTTP chunk size (optional)

For large data, it is common practice to chunk smaller blocks (e.g., 8KB or 64KB chunks) to prevent overwhelming buffers. The maximum HTTP chunk size is 65536 bytes.

Idle writer timeout (optional)

The maximum amount of time the server will wait for a client to send data when writing to an output stream before closing the connection due to inactivity.

Authorization Header Name (optional)

Specify a custom authorization header name to optionally enforce authentication for all received messages. This is a way to add password-like security for this input.

Authorization Header Value (optional)

Specify authorization header value to optionally enforce authentication for all received messages.

Encoding (optional)

All messages need to support the encoding configured for the input. For example, UTF-8 encoded messages should not be sent to an input configured to support UTF-16.

Override source (optional)

By default, messages parse the source field as the provided hostname in the log message. However, if you want to override this setting for devices that output non-standard or unconfigurable hostnames, you can set an alternate source name here.

Decompressed size limit

The maximum size of the message after being decompressed.

After launching a GELF HTTP input you can use the following endpoint to send messages:

Copy 
http://graylog.example.org:[port]/gelf (POST)

Try sending an example message using curl:

Copy 
curl -XPOST http://graylog.example.org:12202/gelf -p0 -d '{"short_message":"Hello there", "host":"example.org", "facility":"test", "_foo":"bar"}'

Both keep-alive and compression are supported via the common HTTP headers. The server will return a  202 Accepted  when the message is accepted for processing.

Enable Bulk Receiving Option for HTTP GELF Input

Graylog provides users with an option to enable bulk receiving of messages via HTTP GELF input, which allows bulk receiving of messages separated by newline characters.

The input will automatically separate multiple GELF messages, which are newline-delimited (\n or \r\n) when this option is enabled.

Warning: Individual GELF messages must be formatted as a valid JSON (containing no line breaks within). Attempts to post formatted JSON to this input will result in an error.

Example curl request:

Copy 
curl -XPOST -v http://127.0.0.1:12202/gelf -p0 \
-d $'{"short_message":"Bulk message 1", "host":"example.org", "facility":"test", "_foo":"bar"}\r\n\
{"short_message":"Bulk message 2", "host":"example.org", "facility":"test", "_foo":"bar"}\r\n\
{"short_message":"Bulk message 3", "host":"example.org", "facility":"test", "_foo":"bar"}\r\n\
{"short_message":"Bulk message 4", "host":"example.org", "facility":"test", "_foo":"bar"}\r\n\
{"short_message":"Bulk message 5", "host":"example.org", "facility":"test", "_foo":"bar"}'

The GELF Kafka input supports collecting logs from Kafka topics with the help of Filebeats. Once logs are generated by the system and pushed to a Kafka topic, they are automatically ingested by this input.

Prerequisites

Before proceeding, ensure that the following prerequisites are met:

  • You must have a configured and working Kafka cluster.

  • Beats must be installed to the log source.

  • Install Zookeeper.

Required Third-Party Setup

To enable integration, complete the following required setup with your third-party service:

Kafka and Filebeat Setup

  1. Provide full access permissions to all Kafka and Filebeat directories.

  2. Configure the filebeat.yml file as shown below.

    Hint: Remember to replace localhost with your unique IP address.
    Copy 
    filebeat.inputs:
                                            - type: log
                                            enabled: true
                                            paths:
                                            - /var/log/syslog

                                            output.kafka:
                                            hosts: ["your_kafka_host:9092"]  # Replace with your Kafka host(s)
                                            topic: 'system_logs'             # Name of the Kafka topic
                                            codec.json:
                                            pretty: false
                                            preset: balanced
                                        

  3. Modify the Kafka server.properties file to advertise the broker to clients.

    Copy 
    advertised.listeners=PLAINTEXT://your.kafka.broker.hostname:9092

  4. Create a Kafka topic.

    1. From the Kafka bin directory, run the following command:

      Copy 
      ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic <Topic name>

Required Configuration Values

In your third-party configuration, make note of the following values that are required when configuring the input in Graylog:

  • Kafka topic

  • Kafka Consumer Group

After launching your new input, configure the following fields based on your preferences: 

Configuration Option Description

Global

Select this check box to enable the input on all Graylog nodes, or keep it unchecked to enable the input on a specific node.

Title  

Assign a unique title to the input. Example: GELF Kafka Input for XYZ Source.
Legacy mode

If you are running an older version of Graylog (prior to 3.3) or need to maintain compatibility with older setups that use ZooKeeper, enable legacy mode.

Bootstrap Servers (optional)

Enter the IP Address and port on which the Kafka server is running.

ZooKeeper address (legacy mode only) (optional)

Enter the IP Address and port on which the Kafka server is running.

Topic filter regex

Enter the topic name filter which is configured in the filebeats.yml file.

Fetch minimum bytes

Enter the minimum byte size a message batch should reach before fetching.

Fetch maximum wait time (ms)

Enter the maximum time (in milliseconds) to wait before fetching.

Processor threads

Enter the number of threads to process. This is based on the number of partitions available for the topic.

Allow throttling this input

If enabled, no new message is read from this input until Graylog catches up with its message load. This configuration parameter is typically useful for inputs reading from files or message queue systems like AMQP or Kafka. If you regularly poll an external system, e.g. via HTTP, you should leave this option disabled.

Auto offset reset (optional)

Choose the appropriate selection from the drop down menu if there is no initial offset in Kafka or if an offset is out of range.

Consumer group id (optional)

Enter the name of the consumer group the Kafka input belongs to.

Override source (optional)

Enter the default hostname derived from the received packet. Only set this source if you want to override it with a custom string.

Encoding (optional)

Default encoding is UTF-8. Set this to a standard charset name if you want to override the default. All messages would need to support the encoding configured for the input. UTF-8 encoded messages shouldn’t be sent to an input configured to support UTF-16.

Decompressed size limit

The maximum size of the message after being decompressed.

Custom Kafka properties (optional)

Provide additional properties to Kafka by separating them in a new line.

The GELF TCP input allows Graylog to receive Common Event Format (CEF) messages over a reliable, connection-oriented TCP stream.

After launching your new input, configure the following fields based on your preferences: 

Configuration Option Description

Global

Select this check box to enable the input on all Graylog nodes, or keep it unchecked to enable the input on a specific node.

Title  

Assign a unique title to the input. Example: GELF TCP Input for XYZ Source.
Bind address

Enter an IP address on which this input listens. The source system/data sends logs to this IP/input.
Port Enter a port to use in conjunction with the IP address.
Receive Buffer Size (optional) Depending on the amount of traffic being ingested by the input, this value should be large enough to ensure proper flow of data but small enough to prevent the system from spending resources trying to process the buffered data.
No. of worker threads (optional) This setting controls how many concurrent threads are used to process incoming data. Increasing the number of threads can enhance data processing speed, resulting in improved throughput. The ideal number of threads to configure depends on the available CPU cores on your Graylog server. A common starting point is to align the number of worker threads with the number of CPU cores. However, it is crucial to strike a balance with other server demands.

TLS cert file (optional)

The certificate file that is stored on a Graylog system. The value of this field is a path (/path/to/file) that Graylog should have access to.

TLS private key file (optional)

The certificate private key file that is stored on a Graylog system. The value of this field is a path (/path/to/file) that Graylog should have access to.

Enable TLS

 Select if this input should use TLS.

TLS key password (optional)

The private key password.

TLS client authentication (optional)

If you want to require the source of the messages sending logs to this input to authenticate themselves, set to optional or required.

TLS Client Auth Trusted Certs (optional)

The path where the client (source) certificates are located on a Graylog system. The value of this field is a path (/path/to/file) which Graylog should have access to.

TCP keepalive

Enable this option if you want the input to support TCP keep-alive packets to prevent idle connections.

Null frame delimeter

This option is typically left unchecked. New line is the delimiter for each message.

Maximum message size

The maximum message size of the message. A default value should suffice but can be modified depending on message length. Each input type usually has specifications that note the maximum length of a message.

Override Source

By default, messages parse the source field as the provided hostname in the log message. However, if you want to override this setting for devices that output non-standard or unconfigurable hostnames, you can set an alternate source name here.

Encoding

All messages would need to support the encoding configured for the input. UTF-8 encoded messages shouldn’t be sent to an input configured to support UTF-16.

Decompressed size limit

The maximum size of the message after being decompressed.

The GELF UDP Input allows Graylog to ingest GELF (Graylog Extended Log Format) messages over UDP.

After launching your new input, configure the following fields based on your preferences: 

Configuration Option Description

Global

Select this check box to enable the input on all Graylog nodes, or keep it unchecked to enable the input on a specific node.

Title  

Assign a unique title to the input. Example: GELF UDP Input for XYZ Source.
Bind address

Enter an IP address on which this input listens. The source system/data sends logs to this IP/input.
Port Enter a port to use in conjunction with the IP address.
Receive Buffer Size (optional) Depending on the amount of traffic being ingested by the input, this value should be large enough to ensure proper flow of data but small enough to prevent the system from spending resources trying to process the buffered data.
No. of worker threads (optional) This setting controls how many concurrent threads are used to process incoming data. Increasing the number of threads can enhance data processing speed, resulting in improved throughput. The ideal number of threads to configure depends on the available CPU cores on your Graylog server. A common starting point is to align the number of worker threads with the number of CPU cores. However, it is crucial to strike a balance with other server demands.

Override Source

By default, messages parse the source field as the provided hostname in the log message. However, if you want to override this setting for devices that output non-standard or unconfigurable hostnames, you can set an alternate source name here.

Encoding

All messages would need to support the encoding configured for the input. UTF-8 encoded messages shouldn’t be sent to an input configured to support UTF-16.

Decompressed size limit

The maximum size of the message after being decompressed.

 

Next Steps

After you complete input setup, visit Input Diagnosis for testing and validation of the new input. Use this functionality to help troubleshoot any connection issues.

Further Reading

Explore the following additional resources and recommended readings to expand your knowledge on related topics: