Kafka Output

The following article exclusively pertains to a Graylog Enterprise feature or functionality. To learn more about obtaining an Enterprise license, please contact the Graylog Sales team.

The Kafka Output forwards messages from a Graylog stream to a topic in an Apache Kafka cluster, so downstream systems can consume log data directly from Kafka.

This article explains how to set up a Kafka output, preserve message order with a message key, and secure the connection to Kafka.

Prerequisites

Before proceeding, ensure that the following prerequisites are met:

  • You have a Graylog Enterprise license.

  • You have permission to create outputs and manage stream outputs in Graylog.

  • Every Graylog node can reach at least one Kafka broker over the network on the broker's listener port (by default, 9092).

  • The destination Kafka topic exists. If the brokers allow automatic topic creation, Kafka creates the topic on the first message. Otherwise, messages queue in the output journal until the topic exists.

  • If your Kafka cluster uses TLS, ensure that the truststore (and, for mutual TLS, the keystore) is stored at the same local path on every Graylog node. This also applies to SASL connections that run over TLS. For details, see Secure the Connection to Kafka.

Set Up a Kafka Output

To create a new Kafka output:

  1. Navigate to System > Outputs.

  2. Select Kafka [Enterprise] from the output type drop-down list.

  3. Click Launch new output. The Create new Output dialog opens.

  4. Complete the output configuration details:

    Configuration Option Description

    Title

    Enter a descriptive name for the output. The name identifies the output on the Outputs in Cluster page and in each stream's output list.

    Send Buffer Size

    Enter the number of messages the sender ring buffer can hold while they wait to be sent. The default is 4096. This value must be a power of two, or the output does not start.

    If you increase the number of concurrent formatters or senders, increase this value as well.

    Number of concurrent message processing pipelines

    Enter the number of output pipeline instances that can run at the same time. The default is 2.

    If you set this value to 0, pipeline processing is skipped, even if you select a pipeline in Output Processing Pipeline.

    Number of concurrent output payload formatters

    Enter the number of payload formatter instances that can run at the same time. Formatters convert each message into the selected Outbound Payload Format. The default is 2.

    This value must be at least 1. If you set it to 0, the output fails.

    Number of concurrent message senders

    Enter the number of sender instances that transmit messages to Kafka at the same time. The default is 2.

    This value must be at least 1. Increase it to raise throughput if messages back up in the journal, provided your Kafka brokers can handle the extra load. To preserve message order per key, set it to 1. See Preserve Message Order with a Message Key.

    Journal Segment Size

    Enter the soft maximum size of a single journal segment file. The default is 100mb.

    Journal Segment Age

    Enter the maximum amount of time Graylog writes to a journal segment file before it starts a new one. The default is 1h.

    Maximum Journal Size

    Enter the maximum size of the on-disk journal for this output. The default is 5gb.

    This is a soft limit, and the journal can grow larger. To make sure journal data is cleaned up in a timely way, also set Maximum Journal Message Age and Journal Segment Age.

    Maximum Journal Message Age

    Enter the maximum amount of time a message is kept in the on-disk journal. The default is 12h.

    Messages that are still undelivered after this time are purged. Set this value longer than the longest Kafka outage you need to tolerate.

    Journal Message Flush Interval

    Enter the number of messages that can be written to the journal before Graylog forces a flush to disk. The default is 1000000.

    Maximum Journal Flush Age

    Enter the maximum amount of time journal data can remain in memory before Graylog forces a flush to disk. The default is 1m.

    Journal Buffer Size

    Enter the size of the buffer that holds messages before they are written to the journal. The default is 1024. This value must be a power of two.

    A larger buffer uses more memory. Review your journal size and sender settings before you increase it.

    Number of Handlers Journal Buffer Encoders

    Enter the number of concurrent encoders that write messages from the journal buffer to the journal. The default is 3, which is sufficient for most environments.

    Increase this value if messages are not flowing into the journal fast enough.

    Outbound Payload Format

    Select the format Graylog uses to build each Kafka record value. This field has no default value, so you must select a format:

    • Convert to JSON: Converts the message's key-value pairs into a JSON object.

    • Pass-through Formatter: Performs no formatting and generates no payload. This formatter is intended for the Google Cloud BigQuery Output. If you select it for a Kafka output, the record payload is empty.

    • GELF: Encodes the message in GELF.

    • Use pipeline-generated 'pipeline_output' field: Uses the value your output pipeline writes to the pipeline_output field as the payload. Use the set_field function in the pipeline to populate this field.

    • Use the 'full_message' field: Uses the content of the full_message field as the payload. Messages without this field, or with an empty value, are ignored. The input must have Store full message enabled.

    For more information on each format, see Configure the Outbound Payload.

    Bootstrap Servers

    Enter a comma-separated list of one or more Kafka brokers in the format host1:port1,host2:port2. The default is 127.0.0.1:9092.

    Graylog uses these brokers to discover the rest of the Kafka cluster, so you do not need to list every broker. List more than one to avoid a single point of failure when the output connects.

    Topic

    Enter the name of the Kafka topic that messages are produced to.

    Output Processing Pipeline (optional)

    Select a pipeline to filter messages or to enrich or modify message content before the messages are sent to Kafka. See Enterprise Output Framework for details on output pipelines.

    Message key field (optional)

    Enter the name of a message field whose value is used as the Kafka record key. Messages that share a key are routed to the same partition. Leave this field empty to send records without a key.

    Per-key ordering also requires setting Number of concurrent message senders to 1. See Preserve Message Order with a Message Key.

    Custom Kafka properties (optional)

    Enter Kafka producer properties, one key=value pair per line, for example security.protocol=SASL_SSL. Use this field to configure TLS, SASL authentication, compression, batching, and any other producer setting.

    Properties entered here override the other settings in this form. If the Kafka client rejects these properties, for example because a value is invalid, the output does not start and the error is written to the server log. See Secure the Connection to Kafka.

    Hint: The buffer, concurrency, and journal fields are shared by all Enterprise outputs, and their default values suit most environments. For guidance on tuning them for higher throughput, see Enterprise Output Framework.

  5. Click Create output.

    The new output appears in the list on the Outputs in Cluster page.

Important: The output does not send any messages until you assign it to a stream. See Assign the Output to a Stream.

Assign the Output to a Stream

An output only receives messages from the streams it is assigned to. Every message routed to an assigned stream is forwarded to Kafka.

To assign the Kafka output to a stream:

  1. Navigate to Streams.

  2. Locate the stream whose messages you want to forward and click More.

  3. Select Manage Outputs.

  4. Select your Kafka output from the Select existing output drop-down list.

  5. Click Assign existing Output.

    The Kafka output is added to the list of outputs for the stream, and messages routed to the stream begin to flow to Kafka.

For more information on assigning and removing outputs, see Outputs.

Verify Message Delivery

After you assign the output to a stream that is receiving messages, confirm that the messages arrive in Kafka and that the output journal is not backing up.

To verify message delivery:

  1. On a host with the Kafka command-line tools, read the topic with the Kafka console consumer, replacing the broker address and topic name with your own values:

    Copy 
    kafka-console-consumer.sh --bootstrap-server <broker-host>:9092 --topic <topic-name> --from-beginning

    Messages from the assigned stream appear in the consumer output in the payload format you selected.

  2. In Graylog, navigate to System > Cluster Configuration.

  3. In the Graylog Nodes table, locate the node and click More.

  4. Select Metrics. The metrics page for the node opens.

  5. Enter KafkaOutput in the filter field and locate the metric ending in .journal.uncommittedMessages. The name includes the output ID, for example, org.graylog.enterprise.integrations.outputs.journaled.KafkaOutput.<output-id>.journal.uncommittedMessages. Review the metric on each node, because each node has its own output journal.

    When delivery is healthy, the value stays at or near 0. A steadily increasing value means messages are queuing in the journal because Graylog cannot deliver them to Kafka.

Preserve Message Order with a Message Key

Kafka guarantees message order only within a single partition. When you set the Message key field, Graylog uses the value of that field as the record key, and Kafka routes all records with the same key to the same partition. For example, setting the key field to source keeps all messages from one host together in one partition.

Keying alone does not guarantee per-key ordering, because the Enterprise Output Framework sends messages with several concurrent senders by default. To preserve per-key order, also set Number of concurrent message senders to 1. Order is preserved only among messages handled by the same Graylog node.

Hint: A single sender reduces throughput. Use per-key ordering only when downstream consumers depend on it.

If a message does not contain the key field, Graylog sends it without a key.

Secure the Connection to Kafka

The Kafka Output does not have dedicated TLS or mutual TLS (mTLS) fields. Instead, you configure encryption and authentication with standard Apache Kafka producer properties in the custom Kafka properties field. For the available properties, see the Apache Kafka documentation on Encryption and Authentication using SSL and Authentication using SASL. Paths in these properties refer to files on the Graylog node, so place the files at the same location on every node.

Warning: Values in Custom Kafka properties are not encrypted. Anyone who can view the output configuration can see any passwords you enter here, including keystore, truststore and SASL passwords. Limit access to this output to trusted users.

Encrypt the Connection with TLS

To connect to brokers that use TLS, add the following properties, replacing the path and password with your own values:

Copy 
security.protocol=SSL
                ssl.truststore.location=/etc/graylog/kafka/kafka.truststore.jks
            ssl.truststore.password=<truststore-password>

Authenticate with Mutual TLS

If your brokers require client certificates, add a keystore that contains the Graylog client certificate and private key:

Copy 
security.protocol=SSL
                ssl.truststore.location=/etc/graylog/kafka/kafka.truststore.jks
                ssl.truststore.password=<truststore-password>
                ssl.keystore.location=/etc/graylog/kafka/kafka.keystore.jks
                ssl.keystore.password=<keystore-password>
            ssl.key.password=<key-password>

Authenticate with SASL

To authenticate with a username and password over an encrypted connection, add the following properties. This example uses the SCRAM-SHA-512 mechanism:

Copy 
security.protocol=SASL_SSL
                sasl.mechanism=SCRAM-SHA-512
                sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="<username>" password="<password>";
                ssl.truststore.location=/etc/graylog/kafka/kafka.truststore.jks
            ssl.truststore.password=<truststore-password>

For the full list of supported security properties, see the Apache Kafka producer configuration reference.

Troubleshooting

Messages Are Queuing in the Output Journal

The uncommitted messages metric for the Kafka output's journal keeps increasing, and no new messages arrive in the Kafka topic.

Graylog cannot reach the Kafka brokers or cannot authenticate to them. Verify that every Graylog node can reach the hosts in Bootstrap Servers, and that any TLS or SASL settings in Custom Kafka properties are correct. Check the Graylog server log for Kafka producer errors. When the connection is restored, Graylog sends the queued messages and the metric returns to 0.

Warning: The journal does not hold messages indefinitely. Messages that remain undelivered longer than Maximum Journal Message Age (default: 12h) are purged. For longer planned Kafka maintenance windows, increase this value and Maximum Journal Size in advance.

Further Reading

Explore the following additional resources and recommended readings to expand your knowledge on related topics: