Beats Kafka Input

Beats are lightweight data shippers that collect and forward different types of data.

The Beats Kafka Input enables Graylog to receive and parse data from any Beats data shipper via a Kafka stream. This input combines the capabilities of Graylog's existing Beats Input (which parses structured data from Beats) with its Kafka Input (which ingests raw data from Kafka topics).

While Graylog can ingest data directly from Beats or from Kafka, some users choose to route Beats data through Kafka before sending it to Graylog. This approach offers several advantages such as:

Helps buffer and regulate traffic flow.
Reduces the risk of data loss during ingestion spikes.
Prevents the Graylog journal from filling up under high load.

Prerequisites

Before proceeding, ensure that the following prerequisites are met:

You must have a configured and working Kafka cluster.
You have Beats installed to the log source.
You have installed Zookeeper.
You must provide full access permissions to all Kafka and Filebeat folders.

Required Third-Party Setup

To enable integration, complete the following required setup with your third-party service:

Hint: Remember to replace localhost with your unique IP address.

Configure the filebeats.yml file as shown below:

Copy

filebeat.inputs:
- type: filestream
  id: var-log
  paths:
    - /var/log/syslog
   
output.kafka:
  hosts: ["your_kafka_host:9092"] # Replace with your Kafka host(s)
  topic: 'system_logs' # Name of the Kafka topic
  codec.json:
    pretty: false
    preset: balanced

Configure the Kafka server.properties file:

advertised.listeners=PLAINTEXT://localhost:9092

Create a Kafka topic by navigating to the Kafka directory bin folder and executing the following command:

Copy

./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic <Topic name>

Required Configuration Values

In your third-party configuration, make note of the following values that are required when configuring the input in Graylog:

Consumer group identifier ID

Input Type

This input is a pull input type. See Inputs to learn about input types.

Input Configuration

Follow the input setup instructions. During setup of this input, you can configure the following options:

Parameter	Description
Node	Select the node on which the input should start.
Title	Enter a unique name for the input.
Bootstrap Servers	Enter the IP address and port on which the Kafka server is running.
Zookeeper address (optional)	The IP address and port on which the Zookeeper server is running.
Topic filter regex	Enter the topic name filter that is configured in the `filebeats.yml` file.
Fetch minimum bytes	Minimum byte size a message batch should reach before fetching.
Fetch maximum wait time	Enter the maximum time (in milliseconds) to wait before fetching.
Processor threads	Enter the number of threads to process. This setting is based on the number of partitions available for the topic.
Auto offset reset (optional)	Choose the appropriate selection from the drop-down menu if there is no initial offset in Kafka or if an offset is out of range.
Consumer group identifier id (optional)	The name of the consumer group the Kafka input belongs to.
Override source (optional)	Enter the default hostname derived from the received packet. Only set this source if you want to override it with a custom string.
Encoding (optional)	Default encoding is UTF-8. Set this to a standard charset name if you want to override the default.
Custom Kafka properties (optional)	Provide a newline separated list of Kafka properties. (e.g. `ssl.keystore.location=/etc/graylog/server/kafka.keystore.jks`).

Next Steps

After you complete input setup, visit Input Diagnosis for testing and validation of the new input. Use this functionality to help troubleshoot any connection issues.