Beats Kafka Input

The Beats Kafka input supports collecting logs from Kafka topics. When logs are generated by Beats data shippers and pushed to a Kafka topic, they are automatically ingested and parsed by this input.

Prerequisites

  • Install Beats, Kafka, and Zookeeper.

  • Provide full access permissions to all Kafka and Filebeat folders.

  • Configure the filebeats.yml file as shown below:

    Copy 
    filebeat.inputs:
    - type: log
      enabled: true
      paths:
        - /var/log/syslog
       
    output.kafka:
      hosts: ["your_kafka_host:9092"] # Replace with your Kafka host(s)
      topic: 'system_logs' # Name of the Kafka topic
      codec.json:
        pretty: false
        preset: balanced

  • Configure the Kafka server.properties file:

    advertised.listeners=PLAINTEXT://localhost:9092

  • Create a Kafka topic:

    Go to the Kafka directory bin folder and execute the following command:

    Copy 
    ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic <Topic name>

Hint: Remember to replace localhost with your unique IP address.

Graylog Input Configuration

When launching a new input from the Graylog Inputs tab, the following options are available:

Parameter Description
Node Select the node on which the input should start.
Title Enter a unique name for the input.
Bootstrap Servers Enter the IP address and port on which the Kafka server is running.
Zookeeper address (optional) Enter the IP address and port on which the Zookeeper server is running.
Topic filter regex Enter the topic name filter that is configured in the filebeats.yml file.
Fetch minimum bytes

Enter the minimum byte size a message batch should reach before fetching.
Fetch maximum wait time

Enter the maximum time (in milliseconds) to wait before fetching.
Processor threads

Enter the number of threads to process. This setting is based on the number of partitions available for the topic.

Auto offset reset (optional)

Choose the appropriate selection from the drop-down menu if there is no initial offset in Kafka or if an offset is out of range.

Consumer group identifier (id) (optional)

Enter the name of the consumer group the Kafka input belongs to.

Override source (optional)

Enter the default hostname derived from the received packet. Only set this source if you want to override it with a custom string.

Encoding (optional)

Default encoding is UTF-8. Set this to a standard charset name if you want to override the default.

Custom Kafka properties (optional)

Provide additional properties to Kafka by separating them in a new line.