Azure Event Hub is a fully managed, real-time data ingestion service that supports the ability to receive various types of event logs from various Azure services. The Graylog Azure Event Logs input supports the ability to retrieve Event Hub events and process them within Graylog.

Prerequisites

An existing Azure subscription with a properly configured Event Hub is required to use the Azure Event Logs input. Please see the Azure Event Hub documentation for help to set up Event Hub. You may also find this overview of features and terminologies helpful.

Azure Event Hub Configuration

When Azure Event Hub is set up and receiving log events from various sources, perform the following configuration steps for the Graylog Azure Log Events input to connect to, and read events from, your event hub.

Steps

  1. Add a Shared Access Signature policy to allow Graylog Azure Logs input to access and communicate with your Event Hub. Before creating a policy, please consult the Azure documentation for security and management best practices.

  2. To create a policy, click the Shared access policies option from the left Event Hub navigation bar. Click the New button at the top to create the policy.

azure_sas_policy_config

  1. Select the Listen permission (Graylog will only need to read events from Event Hub).

azure_sas_policy_name

  1. Once the policy is defined, take note of either the primary or secondary connection string. The connection string is needed to configure the input within Graylog.

Consumer Groups

A Consumer group is required for the Azure Logs input to read events from Event Hub. Azure creates a $Default consumer group, which is sufficient for Graylog to read and ingest logs. If you have defined a custom consumer group, it may also be specified within the Graylog configuration.

The Graylog Azure Logs input currently only supports running on a single Graylog node, so there is no need to configure a consumer group with additional concurrent readers at this time.

Plugin Configuration

Please review this table defining plugin configuration parameters.

Parameter Description
Input Name Provide a unique name for your new Azure Event Logs input.
Azure Event Hub Name The name of your Event Hub within the Azure console.
Connection String The primary or secondary connection string as defined in the Shared Access Signature policy above in the configuration.
Consumer Group The consumer group from which to read events. Use $Default if you have not defined a custom consumer group for your event hub.
Polling Interval (minutes) How often to query the Azure Event hub for new events. We suggest the default of 5 minutes to avoid hitting Azure rate limits.
Maximum Batch Size The maximum batch size to wait for when the input reads Event Hub. The input will block and wait for the specified batch size to be reached before querying the event hub.
Maximum Wait Time The maximum time to wait for the Maximum Batch Size above to be reached.
Store Full Message Stores the entire message payload received from Azure Logs.

Store Full Message

Introduced in Graylog 4.3, Azure Event Hub supports the option to store full messages from Azure log data, which allows you to manually parse data received from all Azure log message types utilizing processing pipelines. To enable this option, select "Store Full Message" in the Azure Event Hub Integrations menu.

Azure_Store_Full_Message

Azure Event Hub Event Sources

This input currently supports parsing and ingesting the following types of Azure event logs. Please see the Azure documentation for instructions on how to Forward events from these services to Event Hub.

  • Azure Active Directory (Audit and Sign-in logs)
  • Azure Audit
  • Azure Network Watcher
  • Azure Kubernetes Service
  • Azure SQL