Azure Event Hubs Input

This is a Graylog Enterprise feature. A valid Graylog Enterprise license is required.

Azure Event Hub is a fully managed, real-time data ingestion service that supports the ability to receive various types of event logs from various Azure services. The Graylog Azure Event Hubs input supports the ability to retrieve event hub events and process them within Graylog.

Prerequisites

An existing Azure subscription with a properly configured Event Hub is required to use the Azure Event Hubs input. Please see the Azure Event Hub documentation for assistance in setting up Event Hub. You may also find this overview of features and terminologies helpful.

Azure Event Hubs Input Configuration

When Azure Event Hub is set up and receiving log events from various sources, perform the following configuration steps for the Graylog Azure Log Events input to connect to and read events from your event hub.

  1. Add a Shared Access Signature policy to allow the input to access and communicate with your Event Hub. (Before creating a policy, please consult the Azure documentation for security and management best practices.)

  2. To create a policy, click the Shared access policies option from the left Event Hub navigation bar. Click the New button at the top to create the policy.

  1. Select the Listen permission (Graylog will only need to read events from Event Hub).

  1. Once the policy is defined, take note of either the primary or secondary connection string. The connection string is needed to configure the input within Graylog.

Consumer Groups

A Consumer group is required for the Azure Event Hubs input to read events from Event Hub. Azure creates a $Default consumer group, which is sufficient for Graylog to read and ingest logs. If you have defined a custom consumer group, it may also be specified within the Graylog configuration.

The Graylog Azure Event Hubs input currently only supports running on a single Graylog node, so there is no need to configure a consumer group with additional concurrent readers at this time.

Graylog Configuration

When launching a new Azure Event Hubs input from the Graylog Inputs tab, the following parameters will need to be completed:

Parameter Description
Input Name Provide a unique name for your new Azure Event Hubs input.
Azure Event Hub Name The name of your Event Hub within the Azure console.
Connection String The primary or secondary connection string as defined in the Shared Access Signature policy above in the configuration. (Note that for Graylog Cloud, this remains encrypted.)
Consumer Group

The consumer group from which to read events. Use $Default if you have not defined a custom consumer group for your event hub.

Proxy URI

If enabled, this refers to the HTTPS forward proxy URI for Azure communication.
Maximum Batch Size The maximum batch size to wait for when the input reads Event Hub. The input will block and wait for the specified batch size to be reached before querying the event hub.
Maximum Wait Time The maximum time to wait for the Maximum Batch Size above to be reached.
Store Full Message Stores the entire message payload received from Azure Event Hubs.

 

Multi-Node Support

The Global configuration option, which allows for the input to be run across multiple Graylog nodes, is enabled by default. Multiple Event Hub Partitions must be specified on the target event hub since one partition will only be read by one Graylog node at a time. Please note that at least one partition per Graylog node would be needed for all nodes to help balance the load.

Hint: Please note that multi-node and proxy support (detailed in the following section) were introduced in the Graylog 5.0.4 release.

Proxy Support

The input supports the ability to specify a “forward proxy," which can be used to relay Azure communication through a proxy host. Only HTTPs-capable forward proxies should be utilized.

When proxy support is enabled, the connection to Azure will be made over port 443 using the “AMQP over Websockets” protocol.

Store Full Message

Azure Event Hub supports the option to store full messages from Azure log data, which allows you to manually parse data received from all Azure log message types utilizing processing pipelines. To enable this option, select "Store Full Message" in the Azure Event Hub Integrations menu.

Azure Event Hub Event Sources

This input currently supports parsing and ingesting the following types of Azure event logs. Please see the Azure documentation for instructions on how to Forward events from these services to Event Hub.

  • Azure Active Directory (audit and sign in logs)
  • Azure Audit
  • Azure Network Watcher
  • Azure Kubernetes Service
  • Azure SQL