Azure Event Hubs Input

The following article exclusively pertains to a Graylog Enterprise feature or functionality. To learn more about obtaining an Enterprise license, please contact the Graylog Sales team.

Azure Event Hubs is a fully managed, real-time data ingestion service that collects event logs from a wide range of Azure services. The Graylog Azure Event Hubs input supports the ability to retrieve event hub events and process them within Graylog.

Prerequisites

Before proceeding, ensure that the following prerequisites are met:

You must have an Azure subscription with a properly configured Event Hubs. Please see the Azure Event Hubs documentation for assistance in setting up Event Hubs. You may also find this overview of features and terminologies helpful.

Supported Log Types

This input supports collecting the following log types:

Azure Active Directory (audit and sign in logs)

Azure Audit

Azure Network Watcher

Azure Kubernetes Service

Azure SQL

Required Third-Party Setup

To enable integration, complete the following required setup with your third-party service:

Add a Shared Access Signature policy to allow the input to access and communicate with your Event Hub. (Before creating a policy, please consult the Azure documentation for security and management best practices.)
Select the Listen permission (Graylog will only need to read events from Event Hub).
A Consumer group is required for the Azure Event Hubs input to read events from Event Hub. Azure creates a $Default consumer group, which is sufficient for Graylog to read and ingest logs. You can also define a custom consumer group, which may also be specified in the Graylog configuration. The Graylog Azure Event Hubs input supports running on multiple nodes. The number of Graylog nodes that can process messages is equal to the number of partitions configured in Event Hubs.

Hint: The input supports the ability to specify a “forward proxy," which can be used to relay Azure communication through a proxy host. Only HTTPs-capable forward proxies should be utilized.

When proxy support is enabled, the connection to Azure will be made over port 443 using the “AMQP over Websockets” protocol.

Required Configuration Values

In your third-party configuration, make note of the following values that are required when configuring the input in Graylog:

The primary or secondary connection string. The connection string is needed to configure the input within Graylog.

Input Type

This input is a pull input type. See Inputs to learn about input types.

Input Configuration

Follow the input setup instructions. During setup of this input, you can configure the following options:

Parameter	Description
Input Name	Provide a unique name for your new Azure Event Hubs input.
Azure Event Hub Name	The name of your Event Hub within the Azure console.
Connection String	The primary or secondary connection string as defined in the Shared Access Signature policy above in the configuration. (Note that for Graylog Cloud, this remains encrypted.)
Consumer Group	The consumer group from which to read events. Use `$Default` if you have not defined a custom consumer group for your event hub. This input currently only supports running on a single Graylog node, so there is no need to configure a consumer group with additional concurrent readers.
Proxy URI	If enabled, this refers to the HTTPS forward proxy URI for Azure communication.
Content Type	Specifies the format used to send data to the input via Azure Event Hubs, enabling support for custom payloads. Here are some examples of the supported formats: Azure Services: `{ "records": [ { "time": "2021-10-04T09:37:55.2484299Z", "resourceId": "/tenants/df620235-50d7-4400-bb7e-3b112e9b1ff4/providers/Microsoft.aadiam", "operationName": "Add service principal", "operationVersion": "1.0", "category": "AuditLogs", "tenantId": "df620235-50d7-4400-bb7e-3b112e9b1ff4" } ] }` JSON Array: Copy `[ { "id": 1, "message": "First event" }, { "id": 2, "message": "Second event" } ]` or Copy `[ "First event", "Second event", ]` Raw Text/Single JSON Document: Copy `{ "event": "UserLogin", "user": "alice", "timestamp": "2024-06-01T12:00:00Z" }` or Copy `literally anything as one message` GELF (Newline Delimited): Copy `{ "version": "1.1", "host": "example.org", "short_message": "A short message", "level": 5, "_some_info": "foo" } { "version": "1.1", "host": "example.org", "short_message": "A short message 2", "level": 5, "_some_info": "foo" }` Text (Newline Delimited): Copy `First log line Second log line Third log line`
Maximum Batch Size	The maximum batch size to wait for when the input reads Event Hub. The input will block and wait for the specified batch size to be reached before querying the event hub.
Maximum Wait Time	The maximum time to wait for the Maximum Batch Size above to be reached.
Store Full Message	Stores the entire message payload received from Azure Event Hubs.

Next Steps

After you complete input setup, visit Input Diagnosis for testing and validation of the new input. Use this functionality to help troubleshoot any connection issues.