AWS Kinesis CloudWatch Input

The AWS Kinesis/CloudWatch input allows Graylog to read log messages from CloudWatch via Kinesis.

Prerequisites

Before proceeding, ensure that the following prerequisites are met:

Kinesis is required to stream messages to Graylog before messages can be read from CloudWatch.

Supported Log Types

This input supports collecting the following log types:

CloudWatch Logs: Raw text strings within CloudWatch.
CloudWatch Flow Logs: Flow Logs within a CloudWatch log group.
Kinesis Raw Logs: Raw text strings written to Kinesis.

Required Third-Party Setup

To enable integration, complete the following required setup with your third-party service:

Follow AWS documentation for pushing CloudWatch logs to Kinesis.
In the AWS Create Kinesis subscription filter wizard, select Other as the log format and accept the default options. Optionally, apply a subscription filter pattern to control which events are forwarded.
Add the minimum required IAM permissions to the IAM role that Graylog assumes. See the AWS documentation for a full description of the required permissions.
For automatic setup, ensure the IAM role includes all the manual setup permissions, plus the following additional permissions:
- iam:CreateRole
- iam:GetRole
- iam:PassRole
- iam:PutRolePolicy
- kinesis:CreateStream
- kinesis:DescribeStream
- kinesis:GetRecords
- kinesis:GetShardIterator
- kinesis:ListShards
- kinesis:ListStreams
- logs:DescribeLogGroups
- logs:PutSubscriptionFilter

Input Type

This input is a pull input type. See Inputs to learn about input types.

Input Configuration

Follow the input setup instructions. During setup of this input, you can configure the following options:

Configuration Option	Description
Input Name	Provide a unique name for your new input.
AWS Authentication Type	Select either to allow the system automatically look for credentials using the AWS default credential provider chain or provide AWS Access and Secret Keys.
AWS assume role (ARN)	The ARN of the Identity and Access Management (IAM) role that Graylog will assume to access the configured Kinesis stream and related CloudWatch resources. AWS recommends using IAM roles with temporary credentials instead of long-term static access keys. This option is preferred and supports cross-account access.
AWS Region	Select the AWS region where the S3 bucket storing logs resides.
Optional AWS VPC Endpoints	These settings let you override the default AWS public API endpoints with VPC endpoint URLs, which is useful when you want traffic to AWS services to stay inside your private network rather than going over the public internet.
Select Stream	Choose an existing Kinesis stream from your AWS account. Graylog will pull messages from this stream. If no suitable stream is available, you can select Setup Kinesis Automatically to create and configure one.
AWS Message Type	This determines the type of AWS message this input receives. There are three supported message types: Kinesis Raw: Select this when your Kinesis stream contains raw, uncompressed data (not routed through CloudWatch Logs). If no messages are detected in your selected stream, Graylog automatically defaults to this option. Kinesis CloudWatch Flow Log: Select this option if the source log group contains VPC Flow Logs. Kinesis CloudWatch Raw: Select this when your Kinesis stream receives general CloudWatch Logs (e.g., application logs, Lambda logs) via a subscription filter, but the data is not VPC Flow Logs.
Advanced Options	Enable Throttling: Enables Graylog to stop reading new data for this input whenever the system falls behind on message processing and needs to catch up. Add Flow Log field name prefix: Adds a prefix flow_log_ to AWS Flow Log field names. For example, src_addr becomes flow_log_src_addr. This helps differentiate AWS flow log fields from other similarly named fields. Override source (optional): By default, the source is a hostname derived from the received packet. You can override the default value with a custom string. This option allows you to optimize the source for your specific needs. Kinesis Record Batch Size: Sets the number of Kinesis records fetched in each polling batch. Each record can be up to 1 MB in size.

Next Steps

After you complete input setup, visit Input Diagnosis for testing and validation of the new input. Use this functionality to help troubleshoot any connection issues.