AWS Kinesis CloudWatch Input

The AWS Kinesis/CloudWatch input allows Graylog to read log messages from CloudWatch via Kinesis.

Prerequisites

Before proceeding, ensure that the following prerequisites are met:

Kinesis is required to stream messages to Graylog before messages can be read from CloudWatch.

Supported Log Types

This input supports collecting the following log types:

CloudWatch Logs: Raw text strings within CloudWatch.
CloudWatch Flow Logs: Flow Logs within a CloudWatch log group.
Kinesis Raw Logs: Raw text strings written to Kinesis.

Required Third-Party Setup

To enable integration, complete the following required setup with your third-party service:

Follow AWS instructions for pushing CloudWatch logs to Kinesis. This process of pushing CloudWatch logs to Kinesis can also be automated.
Apply changes to permissions:
- AWS lists the manual setup AWS permissions and provides a full description of the permissions and an example policy. These are the minimum permissions needed for the input to run.
The automatic setup requires the same permissions as the manual setup, as well as additional permissions, which are listed below:

iam:CreateRole

iam:GetRole

iam:PassRole

iam:PutRolePolicy

kinesis:CreateStream

kinesis:DescribeStream

kinesis:GetRecords

kinesis:GetShardIterator

kinesis:ListShards

kinesis:ListStreams

logs:DescribeLogGroups

logs:PutSubscriptionFilter

Required Configuration Values

In your third-party configuration, make note of the following values that are required when configuring the input in Graylog:

AWS access key
AWS secret key

Input Type

This input is a pull input type. See Inputs to learn about input types.

Input Configuration

Follow the input setup instructions. During setup of this input, you can configure the following options:

Configuration Option	Description
Input Name	Provide a unique name for your new input.
AWS Authentication Type	Select either to allow the system automatically looks for credentials using the AWS default credential provider chain or explicitly provide AWS Access and Secret Keys.
AWS Assume Role (ARN)	This setting is often used for cross-account access.
AWS Region	Select the AWS region where the S3 bucket storing logs resides.
Optional AWS VPC Endpoints	These settings let you override the default AWS public API endpoints with VPC endpoint URLs, which is useful when you want traffic to AWS services to stay inside your private network rather than going over the public internet.
Enable Throttling	Enables Graylog to stop reading new data for this input whenever the system falls behind on message processing and needs to catch up.
AWS access key (optional)	The unique identifier created for the AWS Identity and Access Management (IAM) user. The Credential settings retrieval order documentation provides more information.
AWS secret key (optional)	The access key ID for the IAM user with permission to the subscriber and the SQS queue.
Override source (optional)	By default, the source is a hostname derived from the received packet. You can override the default value with a custom string. This option allows you to optimize the source for your specific needs.
Encoding (optional)	All messages need to support the encoding configured for the input. For example, UTF-8 encoded messages should not be sent to an input configured to support UTF-16.

Next Steps

After you complete input setup, visit Input Diagnosis for testing and validation of the new input. Use this functionality to help troubleshoot any connection issues.