AWS Kinesis/CloudWatch Input

The AWS Kinesis/CloudWatch input allows Graylog to read log messages from CloudWatch via Kinesis.

Prerequisites

Before proceeding, ensure that the following prerequisites are met:

Kinesis is required to stream messages to Graylog before messages can be read from CloudWatch.

Supported Log Types

This input supports collecting the following log types:

CloudWatch Logs: Raw text strings within CloudWatch.
CloudWatch Flow Logs: Flow Logs within a CloudWatch log group.
Kinesis Raw Logs: Raw text strings written to Kinesis.

Required Third-Party Setup

To enable integration, complete the following required setup with your third-party service:

The Graylog AWS Kinesis/CloudWatch input is built on the AWS Kinesis Client Library (KCL). In addition to reading from the Kinesis stream, the KCL uses Amazon DynamoDB to track its progress and Amazon CloudWatch to publish its operational metrics. This means the IAM role or user that Graylog uses needs permissions for Kinesis, DynamoDB, and CloudWatch, not just Kinesis. For the full set of permissions required for the input to run, see the AWS KCL permissions documentation page.

Important: Starting with Graylog 7.2, the input uses KCL 3.5, which requires additional DynamoDB permissions. These requirements apply to every Kinesis/CloudWatch input, including inputs created before you upgraded. See DynamoDB Permissions for details.

To stream CloudWatch logs to Kinesis and give Graylog the permissions it needs to read them, complete the following steps in AWS:

Follow AWS documentation for pushing CloudWatch logs to Kinesis.
In the AWS Create Kinesis subscription filter wizard, select Other as the log format and accept the default options. This ensures logs are delivered to Graylog in the expected format. Optionally, apply a subscription filter pattern to control which events are forwarded. On the Graylog input, set the AWS Message Type to "Kinesis CloudWatch Raw" so the input correctly decompresses and parses the CloudWatch Logs subscription envelope. If the source log group contains VPC Flow Logs, choose "CloudWatch Flow Logs" instead to get automatic parsing of the flow log fields.
Add the minimum required IAM permissions to the IAM role that Graylog assumes. See the AWS documentation for a full description of the required permissions, and ensure the role also grants the DynamoDB permissions listed in DynamoDB Permissions.
For automatic setup, ensure the IAM role includes all the manual setup permissions, plus the following additional permissions:
- iam:CreateRole
- iam:GetRole
- iam:PassRole
- iam:PutRolePolicy
- kinesis:CreateStream
- kinesis:DescribeStream
- kinesis:GetRecords
- kinesis:GetShardIterator
- kinesis:ListShards
- kinesis:ListStreams
- logs:DescribeLogGroups
- logs:PutSubscriptionFilter

DynamoDB Permissions

In addition to the DynamoDB actions the lease table already required (CreateTable, DescribeTable, GetItem, PutItem, Scan, UpdateItem, and DeleteItem), KCL 3.5 requires the following permissions:

Action	Resource	Why It Is Required
dynamodb:Query	arn:aws:dynamodb:<region>:<account>:table/graylog-aws-plugin-/index/	KCL 3.5 discovers leases through a global secondary index on the lease table. An index is a separate IAM resource from its table, so granting Query on the table alone still denies this call.
dynamodb:UpdateTable	arn:aws:dynamodb:<region>:<account>:table/graylog-aws-plugin-*	KCL creates the global secondary index the first time the input starts. If this action is denied, the input fails to start with a generic initialization error.
dynamodb:ConditionCheckItem	arn:aws:dynamodb:<region>:<account>:table/graylog-aws-plugin-*	Required only to migrate an existing input to single-table state tracking. The migration moves state in a DynamoDB transaction. Without this action, the migration never completes and Graylog does not report an error.

The input also needs access to two legacy tables, <application-name>-CoordinatorState and <application-name>-WorkerMetricStats. The level of access depends on when the input was created:

Inputs created in Graylog 7.2 or later need only dynamodb:DescribeTable on both tables. KCL checks whether these tables exist every time the input starts, so a policy scoped to the lease table alone prevents the input from starting, even though the input never uses these tables.
Inputs created before Graylog 7.2 continue to use both tables until you Migrate to Single DynamoDB Table for State Tracking. Until then, you'll need DescribeTable, GetItem, PutItem, UpdateItem, DeleteItem, and Scan on both tables.

A policy whose resource is arn:aws:dynamodb:<region>:<account>:table/graylog-aws-plugin-* covers the lease table and both legacy tables because their names share that prefix. The following example policy grants all the DynamoDB permissions the input needs, including the permission required for migration. Replace <region> and <account> with your values. Refer to the AWS KCL permissions docs.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "GraylogKinesisLeaseAndLegacyTables",
      "Effect": "Allow",
      "Action": [
        "dynamodb:CreateTable",
        "dynamodb:DescribeTable",
        "dynamodb:UpdateTable",
        "dynamodb:GetItem",
        "dynamodb:PutItem",
        "dynamodb:UpdateItem",
        "dynamodb:DeleteItem",
        "dynamodb:Scan",
        "dynamodb:ConditionCheckItem"
      ],
      "Resource": "arn:aws:dynamodb:<region>:<account>:table/graylog-aws-plugin-*"
    },
    {
      "Sid": "GraylogKinesisLeaseIndex",
      "Effect": "Allow",
      "Action": "dynamodb:Query",
      "Resource": "arn:aws:dynamodb:<region>:<account>:table/graylog-aws-plugin-*/index/*"
    }
  ]
}

Graylog monitors the two calls the input cannot work without: DynamoDB lease discovery (Query) and the Kinesis record fetch (GetRecords). If either call is denied continuously for two minutes, Graylog fails the input and names the denied action. A denial that KCL can work around, such as one that affects only lease rebalancing, leaves the input running.

Input Type

This input is a pull input type. See Inputs to learn about input types.

Input Configuration

Follow the input setup instructions. During setup of this input, you can configure the following options:

Configuration Option	Description
Input Name	Provide a unique name for your new input.
AWS Authentication Type	Select either to allow the system automatically look for credentials using the AWS default credential provider chain or provide AWS Access and Secret Keys.
AWS assume role (ARN)	The ARN of the Identity and Access Management (IAM) role that Graylog will assume to access the configured Kinesis stream and related CloudWatch resources. AWS recommends using IAM roles with temporary credentials instead of long-term static access keys. This option is preferred and supports cross-account access.
AWS Region	Select the AWS region where your Kinesis stream resides.
Optional AWS VPC Endpoints	These settings let you override the default AWS public API endpoints with VPC endpoint URLs, which is useful when you want traffic to AWS services to stay inside your private network rather than going over the public internet.
AWS Message Type	This determines the type of AWS message this input receives. There are three supported message types: Kinesis Raw: Select this when your Kinesis stream contains raw, uncompressed data (not routed through CloudWatch Logs). If no messages are detected in your selected stream, Graylog automatically defaults to this option. Kinesis CloudWatch Flow Log: Select this option if the source log group contains VPC Flow Logs. Kinesis CloudWatch Raw: Select this when your Kinesis stream receives general CloudWatch Logs (e.g., application logs, Lambda logs) via a subscription filter, but the data is not VPC Flow Logs.
Select Stream	Choose an existing Kinesis stream from your AWS account. Graylog will pull messages from this stream. If no suitable stream is available, you can select Setup Kinesis Automatically to create and configure one.
Advanced Options	Enable Throttling: Enables Graylog to stop reading new data for this input whenever the system falls behind on message processing and needs to catch up. Add Flow Log field name prefix: Adds a prefix flow_log_ to AWS Flow Log field names. For example, src_addr becomes flow_log_src_addr. This helps differentiate AWS flow log fields from other similarly named fields. Store Full Message: An optional field that is turned off by default. When enabled, the input stores the original Kinesis/CloudWatch log event payload in the full_message field. Override source (optional): By default, the source is a hostname derived from the received packet. You can override the default value with a custom string. This option allows you to optimize the source for your specific needs. Kinesis Record Batch Size: Sets the number of Kinesis records fetched in each polling batch. Each record can be up to 1 MB in size.
Migrate to single DynamoDB table for state tracking	A check box available when you edit an existing input. This option is only relevant for inputs created before Graylog 7.2. When selected, it starts a one-way migration that consolidates the input's DynamoDB state into its lease table and cannot be reverted once complete. Before you select this option, follow the steps in Migrate to Single DynamoDB Table for State Tracking. Inputs created in Graylog 7.2 or later use the single-table format automatically.

Migrate to Single DynamoDB Table for State Tracking

KCL stores the input's coordination state in DynamoDB. Before KCL 3.5, each input used three tables: the lease table plus separate <application-name>-CoordinatorState and <application-name>-WorkerMetricStats tables. KCL 3.5 introduces a single-table format that stores all of this state in the lease table, which reduces the number of DynamoDB tables in your account and helps you stay under AWS account-level table limits.

Inputs created in Graylog 7.2 or later use the single-table format automatically. No action is required.
Inputs created before Graylog 7.2 keep the three-table layout and continue to run without interruption, provided the IAM role grants the DynamoDB Permissions KCL 3.5 requires. These inputs stay on the three-table layout until you migrate them.

Important: AWS has not announced when the three-table layout will no longer be supported, but some of its configuration options are already deprecated in KCL 3.5. We recommend that you migrate existing inputs while running Graylog 7.2 so you are not under time pressure when support is removed in a future release.

The migration is one-way and cannot be reverted once complete. Stream checkpoints are preserved, so ingestion continues without replaying or skipping records. The migration is not instantaneous, and AWS recommends monitoring it until it is complete.

Before you begin, confirm that the IAM role grants dynamodb:ConditionCheckItem on the -CoordinatorState table. Without it, the migration never completes and Graylog does not report an error.

To migrate an existing input:

Upgrade to Graylog 7.2 and let the input run for one hour with the Migrate to single DynamoDB table for state tracking option turned off. KCL 3.5 requires the input to run steadily for a full hour before it accepts the migration. You cannot shorten this readiness period.
Confirm that the input is ready to migrate:
1. In the AWS DynamoDB console, open the input's coordinator state table, graylog-aws-plugin-<stream-name>-CoordinatorState.
2. Find the TableMigration3.5 item and check the value of its tm attribute. If the value is TABLE_MIGRATION_STATUS_DEPLOYED, continue to the next step. If the value is TABLE_MIGRATION_STATUS_INIT, the readiness period has not elapsed. Wait, and then check again.
Warning: Do not enable the migration option before the value is TABLE_MIGRATION_STATUS_DEPLOYED. If you do, the input fails to start.
In Graylog, navigate to System > Inputs.
Locate the input and open the Edit input dialog.
Select the Migrate to single DynamoDB table for state tracking check box, and then select Update input. The migration begins.
After 24 hours, verify that the migration is complete. KCL 3.5 waits 24 hours by default before it finalizes the migration.
In DynamoDB, check the TableMigration3.5 item again. Its tm attribute should be TABLE_MIGRATION_STATUS_COMPLETE.
Optionally, delete the legacy -CoordinatorState and -WorkerMetricStats tables. After the migration is complete, the input no longer uses them.

For details on monitoring the migration and removing the legacy tables, see the AWS documentation:

Next Steps

After you complete input setup, visit Input Diagnosis for testing and validation of the new input. Use this functionality to help troubleshoot any connection issues.