AWS Kinesis/CloudWatch Input

The AWS Kinesis/CloudWatch input allows Graylog to read log messages from CloudWatch via Kinesis.

Prerequisites

Before proceeding, ensure that the following prerequisites are met:

  • Kinesis is required to stream messages to Graylog before messages can be read from CloudWatch.

Message Types Supported

The following message types are supported:

  • CloudWatch Logs: Raw text strings within CloudWatch.

  • CloudWatch Flow Logs: Flow Logs within a CloudWatch log group.

  • Kinesis Raw Logs: Raw text strings written to Kinesis.

Automatic Setup Flow

Review to learn how to add the AWS Kinesis/CloudWatch input to Graylog. For this setup to function as expected, the Recommended Policy must be allowed for the authorized user (see Permission Policies below).

  1. Complete AWS Kinesis Authorize steps as follows:

    1. Add the input name, AWS Access Key, AWS Secret Key, and select AWS Region to authorize Graylog.

    2. Click the Authorize & Choose Stream button to continue.

  2. Complete AWS Kinesis Setup as follows:

    1. In the dialog box, click the Setup Kinesis Automatically button.

    2. Enter a name for the Kinesis stream and select a CloudWatch log group from the drop-down list.

    3. Select Begin Automated Setup.

    4. A Kinesis Auto Setup Agreement prompt appears. Read the agreement, and click I Agree! Create these AWS resources now.

  3. The auto-setup details and references the resources that were created. Click Continue Setup to proceed.

  4. On the AWS CloudWatch Health Check, Graylog reads a message from the Kinesis stream and checks its format. Graylog attempts to automatically parse the message if it is of a known type.

  5. For AWS Kinesis Review, review and finalize the details for the input to complete.

Manual Setup Flow

For this setup to function as expected, the Least Privilege Policy shown below must be allowed for the authorized user (see Permission Policies below).

  1. Complete AWS Kinesis Authorize steps as follows:

    1. Type in the input name, AWS Access Key, AWS Secret Key, and select AWS Region to authorize Graylog. Click the Authorize & Choose Stream button to continue.

  2. Complete AWS Kinesis Setup as follows:

    1. Select the Kinesis stream to pull logs.

    2. Click Verify Stream & Format to continue.

  3. On the AWS CloudWatch Health Check, Graylog reads a message from the Kinesis stream and checks its format. Graylog attempts to automatically parse the message if it is of a known type.

  4. For AWS Kinesis Review, review and finalize the details for the input to complete.

Permission Policies

Manual Setup Flow Permissions

This AWS page lists the manual setup AWS permissions. These are the minimum permissions needed for the input to run. This AWS page provides a full description of the permissions in detail and an example policy.

Automatic Setup Flow Permissions

The automatic setup requires the same permissions as the manual setup, as well as additional permissions, which are listed below.

iam:CreateRole

iam:GetRole

iam:PassRole

iam:PutRolePolicy

kinesis:CreateStream

kinesis:DescribeStream

kinesis:GetRecords

kinesis:GetShardIterator

kinesis:ListShards

kinesis:ListStreams

logs:DescribeLogGroups

logs:PutSubscriptionFilter