AWS CloudTrail Input

The AWS CloudTrail input allows Graylog to read log messages from the AWS CloudTrail service. CloudTrail logs are generated by AWS whenever any action takes place within your account. These logs are useful for tracking user activity, API usage, and changes to your AWS resources.

Prerequisites

A valid AWS account with Amazon CloudTrail enabled.

Required AWS Setup

• Create a trail with AWS CloudTrail.

• Set up SQS for CloudTrail Write Notifications.

Create a Trail with AWS CloudTrail

1. Start by configuring trail attributes.
   

2. Select the following options:
   

   • Trail name: Provide a unique name.

   • Enable for all accounts in my organization: Select this check box to enable/disable the trail for all accounts in your organization.

   • Storage location: Create a new S3 bucket or use an existing S3 bucket. Message contents are stored in the bucket.

   • Trail log bucket name: Enter a unique S3 bucket name. This location is where CloudTrail writes the payload of each message. Graylog reads the message content from here when it receives the SNS message from the queue.

   Additional settings:

   • Log file SSE-KMS encryption: This option is enabled by default. The AWS KMS documentation provides more details.

   • Log file validation: Enable this option to have log digests delivered to your Amazon S3 bucket.

   • SNS notification delivery: Enable.

   • Create a new SNS topic: Specify a name for the topic (e.g. "cloudtrail-log-write") or choose from existing topics. This name is needed to configure the Graylog input.

   Enabling Cloudwatch Logs and adding Tags are optional.

3. Select what types of events you want to log, for example, management events, data events, or insight events.

4. Review and complete the set up.

Set up SQS for CloudTrail Write Notifications

1. Navigate to Amazon SQS and create a queue. All settings can be left at their default values initially.
   

2. Specify a queue name (e.g. "cloudtrail-notifications"). This name is needed to configure the Graylog input. Our recommended default value is cloudtrail-notifications. CloudTrail writes notifications with S3 file name references to this queue.

3. Subscribe the SQS queue to your CloudTrail SNS topic.
   
   

HTTPS Communication

This input uses the AWS SDK to communicate with various AWS resources. Therefore, HTTPS communication must be allowed between the Graylog server and the resources. If communication on the network segment containing the Graylog cluster is restricted, ensure that communication to the following endpoints are explicitly permitted.

monitoring.<region>.amazonaws.com
cloudtrail.<region>.amazonaws.com
sqs.<region>.amazonaws.com
sqs-fips.<region>.amazonaws.com
<bucket-name>.s3-<region>.amazonaws.com 

Configure the Input in Graylog

After launching your new input, configure the following fields based on your preferences: 

• Global

  • Click the Global check box to enable this input on all Graylog nodes, or keep it unchecked to enable the input on a specific node.

• Title

  • Provide a unique name for your input.

• AWS SQS Region

  • Select the AWS region the queue is in.

• AWS S3 Region

  • Select the AWS region where the S3 bucket storing CloudTrail logs resides.

• SQS Queue Name

  • Provide the SQS queue name you created, where SNS is writing CloudTrail notifications to.

• Enable Throttling

  • Enables Graylog to stop reading new data for this input whenever the system falls behind on message processing and needs to catch up.

• AWS access key (optional)

  • The unique identifier created for the AWS Identity and Access Management (IAM) user. The Credential settings retrieval order documentation provides more information.

• AWS secret key (optional)

  • The access key ID for the IAM user with permission to the subscriber and the SQS queue.

• AWS assume role (ARN) (optional)

  • This setting is often used for cross-account access.

• Override source (optional)

  • By default, the source is a hostname derived from the received packet. You can override the default value with a custom string. This option allows you to optimize the source for your specific needs.

• Encoding (optional)

  • All messages need to support the encoding configured for the input. For example, UTF-8 encoded messages should not be sent to an input configured to support UTF-16.

CloudTrail Troubleshooting

If the CloudTrail input is starting and the debug log messages show that messages are being received but no messages are visible when searching in Graylog, then make sure the SQS subscription is not set to deliver the messages in raw format.