AWS S3 Input

The AWS S3 input supports collecting published log files into an Amazon S3 bucket. As new log files are published, the input ingests them automatically. The input supports both newline-delimited (one message per line) and JSON root array messages (multiple log messages contained in a single JSON array). The input utilizes Simple Queue Service (SQS) S3 bucket notifications in order to identify when new data is available for reading by Graylog.

Prerequisites

• An Amazon Web Services (AWS) subscription.

• A defined S3 bucket to which logs may be written.

IAM Permissions

In order for Graylog to connect to AWS S3, an Identity and Access Management (IAM) role must be created and utilized with sufficient permissions to read the target SQS queue, and also to read the files from S3. The following Amazon permissions are required for the input to operate:

• s3: GetObject

• sqs: ReceiveMessage

Required S3 and SQS Setup

In addition to the following steps, these articles are helpful to complete the setup:

• Enabling and configuring event notifications using the Amazon S3 console

• Walkthrough: Configuring a bucket for notifications (SNS topic or SQS queue)

SQS Setup

Create an SQS queue that Graylog can subscribe to in order to receive notifications of new files to read. Most default options can be accepted. Note that an access policy must be defined to allow the S3 bucket to publish notifications to the queue. The following is an sample policy for authorizing S3 to publish the notifications. Note that the S3 bucket creation is defined below.

{
  "Version": "2012-10-17",
  "Id": "example-ID",
  "Statement": [
    {
      "Sid": "s3-publish-policy",
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": "SQS:SendMessage",
      "Resource": "arn:aws:sqs:<region>:<account-number>:<queue-name>",
      "Condition": {
        "StringEquals": {
          "aws:SourceAccount": "<account-number>"
        },
        "ArnLike": {
          "aws:SourceArn": "arn:aws:s3:::<s3-bucket-name>"
        }
      }
    }
  ]
}

Set Up an S3 Bucket

You must have an S3 bucket to write log message files to. If not, create one following the official Amazon documentation. After you create the bucket, you must enable the Event Notifications option for it. This article provides instructions to perform the setup.

When you configure the notifications, in the "Event Types" section, choose the "All object create events" option to ensure the input is notified of all possible methods by which files are created.

In the Destination section, choose the SQS queue that was created above.

Input Configuration

The following options are available when launching a new input from the Graylog Inputs tab:

"field 1", "field 2"

"same line field", "field with 
line breaks"

["log message 1", "log message 2", ...]

[{"key", "value"}, {"key2", "value2"}, ...]