AWS S3 Content Pack

The following content pack is available for use with a Graylog Illuminate license and Graylog Enterprise or Graylog Security. Contact sales to learn more about obtaining Illuminate.

Amazon S3 is an object storage service. This content pack processes two S3 log families: AWS CloudTrail S3 data events (event_source=s3.amazonaws.com) and S3 server access logs.

Messages are normalized into the Graylog Information Model, or GIM (file.create, file.delete, file.access, audit.policy, audit.default) with Security Core coverage. A lightweight single-event detection flags CloudTrail cross-bucket CopyObject operations as potential data exfiltration (301001 host_dlp_detection).

Supported Version(s)

  • AWS CloudTrail S3 data events (CloudTrail record version 1.08 and 1.09)

  • AWS S3 server access log format (current version as of April 2026)

Requirements

  • Graylog 7.1+ with a valid Enterprise license

Stream Configuration

This technology pack includes 1 stream:

  • Illuminate:AWS S3 Messages

Hint: If this stream does not exist prior to the activation of this pack then it is created and configured to route messages to this stream and the associated index set. There should not be any stream rules configured for this stream.

Index Set Configuration

This technology pack includes 1 index set definition:

  • AWS S3 Logs

Hint: If this index set is already defined, then nothing is changed. If this index set does not exist, then it is created with retention settings of a daily rotation and 90 days of retention. These settings can be adjusted as required after installation.

Log Collection and Delivery

This pack parses logs from the following sources:

  • AWS CloudTrail S3 data events forwarded via the AWS CloudTrail input, the AWS Kinesis/CloudWatch input, or an S3-to-Graylog forwarder.

  • AWS S3 server access logs parsed by a forwarder (Lambda or similar) that reads the log bucket and forwards GELF/JSON messages.

CloudTrail Delivery

  1. Enable data events for the S3 buckets of interest in a CloudTrail trail. See the AWS documentation on logging data events for details.

  2. Send the trail to an Amazon SNS topic with an SQS subscription.

  3. In Graylog, create an AWS CloudTrail input that reads from that SQS queue.

If you decode the trail with your own forwarder instead, note the following:

  • You must send one CloudTrail record per message.

  • Map the raw JSON object to the message field.

S3 Server Access Log Delivery

  1. Enable server access logging on each bucket you want to monitor, routing the logs to a dedicated logging bucket.

  2. Create an AWS Lambda function that triggers on S3 ObjectCreated events on the logging bucket.

  3. Configure the function to parse each log file using the space-delimited S3 access log format.

  4. Forward each parsed record as a separate message to a Graylog GELF input.

Log Format Examples

These are example logs for the two supported AWS S3 log families.

CloudTrail S3 Data Event

# AWS CloudTrail S3 Data Event (decoded record, one per message) { "eventVersion": "1.09", "userIdentity": { "type": "IAMUser", "principalId": "AIDAEXAMPLEID1234", "arn": "arn:aws:iam::111122223333:user/alice", "accountId": "111122223333", "userName": "alice" }, "eventTime": "2026-04-15T10:15:22Z", "eventSource": "s3.amazonaws.com", "eventName": "PutObject", "awsRegion": "us-east-1", "sourceIPAddress": "203.0.113.45", "userAgent": "aws-cli/2.13.0 Python/3.11", "requestParameters": { "bucketName": "example-app-uploads", "key": "inbox/2026/04/15/report.csv" }, "responseElements": null, "eventID": "3fcd7213-a8dd-c88a-d151-58665299f60b", "readOnly": false, "resources": [ { "type": "AWS::S3::Object", "ARN": "arn:aws:s3:::example-app-uploads/inbox/2026/04/15/report.csv" }, { "accountId": "111122223333", "type": "AWS::S3::Bucket", "ARN": "arn:aws:s3:::example-app-uploads" } ], "eventType": "AwsApiCall", "managementEvent": false, "recipientAccountId": "111122223333", "eventCategory": "Data" }

S3 Server Access Log (raw AWS format)

# AWS S3 Server Access Log (space-delimited, one line per request) # Field order: bucket_owner bucket [time] remote_ip requester request_id operation key "request_uri" http_status error_code bytes_sent object_size total_time turn_around_time "referer" "user_agent" version_id host_id signature_version cipher_suite authentication_type host_header tls_version access_point_arn acl_required a1b2c3d4e5f67890canonicaluserid1a2b3c4d5e6f7890a1b2c3d4e5f67890 example-app-uploads [15/Apr/2026:10:15:22 +0000] 203.0.113.10 arn:aws:iam::111122223333:user/alice A1B2C3D4E5F60718 REST.PUT.OBJECT inbox/2026/04/15/upload.png "PUT /inbox/2026/04/15/upload.png HTTP/1.1" 200 - 0 524288 145 12 "-" "aws-cli/2.13.0" - abcdef0123456789 SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader example-app-uploads.s3.us-east-1.amazonaws.com TLSv1.3 - - a1b2c3d4e5f67890canonicaluserid1a2b3c4d5e6f7890a1b2c3d4e5f67890 example-static-assets [15/Apr/2026:10:16:02 +0000] 198.51.100.12 - B2C3D4E5F6071819 REST.GET.OBJECT images/logo.png "GET /images/logo.png HTTP/1.1" 200 - 18211 18211 8 2 "-" "Mozilla/5.0" - hostid0123456789 - - - example-static-assets.s3.us-east-1.amazonaws.com TLSv1.3 - - a1b2c3d4e5f67890canonicaluserid1a2b3c4d5e6f7890a1b2c3d4e5f67890 example-data-lake [15/Apr/2026:10:20:45 +0000] 203.0.113.66 arn:aws:iam::444455556666:user/external-user C3D4E5F607181920 REST.GET.OBJECT sensitive/finance.xlsx "GET /sensitive/finance.xlsx HTTP/1.1" 403 AccessDenied 243 2048000 9 - "-" "curl/8.0.1" - hostid9876543210 SigV4 - - example-data-lake.s3.us-east-1.amazonaws.com TLSv1.3 - -

S3 Server Access Log (forwarded shape the pack expects)

# S3 Server Access Log -- forwarded shape (what the pack expects as input) # # The forwarder reads the raw space-delimited record and emits one # JSON/GELF message per line with the fields below. Keep the AWS # field names on the way in; the pack's 10-aws_s3_access_log_processing # rule performs the rename to Graylog schema fields. # # Minimal identification triple: bucket_owner + bucket + operation. # No `source` label required; source=s3-access-log-forwarder is also accepted. { "short_message": "203.0.113.10 REST.PUT.OBJECT example-app-uploads/inbox/2026/04/15/upload.png", "bucket_owner": "a1b2c3d4e5f67890canonicaluserid1a2b3c4d5e6f7890a1b2c3d4e5f67890", "bucket": "example-app-uploads", "timestamp": "2026-04-15T10:15:22Z", "remote_ip": "203.0.113.10", "requester": "arn:aws:iam::111122223333:user/alice", "request_id": "A1B2C3D4E5F60718", "operation": "REST.PUT.OBJECT", "key": "inbox/2026/04/15/upload.png", "request_uri": "PUT /inbox/2026/04/15/upload.png HTTP/1.1", "http_status": 200, "error_code": "-", "bytes_sent": 0, "object_size": 524288, "total_time": 145, "turn_around_time": 12, "referer": "-", "user_agent": "aws-cli/2.13.0", "version_id": "-", "host_id": "abcdef0123456789", "signature_version": "SigV4", "cipher_suite": "ECDHE-RSA-AES128-GCM-SHA256", "authentication_type": "AuthHeader", "host_header": "example-app-uploads.s3.us-east-1.amazonaws.com", "tls_version": "TLSv1.3", "access_point_arn": "-", "acl_required": "-" }

What is Provided

  • Rules to parse, normalize, and enrich AWS S3 CloudTrail data events and S3 server access logs.

  • GIM categorization for S3 object create, access, delete, audit policy change, audit default, and cross-bucket data exfiltration detection.

  • Security Core coverage for file and audit categories.

Events Processed by This Technology Pack

The content pack supports the following log types.

  • AWS CloudTrail S3 data events (PutObject, GetObject, HeadObject, CopyObject, DeleteObject, DeleteObjects, ListObjects, REST.PUT.BUCKETPOLICY, and similar)

  • S3 server access log operations (REST.PUT.OBJECT, REST.GET.OBJECT, REST.DELETE.OBJECT, REST.POST.MULTI_OBJECT_DELETE, REST.GET.BUCKET, REST.PUT.BUCKETPOLICY, and similar)

GIM Categorization

GIM categorization is provided for the following messages:

Event gim_event_type_code GIM Category GIM Subcategory GIM Event Type
PutObject / REST.PUT.OBJECT 200000 file file.create file created
CopyObject 200000 file file.create file created
GetObject / REST.GET.OBJECT 201500 file file.access file accessed
HeadObject / REST.HEAD.OBJECT 201500 file file.access file accessed
DeleteObject / REST.DELETE.OBJECT 200100 file file.delete file deleted
DeleteObjects / REST.POST.MULTI_OBJECT_DELETE / BATCH.DELETE.OBJECT 200100 file file.delete file deleted
ListObjects / ListBuckets / HeadBucket / GetBucketPolicy / GetBucketAcl / GetBucketLocation / REST.GET.BUCKET / REST.HEAD.BUCKET / REST.GET.* 229999 audit audit.default audit event
REST.PUT.BUCKETPOLICY 220500 audit audit.policy audit policy changed
CopyObject (detection: cross-bucket exfil) 301001 detection detection.host_detection host_dlp_detection

Message Fields Included in This Pack

General Parsing for AWS S3

Field Name Example Value Field Type Description
event_source_product aws_s3 string Identifies the log source product as AWS S3
event_created 2026-04-15T10:15:22.000Z date Event time parsed from the CloudTrail eventTime or the S3 access log request time, so search and dashboards use event time rather than ingest time
event_log_name AWS S3 CloudTrail Data Event string Static log-source name per transport: AWS S3 CloudTrail Data Event or AWS S3 Server Access Log
vendor_aws_event_source_api s3.amazonaws.com string AWS service endpoint that emitted the event (vendor passthrough; no canonical SDM field exists)
vendor_aws_region us-east-1 string AWS region the call was made in (CloudTrail awsRegion; vendor passthrough, no canonical SDM field exists)
vendor_event_module PutObject string Raw operation name from CloudTrail event_name or S3 access log operation; keyed by the GIM lookup
vendor_event_action PutObject string Same as vendor_event_module; keyed by the event_action lookup
vendor_event_description PutObject string Human-readable summary; mirrors vendor_event_action. Satisfies audit.* GIM enforcement.
event_action created string Normalized action (created, read, deleted, modified) derived via the aws_s3_event_action_lookup
event_outcome success string success by default; failure when CloudTrail reports error_code or the access log http_response_code >= 400
source_user_name alice string CloudTrail user identity userName (for CloudTrail events)
source_user_id AIDAEXAMPLEID1234 string CloudTrail user identity principalId, or S3 access log requester (ARN/canonical ID)
source_user_domain 111122223333 string AWS account ID of the caller (from CloudTrail user_identity_accountId)
vendor_user_arn arn:aws:iam::111122223333:user/alice string Caller ARN (from CloudTrail user_identity_arn). AWS-specific; no canonical User schema field exists for ARNs.
vendor_user_type IAMUser string CloudTrail user_identity_type (IAMUser, AssumedRole, Root, AWSService, etc.)
user_name alice string Mirror of source_user_name so file.* GIM enforcement (which requires user_name) is satisfied
user_id AIDAEXAMPLEID1234 string Mirror of source_user_id so downstream user-based queries resolve either field
source_ip 203.0.113.45 ip Caller IP address (from CloudTrail source_ip_address or S3 access log remote_ip)
vendor_bucket_name example-app-uploads string S3 bucket name involved in the operation
file_name inbox/2026/04/15/report.csv string S3 object key
file_path s3://example-app-uploads/inbox/2026/04/15/report.csv string s3:// URI built from bucket + object key
host_hostname example-app-uploads.s3.amazonaws.com string Synthesized from the bucket name so the core reference rule derives host_reference for file.* enforcement
http_response_code 200 integer HTTP status from the S3 access log (not present on CloudTrail messages)
http_response_bytes 0 long Access log: number of HTTP response bytes sent (S3 bytes_sent)
file_size 524288 long Access log: total size of the S3 object (S3 object_size)
vendor_total_time_ms 145 long Access log: total request duration in milliseconds
vendor_turn_around_time_ms 12 long Access log: time S3 spent processing the request in milliseconds
vendor_request_id A1B2C3D4E5F60718 string Access log: S3 request ID
http_request_method PUT string Access log: HTTP method parsed from the request line
http_request_path /inbox/2026/04/15/upload.png string Access log: HTTP request path parsed from the request line
http_version 1.1 keyword Access log: HTTP version parsed from the request line (bare version, e.g. 1.1)
vendor_request_line PUT /inbox/2026/04/15/upload.png HTTP/1.1 string Access log: raw HTTP request line (method + path + protocol), retained for forensic completeness
vendor_authentication_type AuthHeader string Access log: S3 auth method (AuthHeader, QueryString, -)
vendor_signature_version SigV4 string Access log: AWS signature version
vendor_cipher_suite ECDHE-RSA-AES128-GCM-SHA256 string Access log: TLS cipher suite
vendor_tls_version TLSv1.3 string Access log: TLS protocol version
vendor_host_id abcdef0123456789 string Access log: S3 host_id
destination_hostname example-app-uploads.s3.us-east-1.amazonaws.com string Access log: HTTP Host header (regional S3 endpoint FQDN). Includes region, unlike the regionless host_hostname synthesized for host_reference.
http_user_agent aws-cli/2.13.0 string User-Agent header from the requester (CloudTrail user_agent / access log user_agent)
event_severity informational string Informational baseline event severity set by the processing rules (routine S3 events are not alerts)
event_severity_level 1 integer Numeric baseline event severity (1=informational); mutual pair of event_severity
alert_severity_level 4 integer Set only by the detection rule on a cross-bucket CopyObject (4=high)
alert_severity high string Derived by core from alert_severity_level on detection events
alert_category Data Exfiltration string Set by the detection rule on cross-bucket CopyObject events
alert_signature AWS S3 Cross-Bucket Object Copy string Set by the detection rule
alert_signature_id aws_s3_cross_bucket_copy string Set by the detection rule
gim_event_type_code 200000 string GIM type code array; file. or audit. depending on the operation; detection adds 301001