AWS VPC via Kinesis Content Pack

The following content pack is available for use with a Graylog Illuminate license and Graylog Enterprise or Graylog Security. Contact sales to learn more about obtaining Illuminate.

Amazon Kinesis (VPC Flow Log) is a real-time streaming record of network traffic flowing to and from AWS resources, delivered through Kinesis for analysis or monitoring.

Supported/Tested Versions

  • AWS VPC Flow logs via Kinesis for Version 2 default schema.

Hint: This pack is tested with Amazon Kinesis VPC Flow logs. Other Kinesis logs have not been tested but field renaming for network related fields should work.

Warning: Custom log formats are not supported.

Requirements

  • Graylog 6.2.0+

Stream Configuration

This technology pack includes 1 stream:

  • "Illuminate:AWS Kinesis Messages"

Hint: If this stream does not exist prior to the activation of this pack then it will be created and configured to route messages to this stream and the associated index set. There should not be any stream rules configured for this stream.

Index Set Configuration

This technology pack includes 1 index set definition:

  • "AWS Kinesis Logs"

Hint: If this index set is already defined, then nothing will be changed. If this index set does not exist, then it will be created with retention settings of a daily rotation and 90 days of retention. These settings can be adjusted as required after installation.

Log Collection

This pack parses logs from the following sources:

  • AWS Kinesis/CloudWatch input

AWS Kinesis/CloudWatch Input Configuration

Refer to the input documentation to set up an AWS Kinesis/CloudWatch input.

Log Format Example

These are example logs for Amazon Kinesis VPC Flow Logs for the message field.

VPC Flow Logs

# Kinesis VPC Flow logs eni-094fad06ebf11ad6b ACCEPT TCP 10.0.2.155:50602 -> 35.92.124.220:443 eni-094fad06ebf11ad6b REJECT TCP 79.124.40.134:49683 -> 10.0.2.155:2838 eni-0e2564b556d45f08d - IP -:0 -> -:0

What Is Provided

  • Rules to parse, normalize, and enrich Amazon Kinesis VPC Flow messages.

Events Processed by This Technology Pack

The content pack supports the following log types. Generic processing will be provided for log types not listed.

  • Amazon Kinesis Flow Logs

GIM Categorization

GIM categorization is provided for the following messages:

Vendor Stream Name gim_event_type_code gim_event_category gim_event_subcategory gim_event_type
grenade-flow-logs 120000 network network.network connection network connection

Message Fields Included in This Pack

General Parsing for Kinesis Logs

Field Name Example Value Field Type Description
vendor_event_action ACCEPT string The network flow action taken by AWS VPC Flow Logs
vendor_message_type DATA_MESSAGE string The AWS Kinesis message type
vendor_stream_name grenade-flow-logs string The AWS Kinesis stream name
vendor_stream_arn arn:aws:kinesis:us-west-2:875222236744:stream/grenade-flow-logs string The ARN of the Kinesis data stream
vendor_log_group graylog-grenade string The associated AWS CloudWatch log group name
vendor_log_stream eni-00468c992fdba48c1-all string The CloudWatch log stream name
vendor_aws_owner 875225226744 string The AWS account owner ID
vendor_aws_source true boolean Indicates if the log source is AWS
vendor_subscription_filters [grenade-all2] string AWS CloudWatch subscription filter name(s)
network_bytes 76 integer Number of bytes transferred in the network flow
event_duration_seconds 1 integer Duration of the capture window in seconds
destination_ip 185.125.190.56 ip Destination IP address of the network flow
destination_port 123 integer Destination port number
network_interface_id eni-00468c992fd2248c1 string The AWS Elastic Network Interface ID
event_status OK string The flow log status (e.g."
vendor_protocol UDP string Protocol name used in the network flow
network_iana_number 17 integer IANA-assigned protocol number
source_ip 10.0.2.199 ip Source IP address of the network flow
source_port 37284 integer Source port number

Amazon Kinesis VPC Flow Content Pack

This spotlight offers a dashboard with 1 tab:

Overview