Google Cloud BigQuery Output

This is a Graylog Enterprise feature. A valid Graylog Enterprise license is required.

Google Cloud BigQuery Output allows you to send data to your Google Cloud BigQuery tables. Each message in the stream is inserted as a new row in the configured BigQuery table.

Unlike the Raw TCP and TCP Syslog Outputs, which require a payload formatter to work, the BigQuery Output does not rely on a payload formatter. Because the Outbound Payload Format is required when setting up any Enterprise Framework Output, we have provided a No-op Formatter specifically for use with the BigQuery Output.

The BigQuery Output uses the key-value pairs in each Graylog message to build a row to be inserted into the BigQuery table with the Graylog message keys mapping to the BigQuery table’s columns. Any Graylog message key that does not have a corresponding column in the BigQuery table will be dropped by Google when the insert is performed. You can use a processing pipeline or the Excluded Fields list in the BigQuery Output configuration to prevent the inclusion of unwanted fields when each row is sent to the BigQuery table.

Required Google Cloud Setup

Prerequisites

To use the Google Cloud BigQuery Output, a working Google Cloud account and access to the Google Cloud console are required. See the steps to create an account below.

Create Service Account

  1. Log into the Google Cloud console.
  2. Navigate to IAM & Admin from the left-hand menu.
  3. Select Service Accounts from the left-hand menu.
  4. Select + CREATE SERVICE ACCOUNT at the top of the right-hand pane.
  5. Create a new service account.
    1. Provide a name for the service account (i.e. “Graylog Data”).
    2. Enter a description for the service account.
    3. Click the CREATE button.
    4. Select appropriate permissions for the new service account. At a minimum, the service account needs the ability to write to your BigQuery table.
    5. Click the ‘’CONTINUE’’ button.
    6. If desired, grant other users access to the service account.
    7. Click the ‘’DONE’’ button to finish service account creation.

Generate and Download Service Account Credentials

  1. Click on the newly created service account in the list of service accounts.
  2. In the Keys section, select Create new key from the ADD KEY drop-down menu.
  3. Select JSON as the key type.
  4. Click on the CREATE button.
  5. Save the generated JSON file.
  6. Copy the downloaded JSON credentials file to Graylog host(s). Store the credentials file in the same location on each host, and take appropriate steps to protect the credentials file (e.g., assigning ownership of the file to the account that runs your Graylog server and setting file permissions to 400).

Output Configuration

The Google Cloud BigQuery Output supports all of the standard Enterprise Output Framework configuration options.

BigQuery Configuration

  • Project ID
    • Google Cloud Project ID.
  • Dataset
    • Output BigQuery Dataset.
  • Table
    • Output BigQuery Table.
  • Excluded Fields
    • A comma-separated list of fields is filtered out when data is sent to BigQuery.
  • Credentials File Location
    • Path to the Service Account credentials file located on Graylog Node(s).