After defining the events that are needed to trigger an alert, it is possible to attach a notification. By attaching a notification to an event or group of events, we can determine how and when information will flow out of Graylog.

In this section we explain how to create a new notification and configure supported notification types. However, it is important to note that notifications are meant to be extensible through plugins. You can find more in the Graylog Marketplace or even create your own.

Prerequisites

The following prerequisites are required before utilizing notifications with Graylog:

• Notifications are triggered by defined events, which must be created first. Please review the Graylog documentation on creating event definitions as needed.

• An enterprise license for Graylog Operations is required for script and PagerDuty notifications.

Notification Types

Graylog supports notifications through the following methods:

• PagerDuty (Operations Only)

• Slack

• Microsoft Teams

• Script (Operations Only)

• Email

• HTTP

• Legacy Alarm Callbacks (depreciated)

Creating a New Notification

Notifications can be created by selecting Notifications under the Alerts tab or by defining them in the event workflow.

Creating a Notification from the Alerts Menu

1. Navigate to the Alerts tab and select Notifications.

2. Select the Create notification button at the top right of the menu.

3. Complete the following fields: 

   1. Title: Create a unique title for your notification.

   2. Description (optional): You may add additional details about your notification in this field if desired.

   3. Notification Type: Select the notification type from the drop-down menu.

4. Once you have selected your notification type, you will be presented with additional fields based on the type selected. These fields will be detailed in the follow sections for each specific notification type.

5. You may also choose to test your notification at this time by selecting Execute Test Notification.

6. Complete your notification by clicking Create notification.

Creating a Notification in the Event Workflow

You may also choose to create a notification while you are in the process of defining a new event.

1. In the New Event Definition menu, there will be a selection for Notifications in the menu bar.

2. Under Add Notification, select Create New Notification from the drop-down.

3. From the menu that populates, complete the following fields: 

   1. Title: Create a unique title for your notification.

   2. Description (optional): You may add additional details about your notification in this field if desired.

   3. Notification Type: Select the notification type from the drop-down menu.

4. Once you have selected your notification type, you will be presented with additional fields based on the type selected. These fields will be detailed in the follow sections for each specific notification type.

5. You may also choose to test your notification at this time by selecting Execute Test Notification.

6. Complete your notification by clicking Create notification.

Data Available to Notifications

When creating notifications you can utilize data from the event definition, the event itself, and the event's backlog messages (if it is configured to retain a backlog). This data can be used when formatting email, Slack, and Microsoft Teams notifications or when providing arguments to a script notification. The fields that are available from each entity are detailed in the section below.

Event Definition Metadata (information about the event definition that created the alert):

• event_definition_id (String): The database ID of the event definition.
• event_definition_type (String): The internal name of the event definition type (aggregation-v1 or correlation-v1).
• event_definition_title (String): The title set in the UI.
• event_definition_description (String): The description set in the UI.
• job_definition_id (String): The internal job definition ID associated with a scheduled event definition.
• job_trigger_id (String): The internal ID associated with the current execution of the job.

Event Data

• event The event as it is stored in Graylog
  • id (String): The message ID of the stored event.
  • event_definition_id (String): Same as event_definition_id in the metadata section.
  • event_definition_type (String): Same asevent_definition_type in the metadata section.
  • origin_context (String): URN of the message or event creating this event (eitherevent or message). Can be empty.
  • timestamp (DateTime): The timestamp this event is describing can be set to the underlying event or message (see origin_context).
  • timestamp_processing (DateTime): The timestamp for when the event has been created by Graylog.
  • timerange_start (DateTime): The start of the window of data Graylog used to create this event. Can be empty.
  • timerange_end (DateTime): The end of the window of data Graylog used to create this event. Can be empty.
  • streams (Strings): The list of stream IDs the event is stored in.
  • source_streams (Strings): The list of stream IDs the event pulled data from.
  • alert (bool): Whether this event is considered to be an alert. Alwaystrue for event definitions that have notifications.
  • message (String): A human-friendly message describing this event.
  • source (String): The host name of the Graylog server that created this event.
  • key_tuple (Strings): The list of values making up the event’s key.
  • key (String): The event’s key as a single string.
  • priority (long): The event’s priority value.
  • fields (Map<String, String>): The custom fields attached to the event.

Backlog

• backlog (List of Message summaries): The list of messages or events which lead to this alert being generated
  • id (String): The message ID.
  • index (String): The name of the index the message is stored in. Use together withid to uniquely identify a message in Graylog.
  • source (String): The source field of the message.
  • message (String): The message field of the message.
  • timestamp (DateTime): Thetimestamp field of the message.
  • stream_ids (Strings): The stream IDs of the message.
  • fields (Map<String, Object>): The remaining fields of the message, can be iterated over.

PagerDuty Notification (Operations Only)

The PagerDuty alert notification allows you to create new incidents in PagerDuty in response to Events in your Graylog server.

These are the supported configuration options to be defined in the New Notification menu for PagerDuty:

• Routing Key
  Your PagerDuty integration routing key.

• Use Custom Incident Key
  If enabled, an incident key will be generated using the provided Incident Key Prefix. This will prevent PagerDuty from creating multiple incidents for a single event. If not selected, an incident key will NOT be generated, and each event notification will create a new incident in PagerDuty.

• Incident Key Prefix
  If a Custom Incident Key is enabled, this will be used as a prefix for the incident key.

• Client Name
  The name of the Graylog system that triggered the PagerDuty incident.

• Client URL
  The URL for your Graylog web interface. If provided, this will be used to construct links that will be embedded in your PagerDuty incident.

Slack Notification

The Slack alert notification allows you to send notifications to your Slack workspace in response to events in your Graylog server.

These are the supported configuration options to be defined in the New Notification menu for Slack:

• Configuration Color
  Highlight the custom message with your chosen color.

• Webhook URL
  The unique URL used to send messages to your Slack instance.

• Channel
  The channel to which you will send a message or you can select @user or @team to receive the notification.

• Custom Message
  The message that will be sent to Slack. The data described above can be used in this template.

• Time Zone for Date/Time Values

  Select your preferred time zone used for timestamps in the notification. This selection will default to UTC.

• Message Backlog Limit (optional)
  Limit the number of backlog messages sent as part of the Slack notification. If set to 0, no limit will be enforced.

• User Name (optional)
  User name of the sender in Slack.

• Selections (you may choose to enable the following options):

  • Include Title: If enabled, this will include the full event definition title and description in the notification.

  • Notify Channel: If enabled, this will notify all the users in a channel with @channel.

  • Notify Here: If enabled, this will notify only active users in the channel with @here.

  • Link Names: If enabled, this will find and link channel names and user names in the notification.

• Icon URL (optional)
  Image to use as the icon for this message.

• Icon Emoji (optional) 
  Emoji to use as the icon for this message (this overrides any icon URL).

Microsoft Teams Notification

The Microsoft Teams notification allows you to send messages to a Teams channel when specific events occur in your Graylog setup. This is done via a webhook, which is the same method used for Slack notifications.

These are the supported configuration options to be defined in the New Notification menu for Microsoft Teams:

• Configuration Color
  Highlight the custom message with your preferred color.

• Webhook URL
  This is the unique URL generated whilst setting up your webhook. Copy the URL that was generated when creating your webhook and paste it into this field.

• Custom Message
  This is the message that will be sent to your MS Teams channel.

• Time Zone for Date/Time Values

  Select your preferred time zone used for timestamps in the notification. This selection will default to UTC.

• Message Backlog Limit (optional)
  Limit the number of backlog messages sent as part of the MS Teams notification. If set to 0, no limit will be enforced.

• Icon URL
  This is an image to use as the icon for this message.

Script Alert Notification [Operations Only]

The script notification option lets you configure a script that will be executed when the alert is triggered.

These are the supported configuration options to be defined in the New Notification menu for script notifications:

• Script Path
  The path to where the script is located. Must be within the permitted script path (which is customizable).

• Script Timeout
  The maximum time (in milliseconds) the script will be allowed to execute before being forcefully terminated.

• Script Arguments
  Space-delimited string of parameters. Any of the data described above can be used.

• Send Alert Data Through STDIN
  Sends the JSON encoded data described above through standard in. You can use a JSON parser in your script.

Script Alert Notification success is determined by its exit value; success equals zero. Any non-zero exit value will cause it to fail. Returning any error text through STDERR will also cause the alarm callback to fail.

Here is a sample Python script that shows all supported Script Alert Notification functionality (argument parsing, STDIN JSON parsing, STDOUT, exit values, and returning an exit value).

#!/usr/bin/env python3
import json
import sys

# Function that prints text to standard error
def print_stderr(*args, **kwargs):
    print(*args, file=sys.stderr, **kwargs)

# Main function
if __name__ == "__main__":

    # Print out all input arguments.
    sys.stdout.write("All Arguments Passed In: " + ' '.join(sys.argv[1:]) + "\n")

    # Turn stdin.readlines() array into a string
    std_in_string = ''.join(sys.stdin.readlines())

    # Load JSON
    event_data = json.loads(std_in_string)

    # Extract some values from the JSON.
    sys.stdout.write("Values from JSON: \n")
    sys.stdout.write("Event Definition ID: " + event_data["event_definition_id"] + "\n")
    sys.stdout.write("Event Definition Title: " + event_data["event_definition_title"] + "\n")
    sys.stdout.write("Event Timestamp: " + event_data["event"]["timestamp"] + "\n")

    # Extract Message Backlog field from JSON.
    sys.stdout.write("\nBacklog:\n")
    for message in event_data["backlog"]:
        for field in message.keys():
            sys.stdout.write("Field: " + field + "\t")
            sys.stdout.write("Value: " + str(message[field]) + "\n")

    # Write to stderr if desired
    # print_stderr("Test return through standard error")

    # Return an exit value. Zero is success, non-zero indicates failure.
    exit(0) 

Email Alert Notification

The email alert notification can be used to send an email to the configured alert receivers when the conditions are triggered. Make sure to check the email-related configuration settings in the Graylog configuration file.

Several configuration options are available for the alert notification to customize the email that will be sent. Note that the email body and email subject are JMTE templates. JMTE is a minimal template engine that supports variables, loops and conditions. See the JMTE documentation for a language reference.
The default body template shows some advanced examples of accessing the available information in the template:

--- [Event Definition] ---------------------------
Title:       ${event_definition_title}Description: ${event_definition_description}
Type:        ${event_definition_type}
--- [Event] --------------------------------------
Timestamp:            ${event.timestamp}
Message:              ${event.message}
Source:               ${event.source}
Key:                  ${event.key}
Priority:             ${event.priority}
Alert:                ${event.alert}
Timestamp Processing: ${event.timestamp}
Timerange Start:      ${event.timerange_start}
Timerange End:        ${event.timerange_end}
Fields:
${foreach event.fields field}${field.key}: ${field.value}
${end}
${if backlog}
--- [Backlog] ------------------------------------
Last messages accounting for this alert:
${foreach backlog message}
${message}
${end}
${end}

These are the supported configuration options to be defined in the New Notification menu for email notifications:

• Subject

  The subject used for the email notification. As noted above, the email subject is a JMTE template.

• Reply-To

  The email address recipients should reply to if necessary.

• Sender

  The email address that will be used as the sending address. If the field is left empty, the default address will be used.

• User Recipient(s) (optional)

  Select all Graylog users that will receive the notification.

• Email Recipient(s) (optional)

  Add any additional email addresses to receive the notification.

• Time Zone for Date/Time Values (optional)

  You can opt to use a specific time zone for the body of the email.

• Body Template

  This is the template used to generate the email body. As noted above, the email body is a JMTE template.

• HTML Body Template

  This is the template used to generate the email HTML body.

HTTP Alert Notification

The HTTP alert notification allows you to configure an endpoint that will be called when the alert is triggered.

Graylog will send a POST request to the notification URL including information about the alert. The body of the request is the JSON encoded data described above.

Here is an example of the payload included in a notification:

{
  "event_definition_id": "this-is-a-test-notification",
  "event_definition_type": "test-dummy-v1",
  "event_definition_title": "Event Definition Test Title",
  "event_definition_description": "Event Definition Test Description",
  "job_definition_id": "<unknown>",
  "job_trigger_id": "<unknown>",
  "event": {
    "id": "NotificationTestId",
    "event_definition_type": "notification-test-v1",
    "event_definition_id": "EventDefinitionTestId",
    "origin_context": "urn:graylog:message:es:testIndex_42:b5e53442-12bb-4374-90ed-0deadbeefbaz",
    "timestamp": "2020-05-20T11:35:11.117Z",
    "timestamp_processing": "2020-05-20T11:35:11.117Z",
    "timerange_start": null,
    "timerange_end": null,
    "streams": [
      "000000000000000000000002"
    ],
    "source_streams": [],
    "message": "Notification test message triggered from user <admin>",
    "source": "000000000000000000000001",
    "key_tuple": [
      "testkey"
    ],
    "key": "testkey",
    "priority": 2,
    "alert": true,
    "fields": {
      "field1": "value1",
      "field2": "value2"
    }
  },
  "backlog": []
}

These are the supported configuration options to be defined in the New Notification menu for HTTP notifications:

• URL

  The URL to POST to when an event occurs.

• Skip TLS Verification (optional)

  Select this option to omit performing a TLS certification of the URL.

• Basic Authentication (optional) 

  The Basic authentication string needs to follow this format: <username>:<password>.

• API Key (optional)

  Enter an optional API key if preferred. Note that if an API secret is set, an API key must also be set.

• API Secret (optional)

  Enter an optional secret if preferred. Note that if an API secret is set, an API key must also be set.

Legacy Alarm Callback [Operations Only] DEPRECIATED

Legacy alarm callbacks are being depreciated and are no longer supported in the documentation. If you are currently using legacy alarm callbacks, please migrate to script notifications as soon as possible. For instructions on creating a new script notification, see the related section in this document.