As detailed in Alerts, alerts are messages that you can configure to inform you about an event. During the process of creating an event definition, you can attach an alert to the event, and you can choose how these alerts are sent (email, Slack or Discord alert, etc.).

Graylog supports alerts through the following methods:

PagerDuty Alerts (Enterprise Only)

The following section exclusively pertains to a Graylog Enterprise feature. To learn more about obtaining an Enterprise license, please contact the Graylog Sales team.

Warning: In order to use PagerDuty alerts, you will need an integration routing key. If you do not already have one, you will need to create a new integration in PagerDuty.

The PagerDuty alert allows you to create new incidents in PagerDuty in response to events in your Graylog server. These are the supported configuration options to be defined in the New Notification menu for PagerDuty:

  • Routing Key
    Your PagerDuty integration routing key.

  • Use a Custom Incident Key
    If enabled, an incident key will be generated using the provided Incident Key Prefix. This will prevent PagerDuty from creating multiple incidents for a single event. If not selected, an incident key will NOT be generated, and each event notification will create a new incident in PagerDuty.

  • Incident Key Prefix
    If a custom incident key is enabled, this will be used as a prefix for the incident key.

  • Client Name
    The name of the Graylog system that triggered the PagerDuty incident.

  • Client URL
    The URL for your Graylog web interface. If provided, this will be used to construct links that will be embedded in your PagerDuty incident.

Hint: PagerDuty alerts are intended as a replacement for the PagerDuty integration, previously available from Graylog Labs. If you are using the Graylog Labs PagerDuty integration, you should migrate to the officially supported PagerDuty alerts at your earliest convenience.

Slack Alerts

Slack allows you to send alerts to your Slack workspace in response to events in your Graylog server. These are the supported configuration options to be defined in the New Notification menu for Slack:

  • Configuration Color
    Highlight the custom message with a color you prefer.

  • Webhook URL
    The unique URL used to send messages to your Slack instance.

  • Channel
    The channel to which you will send a message or you can select @user or @team to receive the alert.

  • Custom Message
    The message that will be sent to Slack. The data described above can be used in this template.

  • Time Zone for Date/Time Values

    Select your preferred time zone used for timestamps in the alert. This selection will default to UTC.

  • Message Backlog Limit (optional)
    Limit the number of backlog messages sent as part of the Slack alert. If set to 0, no limit will be enforced.

  • User Name (optional)
    User name of the sender in Slack.

  • Selections (you may choose to enable the following options):

    • Include Title: If enabled, this will include the full event definition title and description in the alert.

    • Notify Channel: If enabled, this will notify all the users in a channel with @channel.

    • Notify Here: If enabled, this will notify only active users in the channel with @here.

    • Link Names: If enabled, this will find and link channel names and user names in the alert.

  • Icon URL(optional)
    Image to use as the icon for this message.

  • Icon Emoji(optional) 
    Emoji to use as the icon for this message (this overrides any icon URL).

Hint: Slack alerts are intended as a replacement for the Slack integration previously available from Graylog Labs. If you are using the Graylog Labs Slack plugin, you should migrate to the officially supported Slack alerts at your earliest convenience

Microsoft Teams Alerts

Microsoft Teams alerts allow you to send messages to a Teams channel when specific events occur in your Graylog setup. This is done via a webhook, which is the same method used for Slack alerts.

Warning: Make sure to copy the Webhook URL when creating the webhook! This will need to be pasted into the Webhook URL field upon configuration. Also note that, Microsoft Teams requires webhooks to be generated for all individual channels.

Warning: Make sure to copy the Webhook URL when creating the webhook! This will need to be pasted into the Webhook URL field upon configuration. Also note that, Microsoft Teams requires webhooks to be generated for all individual channels.

These are the supported configuration options to be defined in the New Notification menu for Microsoft Teams:

  • Configuration Color
    Highlight the custom message with a color of your choice.

  • Webhook URL
    This is the unique URL generated while setting up your webhook. Copy the URL that was generated when creating your webhook and paste it into this field.

  • Custom Message
    This is the message that will be sent to your MS Teams channel.

  • Time Zone for Date/Time Values

    Select your preferred time zone used for timestamps in the alert. This selection will default to UTC.

  • Message Backlog Limit (optional)
    Limit the number of backlog messages sent as part of the MS Teams alert. If set to 0, no limit will be enforced.

  • Icon URL
    This is an image to use as the icon for this message.

Script Alerts (Enterprise Only)

The following section exclusively pertains to a Graylog Enterprise feature. To learn more about obtaining an Enterprise license, please contact the Graylog Sales team.

The script alert option lets you configure a script that will be executed when the alert is triggered.

These are the supported configuration options to be defined in the New Notification menu for script alerts:

  • Script Path
    The path to where the script is located. Must be within the permitted script path (which is customizable).

  • Script Timeout
    The maximum time (in milliseconds) the script will be allowed to execute before being forcefully terminated.

  • Script Arguments
    Space-delimited string of parameters. Any of the data described above can be used.

  • Send Alert Data Through STDIN
    Sends the JSON encoded data described above through standard in. You can use a JSON parser in your script.

Script Alert success is determined by its exit value; success equals zero. Any non-zero exit value will cause it to fail. Returning any error text through STDERR will also cause the alarm callback to fail.

Here is a sample Python script that shows all supported Script Alert functionality (argument parsing, STDIN JSON parsing, STDOUT, exit values, and returning an exit value).

Copy
#!/usr/bin/env python3
import json
import sys

# Function that prints text to standard error
def print_stderr(*args, **kwargs):
    print(*args, file=sys.stderr, **kwargs)

# Main function
if __name__ == "__main__":

    # Print out all input arguments.
    sys.stdout.write("All Arguments Passed In: " + ' '.join(sys.argv[1:]) + "\n")

    # Turn stdin.readlines() array into a string
    std_in_string = ''.join(sys.stdin.readlines())

    # Load JSON
    event_data = json.loads(std_in_string)

    # Extract some values from the JSON.
    sys.stdout.write("Values from JSON: \n")
    sys.stdout.write("Event Definition ID: " + event_data["event_definition_id"] + "\n")
    sys.stdout.write("Event Definition Title: " + event_data["event_definition_title"] + "\n")
    sys.stdout.write("Event Timestamp: " + event_data["event"]["timestamp"] + "\n")

    # Extract Message Backlog field from JSON.
    sys.stdout.write("\nBacklog:\n")
    for message in event_data["backlog"]:
        for field in message.keys():
            sys.stdout.write("Field: " + field + "\t")
            sys.stdout.write("Value: " + str(message[field]) + "\n")

    # Write to stderr if desired
    # print_stderr("Test return through standard error")

    # Return an exit value. Zero is success, non-zero indicates failure.
    exit(0) 

Email Alerts

Email alerts can be used to send an email to the configured alert receivers when conditions are triggered. Make sure to check the email-related configuration settings in the Graylog configuration file.

Several configuration options are available for the alert to customize the email that will be sent. Note that the email body and email subject are JMTE templates. JMTE is a minimal template engine that supports variables, loops and conditions. See the JMTE documentation for a language reference.
The default body template shows some advanced examples of accessing the available information in the template:

Copy
--- [Event Definition] ---------------------------
Title:       ${event_definition_title}Description: ${event_definition_description}
Type:        ${event_definition_type}
--- [Event] --------------------------------------
Timestamp:            ${event.timestamp}
Message:              ${event.message}
Source:               ${event.source}
Key:                  ${event.key}
Priority:             ${event.priority}
Alert:                ${event.alert}
Timestamp Processing: ${event.timestamp}
Timerange Start:      ${event.timerange_start}
Timerange End:        ${event.timerange_end}
Fields:
${foreach event.fields field}${field.key}: ${field.value}
${end}
${if backlog}
--- [Backlog] ------------------------------------
Last messages accounting for this alert:
${foreach backlog message}
${message}
${end}
${end}

These are the supported configuration options to be defined in the New Notification menu for email alerts:

  • Subject

    The subject used for the email alert. As noted above, the email subject is a JMTE template.

  • Reply-To

    The email address recipients should reply to if necessary.

  • Sender

    The email address that will be used as the sending address. If the field is left empty, the default address will be used.

  • User Recipient(s) (optional)

    Select all Graylog users that will receive the alert.

  • Email Recipient(s) (optional)

    Add any additional email addresses to receive the alert.

  • Time Zone for Date/Time Values (optional)

    You can opt to use a specific time zone for the body of the email.

  • Body Template

    This is the template used to generate the email body. As noted above, the email body is a JMTE template.

  • HTML Body Template

    This is the template used to generate the email HTML body.

Dynamic Email Notification Addresses

Graylog allows email notifications to be sent to multiple recipients. A single event definition is paired with a notification forming an alert that can be emailed to multiple email addresses using a lookup table.

The Reply-To email, Sender email, and Email Recipients values can be modified to use a lookup table by checking the relevant box. You may then select:

  • The lookup table you want to use.

  • The event field to use for the lookup.

For example, by checking the Use lookup table for email recipients box, you can set the email notification to send the alert to a customer found in a lookup table that relates customer IDs with email addresses. This way, you do not have to set up individual alerts for multiple clients. The lookup table does the mapping.

HTTP Alerts

HTTP alerts allow you to configure an endpoint that is called when the alert is triggered.

Graylog sends a POST request to the alert URL, including information about the alert. The body of the request is the JSON encoded data described above.

Here is an example of the payload included in an alert:

Copy
{
  "event_definition_id": "this-is-a-test-notification",
  "event_definition_type": "test-dummy-v1",
  "event_definition_title": "Event Definition Test Title",
  "event_definition_description": "Event Definition Test Description",
  "job_definition_id": "<unknown>",
  "job_trigger_id": "<unknown>",
  "event": {
    "id": "NotificationTestId",
    "event_definition_type": "notification-test-v1",
    "event_definition_id": "EventDefinitionTestId",
    "origin_context": "urn:graylog:message:es:testIndex_42:b5e53442-12bb-4374-90ed-0deadbeefbaz",
    "timestamp": "2020-05-20T11:35:11.117Z",
    "timestamp_processing": "2020-05-20T11:35:11.117Z",
    "timerange_start": null,
    "timerange_end": null,
    "streams": [
      "000000000000000000000002"
    ],
    "source_streams": [],
    "message": "Notification test message triggered from user <admin>",
    "source": "000000000000000000000001",
    "key_tuple": [
      "testkey"
    ],
    "key": "testkey",
    "priority": 2,
    "alert": true,
    "fields": {
      "field1": "value1",
      "field2": "value2"
    }
  },
  "backlog": []
}

These are the supported configuration options to be defined in the New Notification menu for HTTP alerts:

  • URL

    The URL to POST to when an event occurs.

  • Skip TLS Verification (optional)

    Select this option to omit performing a TLS certification of the URL.

  • Basic Authentication (optional) 

    The Basic authentication string needs to follow this format: <username>:<password>.

  • API Key (optional)

    Enter an optional API key if preferred. Note that if an API secret is set, an API key must also be set.

  • API Secret (optional)

    Enter an optional secret if preferred. Note that if an API secret is set, an API key must also be set.

  • Send API Key/Secret as Header (optional)

    Select this option to send the API key and secret as a header in the HTTP request. If this option is not selected, the API key and secret are sent as a query parameter in the URL of the request, which is less secure and might not be accepted by some applications.

Custom HTTP Alerts

Custom HTTP alerts provide all the options of standard HTTP alerts (above) plus additional options to configure the way information is sent and formatted. For instance, you can select from different content types for POST and PUT methods, and you can include additional headers.

You can also customize the HTTP request body using a JMTE template. JMTE is a minimal template engine that supports variables, loops, and conditions. See the JMTE documentation for a language reference.

In addition to the options listed above for HTTP alerts, configure the following options in the New Notification menu to define custom HTTP alerts:

  • Headers (optional)

    Enter additional HTTP headers to include with the alert.

    Hint: If this notification calls the Graylog REST API, in addition to providing proper authorization, you must add a header named X-Requested-By. Without this header, the Graylog API rejects the call.

  • HTTP Method

    Select the method type for the alert. The default value is POST. Any response returned from a GET request is ignored.

  • Content Type

    Select the content type for the alert body. The default value is application/JSON. This option is available only for POST and PUT methods.

  • Time zone for date/time values

    Choose a time zone for formatting any dates or times that are included in the alert body. The default value is UTC. This option is available only for POST and PUT methods.

  • Body Template

    This field is the template used to generate the alert body. Each content type has its own default template. Use the default template or refer to the JMTE documentation to learn how to customize the template for your needs.

    If you leave this field empty, the entire alert object is sent. For application/JSON and text/plain content types, the body is the same as the standard HTTP alert above. For application/x-www-form-urlencoded, an empty body results in a payload formatted as follows:

    Copy
    event_definition_id=NotificationTestId&event_definition_type=test-dummy-v1
    &event_definition_title=Event%20Definition%20Test%20Title&event_definition_description=Event%
    20Definition%20Test%20Description&job_definition_id=%3Cunknown%3E&job_trigger_id=%3Cunknown%
    3E&event_id=TEST_NOTIFICATION_ID&event_origin_context=urn%3Agraylog%3Amessage%3Aes%
    3AtestIndex_42%3Ab5e53442-12bb-4374-90ed-0deadbeefbaz&event_timestamp=2023-11-17T11%3A49%
    3A40.905-06%3A00&event_timestamp_processing=2023-11-17T11%3A49%3A40.905-06%3A00
    &event_timerange_start=&event_timerange_end=&event_streams=%5B000000000000000000000002%
    5D&event_source_streams=%5B%5D&event_message=Notification%20test%20message%20triggered%20from%
    20user%20%3Clocal%3Aadmin%3E&event_source=000000000000000000000001&event_key_tuple=%5Btestkey%
    5D&event_key=testkey&event_priority=2&event_fields=%7Bfield1%3Dvalue1%2C%20field2%3Dvalue2%
    7D&backlog=%5B%5D

Legacy Alarm Callback [Enterprise Only] DEPRECATED

The following section exclusively pertains to a Graylog Enterprise feature. To learn more about obtaining an Enterprise license, please contact the Graylog Sales team.

Legacy alarm callbacks are depreciated and are no longer supported in the documentation. If you are currently using legacy alarm callbacks, please migrate to script alerts as soon as possible. You can read more about creating script alerts above.