Alerts

Alerts are notifications that you can set up to inform you about an event. They can be created via the Create Notification selection on the Notifications page under Alerts or can be defined in the new event definition workflow.

Assigned alerts are displayed on the Notifications page. This page is also bulk friendly and allows you to edit multiple entities. You can see if your alerts are active by clicking on the Test Notifications button under More Actions. You will then see a success or error message under the entity title.

In this section, we explain how to create an alert and how to configure supported alert types.

Hint: Alerts are meant to be extensible through plugins. You can find more options in Graylog Marketplace or even create your own.

Prerequisites

The following prerequisites are required before utilizing alerts with Graylog:

Alerts can be created by selecting Notifications under the Alerts menu or by defining them in the event workflow.

Create an Alert from the Alerts Menu

  1. Navigate to the Alerts menu and select Notifications.

  2. Select the Create notification button.

  3. Complete the following fields: 

    1. Title: Create a unique title for your alert.

    2. Description (optional): You may add additional details about your alert in this field if desired.

    3. Notification Type: Select the alert type from the drop-down menu.

  4. After you select an alert type, you are presented with additional fields based on the type selected. The alert types and key fields to configure for each are detailed in Alert Types.

  5. You may also choose to test your alert at this time by selecting Execute Test Notification.

  6. Click Create notification.

Create an Alert in the New Event Definition Workflow

You may also choose to create an alert while you are in the process of defining a new event.

  1. In the New Event Definition menu, there will be a selection for Notifications in the menu bar.

  2. Under Add Notification, select Create New Notification from the drop-down menu.

  3. From the menu that populates, complete the following fields: 

    1. Title: Create a unique title for your alert.

    2. Description (optional): You may add additional details about your alert in this field if desired.

    3. Notification Type: Select the alert type from the drop-down menu.

  4. After you select an alert type, you are presented with additional fields based on the type selected. The alert types and key fields to configure for each are detailed in Alert Types.

  5. You may also choose to test your alert at this time by selecting Execute Test Notification.

  6. Click Create notification.

Metadata Available to Alerts

When creating alerts you can utilize metadata from the event definition, the event itself, and the event's backlog messages (if it is configured to retain a backlog). This metadata can be used when formatting email, Slack, and Microsoft Teams alerts or when providing arguments to a script alert.

For example, if you wish to include more information in your Slack alerts, you may add new fields to the Custom Message section. You may also remove any fields that you do not wish to see by deleting them from this section.

Or you could add arguments to a script alert to include more information in your alerts.

Fields that are available for each entity type are detailed below.

Event Definition Metadata

Field Type Description
event_definition_id

String

The database ID of the event definition
event_definition_type

String

The internal name of the event definition type (aggregation-v1 or correlation-v1)
event_definition_title

String

The title set in the UI
event_definition_description

String

The description set in the UI
job_definition_id

String

The internal job definition ID associated with a scheduled event definition
job_trigger_id

String

The internal ID associated with the current execution of the job.

Event Metadata

Field Type Description
event

 

The event as it is stored in Graylog

id

String

The message ID of the stored event

event_definition_id

String

The database ID of the event definition

event_definition_type 

String

The internal name of the event definition type (aggregation-v1 or correlation-v1)

origin_context

String

URN of the message or event creating this event (eitherevent or message) (can be empty)

timestamp

DateTime

The timestamp can be set to the underlying event or message (see origin_context above)

timestamp_processing

DateTime

The timestamp for when the event was created by Graylog

timerange_start

DateTime

The start of the window of data Graylog used to create this event (can be empty)

timerange_end

DateTime

The end of the window of data Graylog used to create this event (can be empty)

streams

String

The list of stream IDs the event is stored in

source_streams 

String

The list of stream IDs the event pulled data from

alert

Boolean

Whether this event is considered to be an alert; always true for event definitions that have alerts

message 

String

A human-friendly message describing this event

source

String

The host name of the Graylog server that created this event

key_tuple

String

The list of values making up the event’s key

key

String

The event’s key as a single string

priority

Long

The event’s priority value

fields

Map

The custom fields attached to the event

Backlog Metadata

Field Type Description
backlog

 

The list of messages or events which lead to the alert being generated

id 

String

The message ID

index

String

The name of the index the message is stored in; use together with id to uniquely identify a message in Graylog

source

String

The source field of the message

message

String

The message field of the message

timestamp

DateTime

Thetimestamp field of the message

stream_ids

String

The stream IDs of the message

fields

Map

The remaining fields of the message (can be iterated)

Delete Queued Alerts

If processing stops and event updates begin to pile up in the queue, then you might have unknowingly fired too many alerts. To avoid an influx of alerts, make sure to set an alert grace period for event definitions. The grace period enforces a rate limit on how many alerts are triggered for identical events. This effectively prevents queued event alerts. Without a grace period in place, too many event triggers can cause a backlog of alerts.

If you are faced with queued event alerts, there are two ways of clearing the alert queue.

Clear Alert Queue Manually

Clear the alert queue manually through the interface:

  1. Navigate to the Events Definition menu by selecting Alerts > Event Definitions.

  2. From the list of definitions available, click on the Information icon under Scheduling.

  3. The event definition menu will expand. Here, you will see the number of queued alerts. If there are a lot of queued alerts, this typically suggests an abnormality. On the Queued alerts line, click on clear to clear queued alerts for the selected event definition.

Disable an Event

You can also clear the alert queue by disabling an event.

  1. As in the previous example, navigate to the Events Definitions tab.

  2. Next to your event definition, click the More drop-down button, and select Disable from the menu option for the event you wish to disable.

  3. Upon selecting the Disable option, a pop-up dialog screen appears, prompting you to confirm the selection.

  4. When disabled an alert is displayed confirming that the selected event definition has been disabled.