Alerts

Alerts are notifications that you can set up to inform you about an event. They can be created via the Create Notification selection on the Notifications page under Alerts or can be defined in the new event definition workflow.

Assigned alerts are displayed on the Notifications page. This page is also bulk friendly and allows you to edit multiple entities. You can see if your alerts are active by clicking on the Test Notifications button under More Actions. You will then see a success or error message under the entity title.

In this section, we explain how to create an alert and how to configure supported alert types.

Hint: Alerts are meant to be extensible through plugins. You can find more options in Graylog Marketplace or even create your own.

Prerequisites

The following prerequisites are required before utilizing alerts with Graylog:

• Alerts are triggered by defined events, therefore an event definition is required.

• Graylog Enterprise is required for script and PagerDuty alerts.

Alerts can be created by selecting Notifications under the Alerts menu or by defining them in the event workflow.

Events and Alerts Page

Beneath the metrics graph, all your events and alerts are listed in a table, grouped into various columns, beginning with a description of the event/alert, priority, key, type, event definition, event definition type, timestamp and actions. Additional columns can be displayed or removed by clicking on the Columns dropdown button and selecting available column categories.

From the list of all displayed alerts and events, you can either click on an event or alert to expand and view its details or you can click on the Details button in the Actions column.. You can also add notifications to an event by clicking on the More dropdown menu button in the Actions column and select Send Notifications.

Create an Alert from the Alerts Menu

1. Navigate to the Alerts menu and select Notifications.

2. Select the Create notification button.

3. Complete the following fields: 

   1. Title: Create a unique title for your alert.

   2. Description (optional): You may add additional details about your alert in this field if desired.

   3. Notification Type: Select the alert type from the drop-down menu.

4. After you select an alert type, you are presented with additional fields based on the type selected. The alert types and key fields to configure for each are detailed in Alert Types.

5. You may also choose to test your alert at this time by selecting Execute Test Notification.

6. Click Create notification.

Create an Alert in the New Event Definition Workflow

You may also choose to create an alert while you are in the process of defining a new event.

1. In the New Event Definition menu, there will be a selection for Notifications in the menu bar.

2. Under Add Notification, select Create New Notification from the drop-down menu.

3. From the menu that populates, complete the following fields: 

   1. Title: Create a unique title for your alert.

   2. Description (optional): You may add additional details about your alert in this field if desired.

   3. Notification Type: Select the alert type from the drop-down menu.

4. After you select an alert type, you are presented with additional fields based on the type selected. The alert types and key fields to configure for each are detailed in Alert Types.

5. You may also choose to test your alert at this time by selecting Execute Test Notification.

6. Click Create notification.

Metadata Available to Alerts

When creating alerts you can utilize metadata from the event definition, the event itself, and the event's backlog messages (if it is configured to retain a backlog). This metadata can be used when formatting email, Slack, and Microsoft Teams alerts or when providing arguments to a script alert.

For example, if you wish to include more information in your Slack alerts, you may add new fields to the Custom Message section. You may also remove any fields that you do not wish to see by deleting them from this section.

Or you could add arguments to a script alert to include more information in your alerts.

Fields that are available for each entity type are detailed below.

Event Definition Metadata

Field	Type	Description
event_definition_id	String	The database ID of the event definition
event_definition_type	String	The internal name of the event definition type (aggregation-v1 or correlation-v1)
event_definition_title	String	The title set in the UI
event_definition_description	String	The description set in the UI
job_definition_id	String	The internal job definition ID associated with a scheduled event definition
job_trigger_id	String	The internal ID associated with the current execution of the job.

Event Metadata

Field	Type	Description
event		The event as it is stored in Graylog
id	String	The message ID of the stored event
event_definition_id	String	The database ID of the event definition
event_definition_type	String	The internal name of the event definition type (aggregation-v1 or correlation-v1)
origin_context	String	URN of the message or event creating this event (eitherevent or message) (can be empty)
timestamp	DateTime	The timestamp can be set to the underlying event or message (see origin_context above)
timestamp_processing	DateTime	The timestamp for when the event was created by Graylog
timerange_start	DateTime	The start of the window of data Graylog used to create this event (can be empty)
timerange_end	DateTime	The end of the window of data Graylog used to create this event (can be empty)
streams	String	The list of stream IDs the event is stored in
source_streams	String	The list of stream IDs the event pulled data from
alert	Boolean	Whether this event is considered to be an alert; always true for event definitions that have alerts
message	String	A human-friendly message describing this event
source	String	The host name of the Graylog server that created this event
key_tuple	String	The list of values making up the event’s key
key	String	The event’s key as a single string
priority	Long	The event’s priority value
fields	Map	The custom fields attached to the event

Backlog Metadata

Field	Type	Description
backlog		The list of messages or events which lead to the alert being generated
id	String	The message ID
index	String	The name of the index the message is stored in; use together with id to uniquely identify a message in Graylog
source	String	The source field of the message
message	String	The message field of the message
timestamp	DateTime	Thetimestamp field of the message
stream_ids	String	The stream IDs of the message
fields	Map	The remaining fields of the message (can be iterated)

Delete Queued Alerts

If processing stops and event updates begin to pile up in the queue, then you might have unknowingly fired too many alerts. To avoid an influx of alerts, make sure to set an alert grace period for event definitions. The grace period enforces a rate limit on how many alerts are triggered for identical events. This effectively prevents queued event alerts. Without a grace period in place, too many event triggers can cause a backlog of alerts.

If you are faced with queued event alerts, there are two ways of clearing the alert queue.

Clear Alert Queue Manually

Clear the alert queue manually through the interface:

1. Navigate to the Events Definition menu by selecting Alerts > Event Definitions.

2. From the list of definitions available, click on the Information icon under Scheduling.

3. The event definition menu will expand. Here, you will see the number of queued alerts. If there are a lot of queued alerts, this typically suggests an abnormality. On the Queued alerts line, click on clear to clear queued alerts for the selected event definition.

Disable an Event

You can also clear the alert queue by disabling an event.

1. As in the previous example, navigate to the Events Definitions tab.

2. Next to your event definition, click the More drop-down button, and select Disable from the menu option for the event you wish to disable.

3. Upon selecting the Disable option, a pop-up dialog screen appears, prompting you to confirm the selection.

4. When disabled an alert is displayed confirming that the selected event definition has been disabled.