Event Definitions Use Case

After reviewing how to create an event definition, this article will provide you with an in-depth example on how you might set up a new event definition from the wizard and attach an alert to this event.

Scenario

The following use case scenario describes an instance where Graylog receives log messages for a service you need to authenticate in order to log in (e.g. SSH, web app). Let's say you want to receive an email from Graylog for possible brute force attacks being run on the authentication of the service. If one user fails to log in to the system 10 times in one minute, you want to get an email from Graylog.

Enter Event Details

After clicking on Create Event Definition, you are presented with the event definition wizard. Here, enter the title and description of the event definition and define a priority.

Since you got a message about a brute force attack, which should be treated urgently, set the priority to High and proceed to the next step.

Specify the Condition

Choose Filter & Aggregation for the Condition Type because you only want to see messages that indicate a failed authentication. Enter specific information about the streams and messages you would like to include under Filter.

Define a Filter

First, select the stream in which your log files are routed. If no stream was created for the web application, we highly recommend visiting Streams for more information. The query result will be limited to web application logs and no other logs will influence the filter process.

Now, you need to filter the incoming messages, so you can later count the messages that match the filter. A log message that indicates a failed authentication, will look like this:

Login failed for user admin from ip 240.210.133.39

Add Login failed for user to the Search Query field. Now, a preview table appears on the right side of the screen that shows the messages matching the query. The preview uses Search within the last as the time range. If you provoke a log entry now, you should see this after updating the query.

Set Search within the last to 10 seconds and Execute search every to 10 seconds. The event engine will execute the query every 10 seconds for a time range of 10 seconds.

Warning: If Search within the last is greater than Execute search every (which would be called a hopping window), the event engine will generate multiple events for the same log line. For example, if you set Search within the last to one minute and Execute search every to 10 seconds, the engine will find the same log entry 6 times because the log entry is visible for one minute and we are searching every 10 seconds. If Search within the last is the same as theExecute search every period, you will create something known as a tumbling window, which is recommended for most situations.

Create an Aggregation

Since you want to aggregate on your events (to see if you have more than 10 messages in 10 seconds), select Aggregation of results reaches a threshold under Create Events for Definition if....

To be able to find how many failed login attempts there were per user, you need to add a pipeline to incoming messages. The pipeline will extract the user name and store it in the User field. Every message with a failed login has a User field. Enter that field in the selection Group by Field(s).

Finally, add an aggregation rule: If count() is >=10.

Review Previous Steps

To summarize what you have done so far:

  1. Added a stream to minimize the messages you need to filter on.
  2. Inserted a query to filter the logs down to only failed logins.
  3. Grouped your logs so the aggregation is applied per user.
  4. Added a rule: only raise an alert if the count is more than, or equals 10.

Now, click Next to reach the Fields tab.

Add a Custom Field

Add a custom field to the event by clicking on Add custom field. Since you have an aggregated event definition, the only accessible fields are the ones you configured in Group by Fields. When aggregating multiple messages to one event, you lose all fields you do not group by. Add the user key, (which you put in Group by Fields) to the alert. Then, fill in the required information:

  1. Give the key a name.

  2. Check the box under Use Field as Event Key. This means that alerts will be grouped by this key. This is important in the next step because the group key is considered to be in a grace period. The number that is entered here, reflects the order in the grouping. If we were to add another key (like the IP address from which the user connects), then we would first group by user and then group by IP, entering 2 for the IP address here.

  3. Select a template for Set Value From. This will extract the field from the resulting aggregation or filtered log message. Enter ${source.user}. The resulting aggregation will be the source containing a field named User since we entered it in Group By Fields.

If you only want events which set the key (important if you have filtered log messages), then you could check the Require all template values to be set box. But since this is an aggregation, this step is not needed, and you can move on to the next page.

Add an Alert

You want to receive an email when an event is raised. Configuring an event will elevate it to an alert. You can read more about setting up an email alert under alert types.

Select your predefined email alert and set the Grace Period to 5 minutes. If you are targeted by a brute force attack, you do not want to get an email every 10 seconds reminding you that you are being attacked. This grace period will only be respected per the event key you selected in custom fields. So you will only get an email for usernames the attackers are targeting.

Since this is an aggregated event, setting a number in Message Backlog might not be helpful. You can leave it unchecked. The backlog will show all messages within the time range of Search within the last and it will use the query you entered. Selecting a number here will limit the amount of messages in the backlog.

Save the Event Definition

Take a final look at the event definition on the Summary page. After checking that all configurations are correct, click on Create event definitionto save the event definition.

Sample Alert

If an attacker tries to login 10 times in under 10 seconds, you will receive an email like the one below:

Notice that the user was set as key.

Create a Widget

Navigate to the search page and create a widget following these steps:

  1. Select the All events stream to narrow down all messages to events.

  2. Add event_definition_id to the query in order to only display the events related to the newly created event definition.

  3. Create a new aggregation widget and set Direction as Row and Field as timestamp.

  4. Type key in the Columns field and select count() under Metrics.

The Brute force events per user widget can be added to a report.