Watchlists

Watchlists are named collections of indicators that you maintain, such as IP addresses, user names, file hashes, ports, domains, country codes, and file names or paths. Graylog checks your log data against them. When certain fields in a log message contain a value that is on a watchlist, Graylog flags the value in the message details. You can also reference a watchlist directly in searches, dashboards, event definitions, and pipeline rules.

This article explains how watchlists work, including the built-in watchlist types, entry validation and expiry, how matching works, and how existing watchlist data is migrated when you upgrade.

Permissions apply to all watchlists in the cluster. You cannot grant access to individual watchlists. If you do not have the watchlist:read permission, the Watchlists entry does not appear in the navigation.

Upgrading From Earlier Versions

When you upgrade to Graylog 7.2, a one-time migration runs automatically. The migration creates the seven built-in watchlists, each with its own lookup table, data adapter, and cache. It then copies your existing IP, user, and file hash watchlist entries into the new watchlists. The Port, Domain, GeoLocation, and File watchlists start empty.

Note the following migration behavior:

• Existing entries that do not pass the new validation rules are skipped and logged as warnings. If you expect a value that is missing after the upgrade, search the server log for Skipping invalid legacy watchlist value.

• Single IP addresses migrate as host entries (/32 for IPv4, /128 for IPv6) and continue to match exactly as before.

• Lookup table parameters in searches, dashboards, and event definitions that reference the legacy watchlist lookup table with the ip, user, or hash key are converted to the new Watchlist parameter type.

• Existing pipeline rules that use the watchlist_* functions, and the previous watchlist REST API endpoints, continue to work, because they resolve the same type names (ip, user, hash) to the migrated watchlists.

• The migration runs once per cluster. If it fails, Graylog retries it automatically the next time the server starts. No administrator action is required.

Legacy Watchlist Lookup Table

The migration does not delete the legacy watchlist lookup table, data adapter, and cache. It leaves them in place and marks them as deprecated, with a note that they are no longer used. Graylog's watchlist features no longer rely on them.

If any of your pipeline rules reference the legacy lookup table directly with lookup_* functions, update those rules to use the watchlist functions described in Pipeline Functions. Because the legacy lookup table is no longer used, rules that read it directly do not reflect your current watchlist entries.

After you remove all direct references to the legacy lookup table, you can safely delete the deprecated lookup table, data adapter, and cache.

Built-In Watchlists

Graylog provides seven built-in watchlists, one for each watchlist type. The built-in watchlists are system entities. You cannot rename or delete them, and you cannot create additional watchlists. You manage the entries in each one.

Accepted Values

Graylog validates each value when you add it and stores it in a normalized form. If a value in a batch is invalid, Graylog rejects that value and reports the reason, and still stores the valid values in the same batch.

Entries are unique by their normalized form. For example, 2001:db8:0:0:0:0:0:1 and 2001:db8::1 are the same IP entry. If you add a value that normalizes to an existing entry, the new entry replaces the existing one and takes its expiry from the new batch's TTL.

Entry Expiry

Watchlist entries can expire automatically after a set time to live (TTL). Expiry is optional and works on two levels:

• Default TTL: Each watchlist has a default TTL that applies to new entries added without their own TTL. A default TTL of 0, or no default TTL, means new entries never expire. Changing the default TTL affects only entries you add afterward. Existing entries keep their current expiry.

• Per-batch TTL: When you add entries, you can set a TTL for that batch that overrides the watchlist default. A per-batch TTL of 0 makes those entries permanent, even if the watchlist has a default TTL.

Expired entries stop matching and are removed automatically. Removal is handled by a background job, so an expired entry might remain visible on the watchlist page for up to about a minute after its expiry time.

How Matching Works

For every watchlist type except IP and Port, a field value matches an entry when their normalized forms are identical. As a result, hashes, domains, and user names match regardless of case.

IP and Port Ranges

The IP watchlist matches an address against both exact entries and CIDR ranges. For example, if the IP watchlist contains 10.0.0.0/8, a message with source_ip: 10.1.2.3 matches, even though that exact address was never added.

The Port watchlist works the same way with port ranges. It matches a port against both exact entries and ranges. For example, if the Port watchlist contains 8000-8100, a message with destination_port: 8080 matches, even though port 8080 was never added.

When a value matches through a range, the message details view shows the matched CIDR range beneath the value. If you remove that value from the watchlist in the message details view, Graylog removes the entire range, not only the single address, because removing only the address would leave the match in place. Graylog asks you to confirm before removing the range.

Fields Checked in Message Details

When you open a message's details, Graylog checks a fixed set of Graylog Information Model (GIM) fields against the watchlist of the matching type. Values found on a watchlist appear in a Found on watchlist section. The following fields are checked:

How You Use Watchlists

Watchlists are referenced, not copied. Message details, query parameters, and pipeline functions all look up a watchlist each time they run, so a change to its entries, including an entry expiring, applies the next time each of them runs. There is exactly one built-in watchlist per type, so every reference to a type uses the same set of entries. Decide what each watchlist represents in your environment (for example, that the IP Watchlist holds addresses under suspicion), and use each watchlist consistently across searches, event definitions, and pipeline rules.

Message Details

When you open a message, Graylog checks the fields listed in Fields Checked in Message Details against the watchlist of the matching type. For example, if the IP Watchlist contains 10.0.0.0/8 and a message has source_ip: 10.1.2.3, the message details list source_ip under Found on watchlist, name the IP Watchlist, and show Matched CIDR range: 10.0.0.0/8.

You can add a value to a watchlist from a field value's action menu in search results. For example, after you add a suspicious destination_domain value to the Domain Watchlist, every search, event definition, and pipeline rule that references the Domain Watchlist uses it the next time it runs.

Query Parameters

A watchlist parameter is a query parameter of the Watchlist type that you can declare in a search, dashboard, or event definition. When the query runs, Graylog replaces $parameter_name$ with the watchlist's current entries, each quoted and escaped, joined with OR, and wrapped in parentheses. Always place the parameter on the right-hand side of a field query.

For example, to find successful logins by users on the User Watchlist, declare a parameter named watched_users bound to the User Watchlist and run the following query:

user_name:$watched_users$ AND event_outcome:success

If the User Watchlist contains alice, bob, and john smith, Graylog sends the following query to the search backend:

The same pattern applies to the other types. To find successful activity from countries on the GeoLocation Watchlist, use a parameter bound to the GeoLocation Watchlist:

source_geo_country_iso:$watched_countries$ AND event_outcome:success

To find traffic to ports on the Port Watchlist, such as ports used by remote-access tools you do not allow, use a parameter bound to the Port Watchlist:

destination_port:$watched_ports$

In an event definition, declare the parameter with the Watchlist type in the filter configuration and reference it in the search query. The parameter is resolved on every execution, so adding or removing an entry changes what the event definition matches on its next run, without editing the definition.

To exclude watched values instead, negate the field query, for example NOT source_ip:$watched_ips$. Because each type has a single watchlist, if you use the IP Watchlist for exclusion, for example as a list of your own vulnerability scanners, it cannot also serve as a list of suspicious addresses.

A parameter expands to at most 1,000 entries. CIDR entries match as ranges only on fields mapped as the ip type. On fields of any other type, a CIDR entry matches only the exact literal string, for example 10.0.0.0/8. For all parameter limits, see Manage Watchlists.

Pipeline Functions

Three pipeline functions read and update watchlists during message processing. Each takes a watchlist type (ip, user, hash, port, domain, geolocation, file) and acts on the built-in watchlist of that type:

The functions are most useful in combination with query parameters. The following rule adds the source address of any message reporting more than 25 failed logins to the IP Watchlist with a TTL of one hour (3,600 seconds). The example assumes your messages carry a failed_login_count field:

rule "Add brute-force sources to the IP Watchlist for one hour"
when
  has_field("source_ip") && to_long($message.failed_login_count) > 25
then
  watchlist_add("ip", [to_string($message.source_ip)], 3600);
end

The watchlist_add function takes a list, so wrap a single value in brackets. While the entry exists, an event definition with the query source_ip:$watched_ips$ matches any further activity from that address. After one hour, the entry expires and the event definition stops matching it, with no cleanup required.

To mark messages from watched sources as they are processed, use watchlist_contains. Unlike a query parameter, this function matches IP addresses against CIDR ranges regardless of how the field is mapped. The following rule sets a watchlist_hit field that you can then search on, for example with watchlist_hit:true:

rule "Flag messages from watched IPs"
when
  watchlist_contains("ip", to_string($message.source_ip))
then
  set_field("watchlist_hit", true);
end

Illuminate Content

Illuminate event definitions and dashboards can use the built-in watchlists. Illuminate provides the detection logic, and your watchlists provide the values to look for. Illuminate doesn't ship any watchlist entries. You add them yourself.

Content Packs

Content packs don't include watchlist entries. When a content pack contains a saved search, dashboard, or event definition that uses a watchlist, the pack stores only the watchlist's name. When you install the pack, Graylog looks for a watchlist with that name on the target system and uses that system's entries. If there's no watchlist with that name, the installation fails with a content pack error.

Backing Lookup Tables

Each watchlist is stored in its own lookup table, with each watched value stored as a separate key. Storing values as keys is what allows individual entries to expire and IP entries to match as CIDR ranges. The lookup table appears under System > Lookup Tables as watchlist_<type>_watchlist, with a data adapter named watchlist_adapter_<type>_watchlist and a cache named watchlist_cache_<type>_watchlist, where <type> is the watchlist type. For example, the IP Watchlist uses the lookup table watchlist_ip_watchlist.

Auditing

Graylog records watchlist changes made through the API in the audit log with the event types watchlist:entry:add, watchlist:entry:remove, and watchlist:watchlist:update_ttl. Read operations are not audited.

Further Reading

Explore the following additional resources and recommended readings to expand your knowledge on related topics:

• Manage Watchlists

• Pipeline Functions

• Lookup Tables

• Search Parameters

• Event Definitions