HTTP JSONPath Data Adapter

The HTTP JSONPath data adapter in Graylog enables the extraction of specific data from HTTP requests or responses formatted in JSON. This is particularly useful for parsing JSON-formatted logs or messages and extracting key information for indexing, analysis, and enrichment within Graylog.

Prerequisites

Before proceeding, ensure that the following prerequisites are met:

Familiarize yourself with JSONPath syntax and usage, as Graylog uses JSONPath expressions to parse and extract data from JSON-formatted log messages.

Ensure that the logs or data you want to parse with the HTTP JSONPath data adapter are accessible through HTTP endpoints or services, such as APIs that return JSON-formatted data.

Configure the Data Adapter

You can create a data adapter during the lookup table creation workflow, or they can be created separately on the Data Adapters tab. The following configuration options are available for this data adapter:

Configuration Parameters

Title	A short and unique title for this data adapter.
Description	Data adapter description.
Name	The name used to refer to this data adapter. This should be something unique and recognizable within your Graylog environment.
Custom Error TTL	Time-to-live for custom error messages in seconds. This controls how long custom error responses are cached. If no value is specified, the default is 5 seconds.
Lookup URL	This parameter specifies the URL where Graylog should perform HTTP requests to retrieve additional data or perform lookups. For example, you might use this to enrich log messages with geolocation data based on IP addresses. To use the lookup key in the URL, the`${key}`value can be used. This variable will be replaced by the actual key that is parsed to a lookup function. (example: `https://example.com/api/lookup?key=${key}`)
Single value JSONPath	This parameter defines a JSONPath expression that Graylog uses to extract a single specific value from the JSON payload of an HTTP request or response. For instance, if your JSON payload contains information like `"username": "john.doe"`, your JSONPath expression might be `$['username']` to extract the value `john.doe`.
Multi value JSONPath	Similar to the single value JSONPath, but this parameter allows you to extract multiple values from the JSON payload using JSONPath expressions. You can specify multiple expressions to extract various fields or values from the JSON data. This JSONPath expression will be used to parse the multi value of the lookup result. (Example: `$.users[]`) The multi value JSONPath setting is optional. Without it, the single value is also present in the multi-value result. Hint:* Note that if incorrect values for 'Single value JSONPath' or 'Multi value JSONPath' are used, this data adapter will not return a result. If you are certain that the key you are querying has a valid result, but you get no result using this data adapter, verify your values are correct for 'Single value JSONPath' or 'Multi value JSONPath'.
HTTP User-Agent	This parameter refers to the User-Agent header used in HTTP requests. It allows Graylog to extract and analyze the user-agent string sent by clients making HTTP requests. This can be useful for identifying the types of devices or browsers accessing your services.
HTTP Headers	The custom HTTP headers to use for the HTTP request. This can be used to add Authentication information. For example, the HTTP Basic Auth header 'Authorization'.

Example HTTP JSONPath Output

This example demonstrates an example configuration and the values returned from a lookup.
The configured lookup URL is https://example.com/api/users/${key} and the ${key}gets replaced by jane during the lookup request.

This is the resulting JSON document:

{
  "user": {
    "login": "jane",
    "full_name": "Jane Doe",
    "roles": ["admin", "developer"],
    "contact": {
      "email": "jane@example.com",
      "cellphone": "+49123456789"
    }
  }
}

The following examples show how different JSONPath configurations affect the lookup results returned by this example. Each configuration and result demonstrates a unique combination of single-value and multi-value JSONPath expressions and the corresponding output Graylog produces from the same JSON document.

Configuration	Result
Single value JSONPath: `$.user.full_name` Multi value JSONPath: empty	Single value: `Jane Doe` Multi value: {"value": "Jane Doe"}
Single value JSONPath: `$.user.full_name` Multi value JSONPath: `$.user`	Single value: `Jane Doe` Multi value: { "login": "jane", "full_name": "Jane Doe", "roles": ["admin", "developer"], "contact": { "email": "jane@example.com", "cellphone": "+49123456789" } }
Single value JSONPath: `$.user.contact.email` Multi value JSONPath: `$.user.roles[*]`	Single value: `jane@example.com` Multi value: { "value": ["admin", "developer"] }
Single value JSONPath: `$.user.full_name` Multi value JSONPath: `$.user.contact`	Single value: `Jane Doe` Multi value: { "email": "jane@example.com", "cellphone": "+49123456789" }

Example HTTP JSONPath Pipeline Rule

This rule enriches messages containing a user_login by querying an external user API (via a Graylog lookup table) to add the user’s full name, email, and cellphone number to the log message.

Copy

rule "lookup user"
when has_field("user_login")
then
  // Get the user login from the message
  let userLogin = to_string($message.user_login);
  // Lookup the single value, in our case the full name, in the user-api lookup table
  let userName = lookup_value("user-api", userLogin);
  // Set the field "user_name" in the message
  set_field("user_name", userName)

  // Lookup the multi value in the user-api lookup table
  let userData = lookup("user-api", userLogin);
  // Set the email and cellphone as fields in the message
  set_field("user_email", userData["email"]);
  set_field("user_cellphone", userData["cellphone"]);
end