Search Scripting API

Prerequisites

Before proceeding, ensure that the following prerequisites are met:

• You have credentials for authenticating API requests. The search scripting API uses the same authentication as the core Graylog REST API.

• You are familiar with Graylog search syntax and basic REST API usage.

• You know which streams, fields, or metrics you want to query.

Response Formats Explained

The search scripting API returns search results through API calls. Responses can contain either a list of messages or the results of an aggregation query.

The format of the response depends on the Accept header included in the request. Supported response formats are:

• text/plain

• text/csv

• application/json

Plain text and CSV responses contain the same data but present it in different output formats. JSON responses include the same result data along with additional details, such as the result schema and metadata including the effective time range.

Retrieve Messages with the Search Scripting API

You can use the messages endpoint to retrieve messages from a search query. Results are sorted by timestamp with the most recent messages returned first by default. This endpoint is useful when you want to return a limited set of recent messages and control which fields appear in the response.

1. Send a GET request to retrieve messages with selected fields, or use a POST request to specify query parameters in a structured request body:

   GET https://example.org/api/search/messages?fields=source,http_method,http_response_code,took_ms

   POST https://example.org/api/search/messages

   Hint: For the streams, sources, and inputs endpoints, the default field returned is title. To return the ID instead, append .id to the field name.

2. If you use a POST request, define the query parameters in the JSON body, as in this example:

   Copy

   {
       "query": "http_method:POST",
       "streams": ["620f890b70fb980467aca611"],
       "fields": [
           "timestamp",
           "source",
           "http_method",
           "http_response_code",
           "took_ms"
       ],
       "from": 2,
       "size": 15,
       "timerange": {
           "type": "keyword",
           "keyword": "last five minutes"
       },
       "sort": "took_ms",
       "sort_order": "desc"
   }

3. Set the Accept header to control the response format.

4. Review the response output in the format required for your script or workflow.

Example Message Responses

The following examples show how the same message query is returned in CSV, plain text, and JSON formats.

This example sends a request with the Accept: text/csv header. The response returns the search results in CSV format, which is useful when exporting data or processing results with spreadsheet tools.

➜  curl  -H 'Accept: text/csv' -H 'X-Requested-By: cli'  -u username:password 'https://example.org/api/search/messages?fields=source,http_method,http_response_code,took_ms'
"field: source","field: http_method","field: http_response_code","field: took_ms"
"example.org","GET","200","36"
"example.org","GET","200","36"
"example.org","GET","200","48"
"example.org","PUT","200","129"
"example.org","POST","201","134"
"example.org","POST","201","134"
"example.org","GET","200","52"
"example.org","GET","200","48"
"example.org","GET","200","48"
"example.org","GET","200","63"

This example sends the same request with the Accept: text/plain header. The response displays the results in a readable table format in the terminal.

➜  curl  -H 'Accept: text/plain' -H 'X-Requested-By: cli'  -u username:password 'https://example.org/api/search/messages?fields=source,http_method,http_response_code,took_ms'
┌────────────────────────┬────────────────────────┬────────────────────────┬───────────────────────┐
│field: source           │field: http_method      │field:                  │field: took_ms         │
│                        │                        │http_response_code      │                       │
├────────────────────────┼────────────────────────┼────────────────────────┼───────────────────────┤
│example.org             │GET                     │200                     │56                     │
│example.org             │GET                     │200                     │45                     │
│example.org             │GET                     │200                     │44                     │
│example.org             │GET                     │200                     │56                     │
│example.org             │GET                     │200                     │42                     │
│example.org             │DELETE                  │204                     │89                     │
│example.org             │DELETE                  │204                     │89                     │
│example.org             │GET                     │200                     │38                     │
│example.org             │GET                     │200                     │38                     │
│example.org             │DELETE                  │204                     │95                     │
└────────────────────────┴────────────────────────┴────────────────────────┴───────────────────────┘%     

This example sends the request with the Accept: application/json header. The response returns structured JSON output, including the result data, schema information, and metadata such as the effective time range.

➜  curl  -H 'Accept: application/json' -H 'X-Requested-By: cli'  -u username:password 'https://example.org/api/search/messages?fields=source,http_method,http_response_code,took_ms'
{
  "schema": [
    {
      "column_type": "field",
      "type": "unknown",
      "field": "source",
      "name": "field: source"
    },
    {
      "column_type": "field",
      "type": "string",
      "field": "http_method",
      "name": "field: http_method"
    },
    {
      "column_type": "field",
      "type": "unknown",
      "field": "http_response_code",
      "name": "field: http_response_code"
    },
    {
      "column_type": "field",
      "type": "numeric",
      "field": "took_ms",
      "name": "field: took_ms"
    }
  ],
  "datarows": [
    [
      "example.org",
      "GET",
      200,
      50
    ],
    [
      "example.org",
      "GET",
      200,
      50
    ],
    [
      "example.org",
      "GET",
      200,
      55
    ],
    [
      "example.org",
      "GET",
      200,
      63
    ],
    [
      "example.org",
      "GET",
      200,
      63
    ],
    [
      "example.org",
      "GET",
      200,
      63
    ],
    [
      "example.org",
      "GET",
      200,
      37
    ],
    [
      "example.org",
      "GET",
      200,
      46
    ],
    [
      "example.org",
      "GET",
      200,
      59
    ],
    [
      "example.org",
      "GET",
      200,
      52
    ]
  ],
  "metadata": {
    "effective_timerange": {
      "from": "2023-04-05T09:08:23.193Z",
      "to": "2023-04-06T09:08:23.193Z",
      "type": "absolute"
    }
  }
} 

Run Aggregations with the Search Scripting API

Use the aggregations endpoint when you need summarized search results instead of individual messages. Aggregations support multiple groupings and metrics, letting you analyze patterns such as response counts, minimum values, maximum values, and averages across selected fields.

1. Send a GET request for a basic aggregation, or use a POST request for more advanced configuration options:

   GET https://example.org/api/search/aggregate?groups=http_method&metrics=avg:took_ms&metrics=count

   POST https://example.org/api/search/aggregate

2. If you use a POST request, define the aggregation in the request body, as in this example:

   Copy

   {
       "query": "source:example.org",
       "streams": [
           "620f890b70fb980467aca611"
       ],
       "timerange": {
           "type": "keyword",
           "keyword": "last five minutes"
       },
       "group_by": [
           {
               "field": "http_method"
           },
           {
               "field": "http_response_code",
               "limit": 2
           }
       ],
       "metrics": [
           {
               "function": "min",
               "field": "took_ms"
           },
           {
               "function": "max",
               "field": "took_ms",
               "sort": "desc"
           }
       ]
   }

3. Define metrics using function:field_name pairs. In POST requests, you can also define sorting behavior for selected metrics.

4. Review the response format returned by the Accept header you selected.

GET and POST requests return the same aggregation data, but POST requests support additional configuration such as grouping limits and sorting. This makes POST requests better suited for more complex scripted workflows.

The following metric functions are available: average, count, latest, max, min, percentile, stdDev, sum, sumOfSquares, and variance.

If you use the percentile metric, you can add configuration details to the metric object.

{
        "function": "percentile",
        "field": "took_ms",
        "configuration":
        {"percentile": 90}
}

Example Aggregation Responses

The following examples show how the same aggregation query is returned in plain text, CSV, and JSON formats.

This request sets the Accept header to text/plain, returning the aggregation results as a formatted table in the terminal:

➜  curl  -H 'Accept: text/plain' -H 'X-Requested-By: cli'  -u username:password 'https://example.org/api/search/aggregate?groups=http_method&metrics=avg:took_ms&metrics=count'
┌────────────────────────┬────────────────────────┬───────────────────────┐
│grouping: http_method   │metric: avg(took_ms)    │metric: count()        │
├────────────────────────┼────────────────────────┼───────────────────────┤
│GET                     │84.59519448510214       │3326553                │
│DELETE                  │96.02973055642204       │330199                 │
│POST                    │170.11398232130765      │329323                 │
│PUT                     │132.19959919839678      │154690                 │
└────────────────────────┴────────────────────────┴───────────────────────┘

This request sets the Accept header to text/csv, returning the aggregation results in CSV format.

➜  curl  -H 'Accept: text/csv' -H 'X-Requested-By: cli'  -u username:password 'https://example.org/api/search/aggregate?groups=http_method&metrics=avg:took_ms&metrics=count'
"grouping: http_method","metric: avg(took_ms)","metric: count()"
"GET","84.58997204888469","3326522"
"DELETE","96.02450487553752","330220"
"POST","170.08947537817832","329342"
"PUT","132.2030966157029","154685"

This request sets the Accept header to application/json, returning the aggregation results as structured JSON with schema and metadata information.

➜ curl  -H 'Accept: application/json' -H 'X-Requested-By: cli'  -u username:password 'https://example.org/api/search/aggregate?groups=http_method&metrics=avg:took_ms&metrics=count'
{
  "schema": [
    {
      "column_type": "grouping",
      "type": "string",
      "field": "http_method",
      "name": "grouping: http_method"
    },
    {
      "column_type": "metric",
      "type": "numeric",
      "function": "avg",
      "field": "took_ms",
      "name": "metric: avg(took_ms)"
    },
    {
      "column_type": "metric",
      "type": "numeric",
      "function": "count",
      "name": "metric: count()"
    }
  ],
  "datarows": [
    [
      "GET",
      84.6043288978779,
      3326528
    ],
    [
      "DELETE",
      96.02355285311111,
      330236
    ],
    [
      "POST",
      170.09012904205252,
      329350
    ],
    [
      "PUT",
      132.20231038249952,
      154693
    ]
  ],
  "metadata": {
    "effective_timerange": {
      "from": "2023-04-05T09:14:39.731Z",
      "to": "2023-04-06T09:14:39.731Z",
      "type": "absolute"
    }
  }
}

Group Results by Date Fields

When you include a date-type field in a group_by clause, the search scripting API automatically groups results into time-based intervals. This allows you to analyze how values change over time without manually creating time buckets in your scripts.

This behavior works the same way as time interval grouping in the Graylog interface. For example, when a date field such as timestamp is used in an aggregation widget's Group By section, Graylog automatically groups results into time intervals.

1. Add a date field to the group_by object in your aggregation request.

2. Use timeunit if you want to define a specific time interval for grouping results. Supported suffixes are s (seconds), m (minutes), h (hours), d (days), w (weeks), M (months), and y (years). For example, 10m groups results into 10-minute buckets.

   Copy

       "group_by": [
           {
               "field": "timestamp",
               "timeunit": string
           }
       ],

   Or, use scaling if you want Graylog to determine the time interval automatically and adjust it using a floating-point scaling factor. This is useful when you want the interval size to adapt to the selected time range rather than specifying a fixed interval. If you do not specify either parameter when grouping by a date field, Graylog uses the default scaling value of 1.0.

   Copy

       "group_by": [
           {
               "field": "timestamp",
               "scaling": double
           }
       ],

3. Run the request and review the grouped output.

Example Date Field Grouping Requests

The following examples demonstrate how date fields can be used in a group_by clause to create time-based aggregations.

This example groups results by the timestamp field using the timeunit parameter. The request buckets messages into one-hour intervals and returns the total count of messages in each time bucket.

{
    "query": "source:example.org",
    "timerange": {
        "type": "relative",
        "range": 0
    },
    "group_by": [
        {
            "field": "timestamp",
            "timeunit": "1h"
        }
    ],
    "metrics": [
        {
            "function": "count"
        }
    ]
}

Further Reading

Explore the following additional resources and recommended readings to expand your knowledge on related topics: