Metrics
Graylog collects metrics throughout its operation and stores them in memory on each Graylog
node. The metrics include event counts, timers and statistics for many parts of the Graylog application and its
subsystems. Standard metrics are viewable within Graylog on the System > Nodes page through the
Metrics
button.
Standard metrics can also be queried through the Graylog REST API Browser.
Export via Prometheus Metric Exporting
Metrics can be exported to other systems via the
Prometheus Metrics Exporting feature. This feature can be optionally enabled. When enabled, Graylog will export
metrics via a standard Prometheus https
exporter on port 9833
. Prometheus can then scrape and ingest the metrics.
Configuration
To begin exporting Prometheus metrics, enable it with the following configuration property
in the server.conf
file.
prometheus_exporter_enabled = true
Once enabled, metrics are exported in the standard export format on the following URI by default.
http://127.0.0.1:9833/api/metrics/prometheus
To specify a custom export hostname or IP address, the following configuration can be used. We suggest leaving the default export port , since it is already registered with Prometheus.
prometheus_exporter_bind_address = 10.0.0.1:9090
Depending on the Prometheus scrape_config setting, an explicit scrape target might need to be specified in your Prometheus target hosts configuration.
Default Core Metric Mappings
A pre-defined core set of Graylog-to-Prometheus metric mappings are enabled by default.
All metric names are
prefixed with gl_...
to clearly indicate that the metrics originated from
Graylog. For example, the metric name
gl_input_throughput
.
All metrics are automatically assigned a
node
metric label,
which contains the Graylog Node ID where the metric originated from. This can be useful for visualizing the same
metric across Graylog
nodes.
Many
metrics are exported with a consolidated name, which allows breakouts by labels to allow easier visualization of
metrics.
See the pre-defined metric mappings for a full list of metric mappings.
Customized Metric Mappings
We understand that metric mapping requirements can be very specific the each per-customer use case, so we provide the ability to override the default core mappings. We also support the ability to specify any additional needed mappings.
Graylog metrics exports heavily utilize Prometheus labels to help you effectively visualize and analyze your metrics data.
To completely replace the default core Prometheus
mappings, provide the path to a core replacement mapping file
file
containing just the desired mappings with the following configuration property. This file is monitored for changes
at runtime, so you can customize as needed without restarting your Graylog server. The file path is interpreted
relative to the Graylog server working directory. Absolute paths are also supported.YML
prometheus_exporter_mapping_file_path_core = prometheus-exporter-mapping-core.yml
To provide mappings in addition to the core defaults, provide the path to an additional mapping file
file containing the additional desired mappings with the following
configuration property. This file is also monitored for changes at runtime.yaml
prometheus_exporter_mapping_file_path_custom = prometheus-exporter-mapping-custom.yml
Custom Mappings Format Example 1
This example mapping produces the gl_stream_incoming_messages
metric in Prometheus, which shows the number of messages received by for each stream in Graylog.
The standard metric name in Graylog contains the ID of the stream, and one distinct metric is recorded for each stream.
For example:
metric_mappings:
-
match_pattern: org.graylog2.plugin.streams.Stream.*.incomingMessages
metric_name: stream_incoming_messages
wildcard_extract_labels:
- stream
The mapping definition that follows provides a match_pattern
with a wildcard
*
for the stream-id
, which provides one label for
each stream id automatically. This allows for the visualization of messages received for all streams together, but
broken-out by stream-id via the labels functionality. The wildcard_extract_labels
is
an ordered array, which provides the label names for any specified wildcards in the match_pattern
.
Mapping Definition:
- metric_name : "stream_incoming_messages" match_pattern : "org.graylog2.plugin.streams.Stream.*.incomingMessages" wildcard_extract_labels : - "stream"
Custom Mappings Example 2
This example mapping produces the
metric in Prometheus, which provides the current usage state of
the Graylog gl_buffer_usage
input
, process
, and
output
buffers.
Note that three separate mappings are provided to
consolidate three independent buffer Graylog metrics into a single Prometheus metric with three unique labels
(input
, process
,
output
) that correspond to each Graylog metric.
The additional_labels
property allows for the assignment of an explicit label corresponding to the metric defined in the
match_pattern
.
Mapping Definition:
metric_mappings:
- metric_name: buffer_usage
match_pattern: org.graylog2.buffers.input.usage
additional_labels:
type: input
- metric_name: buffer_usage
match_pattern: org.graylog2.buffers.output.usage
additional_labels:
type: output
- metric_name: buffer_usage
match_pattern: org.graylog2.buffers.process.usage
additional_labels:
type: process
Custom Mappings Refresh Interval
By default, custom and core mapping files are refreshed (re-read from disk)
every 30 seconds. You can override this with a custom duration if desired. Use the standard Graylog duration
notation (eg 60s
,
or
m5
1h
).
prometheus_exporter_mapping_file_refresh_interval = 5m