Index Model

Graylog transparently manages one or more sets of indices to optimize search and analysis operations for speed and low resource consumption.

To enable managing indices with different mappings, analyzers, and replication settings, Graylog uses index sets, which are an abstraction of all these settings. For information on creating index default configuration templates, see Index Set Templates.

Index Sets

Each index set contains the necessary settings for Graylog to create, manage, and fill search backend indices and handle index rotation and data retention depending on your specific requirements or needs.

Graylog is maintaining an index alias per index set that is always pointing to the current write-active index from that index set. There is always exactly one index to which new messages are written until the configured rotation criterion (number of documents, index size, or index age) has been met.

A background task continuously checks if the rotation criterion of an index set has been met, and a new index is created and prepared when that happens. Once the index is ready, the index alias is automatically switched to it. That means that all Graylog nodes can write messages into the alias without even knowing what the current write-active index of the index set is.

Almost every read operation is performed with a given time range. Because Graylog writes messages sequentially into search backend, it can keep information about the time range each index covers. It selects a list of indices to query when having a time range is provided. If no time range was provided, it will search in all known indices.

Eviction of Indices and Messages

There are configuration settings for the maximum number of indices Graylog is managing in a given index set.

Depending on the configured retention strategy, the oldest indices of an index set will automatically be closed, deleted, or exported when the configured maximum number of indices has been reached.

The deletion is performed by the Graylog leader node in a background thread, which is continuously comparing the number of indices against the configured maximum:

Copy 
INFO : org.graylog2.indexer.rotation.strategies.AbstractRotationStrategy - Deflector index <graylog_95> should be rotated, Pointing deflector to new index now!
INFO : org.graylog2.indexer.MongoIndexSet - Cycling from <graylog_95> to <graylog_96>.
INFO : org.graylog2.indexer.MongoIndexSet - Creating target index <graylog_96>.
INFO : org.graylog2.indexer.indices.Indices - Created Graylog index template "graylog-internal" in Elasticsearch.
INFO : org.graylog2.indexer.MongoIndexSet - Waiting for allocation of index <graylog_96>.
INFO : org.graylog2.indexer.MongoIndexSet - Index <graylog_96> has been successfully allocated.
INFO : org.graylog2.indexer.MongoIndexSet - Pointing index alias <graylog_deflector> to new index <graylog_96>.
INFO : org.graylog2.system.jobs.SystemJobManager - Submitted SystemJob <f1018ae0-dcaa-11e6-97c3-6c4008b8fc28> [org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob]
INFO : org.graylog2.indexer.MongoIndexSet - Successfully pointed index alias <graylog_deflector> to index <graylog_96>.

Index Set Configuration

There are a variety of options for index sets related to how Graylog will store messages into the search backendcluster. These sets can be managed on the Indices & Index Sets page by navigating to System > Indices.

To create a new set, select the Create index set button at the top right of the page. From here complete the following fields to create the index set:

  • Title: A descriptive name of the index set.
  • Description: A description of the index set for human consumption.
  • Index prefix: A unique prefix used for search backendindices managed by the index set. The prefix must start with a letter or number, and can only contain letters, numbers, _, -and +. The index alias will be named accordingly, e. g. graylog_deflector if the index prefix was graylog.
  • Analyzer: (default: standard) The search backend analyzer for the index set.
  • Index shards: (default: 1) The number of search backend shards used per index.
  • Index replicas: (default: 0) The number of search backend replicas used per index.
  • Max. number of segments: (default: 1) The maximum number of segments per search backend index after index optimization (force merge). (See Segment Merging for details.)
  • Disable index optimization after rotation: Disable search backend index optimization (force merge) after index rotation. Only activate this if you have serious problems with the performance of your search backend cluster during the optimization process.
  • Field type refresh interval: Determines how often the field type information for the active write index will be updated in increments of seconds or minutes.

Index Rotation Configuration

You must also determine the index rotation strategy for this index set. By default, Data Tiering is selected under Rotation & Retention, and this is the recommended option. See Data Tiering for complete information about this method and the different configuration options.

For other indexing methods, select Legacy (Deprecated), then choose from the following rotation options:

  • Index Message Count: Rotates the index after a specific number of messages have been written.

    Hint: Index Message Count is a legacy feature that is not available by default on newer installations.

  • Index Size: Rotates the index after an approximate size on the disk (before optimization) has been reached.
  • Index Time: Rotates the index after a specific time interval (e. g. 1 hour or 1 week).
  • Index Time Size Optimizing: A flexible strategy that combines a time-based approach with the goal of maintaining an optimal shard size. (See Index Time Size Optimizing for more information on this strategy.) This strategy is implemented by default to all new index sets.

Index Retention Configuration

Retention strategy is configured under the Legacy (Deprecated) option for index set configuration. Select your strategy for index set retention:

  • Archive: Automatically archive an index before closing or deleting it. (This feature is only available with Graylog Enterprise; see Archiving for more details.)
  • Delete: Delete indices in search backend to minimize resource consumption.

Maintenance

Keeping the Index Ranges in Sync

Graylog will take care of calculating index ranges automatically as soon as a new index has been created.

In case the stored metadata about index time ranges has run out of sync, Graylog will notify you in the web interface. This can happen if an index was deleted manually or messages from already “closed” indices were removed.

The system will offer to re-generate all time range information.

You can easily re-build the information yourself after manually deleting indices or doing other changes that might cause synchronization problems:

Copy 
$ curl -XPOST http://127.0.0.1:9000/api/system/indices/ranges/rebuild

This will trigger a system job:

Copy 
INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Recalculating index ranges.
INFO : org.graylog2.system.jobs.SystemJobManager - Submitted SystemJob <9b64a9d0-dcac-11e6-97c3-6c4008b8fc28> [org.graylog2.indexer.ranges.RebuildIndexRangesJob]
INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Recalculating index ranges for index set Default index set (graylog2_*): 5 indices affected.
INFO : org.graylog2.indexer.ranges.MongoIndexRangeService - Calculated range of [graylog_96] in [7ms].
INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Created ranges for index graylog_96: MongoIndexRange{id=null, indexName=graylog_96, begin=2017-01-17T11:49:02.529Z, end=2017-01-17T12:00:01.492Z, calculatedAt=2017-01-17T12:00:58.097Z, calculationDuration=7, streamIds=[000000000000000000000001]}
[...]
INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Done calculating index ranges for 5 indices. Took 44ms.
INFO : org.graylog2.system.jobs.SystemJobManager - SystemJob <9b64a9d0-dcac-11e6-97c3-6c4008b8fc28> [org.graylog2.indexer.ranges.RebuildIndexRangesJob] finished in 46ms.

Manually Rotating the Active Write Index

Sometimes you might want to rotate the active write index manually and not wait until the configured rotation criterion for the latest index has been met, if for example, you’ve changed the index mapping or the number of shards per index.

You can do this either via an HTTP request against the REST API of the Graylog leader node or via the web interface:

Copy 
$ curl -XPOST http://127.0.0.1:9000/api/system/deflector/cycle

A manual rotation produces logs similar to the following:

Copy 
INFO : org.graylog2.rest.resources.system.DeflectorResource - Cycling deflector for index set <58501f0b4a133077ecd134d9>. Reason: REST request.
INFO : org.graylog2.indexer.MongoIndexSet - Cycling from <graylog_97> to <graylog_98>.
INFO : org.graylog2.indexer.MongoIndexSet - Creating target index <graylog_98>.
INFO : org.graylog2.indexer.indices.Indices - Created Graylog index template "graylog-internal" in Elasticsearch.
INFO : org.graylog2.indexer.MongoIndexSet - Waiting for allocation of index <graylog_98>.
INFO : org.graylog2.indexer.MongoIndexSet - Index <graylog_98> has been successfully allocated.
INFO : org.graylog2.indexer.MongoIndexSet - Pointing index alias <graylog_deflector> to new index <graylog_98>.
INFO : org.graylog2.system.jobs.SystemJobManager - Submitted SystemJob <024aac80-dcad-11e6-97c3-6c4008b8fc28> [org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob]
INFO : org.graylog2.indexer.MongoIndexSet - Successfully pointed index alias <graylog_deflector> to index <graylog_98>.
INFO : org.graylog2.indexer.retention.strategies.AbstractIndexCountBasedRetentionStrategy - Number of indices (5) higher than limit (4). Running retention for 1 index.
INFO : org.graylog2.indexer.retention.strategies.AbstractIndexCountBasedRetentionStrategy - Running retention strategy [org.graylog2.indexer.retention.strategies.DeletionRetentionStrategy] for index <graylog_94>
INFO : org.graylog2.indexer.retention.strategies.DeletionRetentionStrategy - Finished index retention strategy [delete] for index <graylog_94> in 23ms.