Index Time Size Optimizing

Hint: The Index Time Size Optimizing strategy is part of the Legacy configuration options for index sets in Graylog Enterprise. Data Tiering is selected by default and is the recommended strategy for index rotation and retention. To enable Index Time Size Optimizing, as well as additional options, select the Legacy option.

Graylog includes the Index Time Size Optimizing index rotation strategy for index sets. This strategy aims to achieve optimal resource usage (that is, shards that are neither too small nor too big) while avoiding an excessive number of shards even as the ingest rate varies. It combines the benefits of the Index Time strategy, which allows you to select a specific number of days before the active write index is rotated, with the additional ability to restrict the size an index may reach before rotation, or the Index Size strategy.

When you select this strategy, Graylog attempts to rotate the desired index daily. It can, however, skip this rotation in order to achieve optimal sized indices by maintaining the shard size within an acceptable range. The delay of the rotation occurs within a range of time that you configure. So, if the index reaches the maximum amount of time allotted, it is rotated regardless of its size, meaning that an index never exceeds its maximum lifetime parameter without rotation.

Hint: For more information about configuring index set defaults, see the related Graylog documentation.

Index Time Size Optimizing Configuration

To enable this strategy for a new index set, select the following parameters when configuring a new set from the Indices & Index Sets menu in Graylog:

  • Index Rotation Configuration

    • Select rotation strategy: Index Time Size Optimizing

  • Index Retention Configuration

    • Select retention strategy:

      • Archive: Automatically archive an index before closing or deleting it. (This feature is only available with Graylog Enterprise; see Archiving for more details.)
      • Delete: Delete indices in Elasticsearch/OpenSearch to minimize resource consumption.

    • Lifetime in days: Describes the minimum and maximum number of days that data is kept before the index is retained.

You may also opt to utilize this strategy for existing index sets. To do so select the specific index set from the Indices and Index Sets menu and click Edit. Now you can make your desired selections as described above.

Use Case Scenario

Let's look at an example of how this strategy might be implemented.

In this example the user has opted to implement Index Time Size Optimizing for this specific index set. Either depending on preferences or organizational requirements, they have determined that the minimum and maximum lifetime for indices in this set are 30 and 40 days respectively. Also, the time_size_optimizing_rotation_max_shard_size property in server.conf has been set to 50g.

Now, Graylog begins the process of determining whether to rotate the index depending on its size, and it attempts to perform this process daily. As long as the maximum size (50 GB) is not reached, Graylog does not rotate the index before the 30-day minimum has passed. And, as the user has set a 40-day maximum, Graylog does not allow the active write index to exceed a 40-day lifetime without being retained, regardless of size.

Therefore, this example configuration allows Graylog a 10-day period to determine the optimal timeline for rotation while maintaining the shard size between 20 GB and 50 GB. While a specific day is not guaranteed for rotation to occur, the index is optimally rotated sometime between day 30 and day 40 after creation.

Hint: It is possible for Graylog administrators to adjust the minimum and maximum shard size parameters if necessary (as detailed in the following section).

In this example, the period allotted for optimization is 10 days. However, the time period can be adjusted based on a number of environmental or organizational factors and must be determined by the user.

Warning: When selecting the Index Time Size Optimizing strategy for an existing index set, or when adjusting the minimum and maximum lifetime parameters, be aware that these changes only apply to newly created indices within this index set.

Server.conf Configuration Parameters

The configuration parameters for this index strategy can be adjusted by administrators in the Graylog server.conf file. These parameters and their defaults are as follows:

  • time_size_optimizing_rotation_min_shard_size=20g

    Index should be at least this big before rotating.

  • time_size_optimizing_rotation_max_shard_size=20g

    Rotate if index exceeds this size limit.

By default, the minimum and maximum shard sizes are set the same, which allows indices to rotate at staggered intervals rather than multiple indices rotating at the same time of day. Note that this setting is optimized for recommended installation configurations and might need to be adjusted for other setups.