Graylog 5.1 introduces a new index rotation strategy for index sets. This strategy aims to achieve optimal resource usage (i.e. shards that are neither too small nor too big) while avoiding an excessive number of shards even as the ingest rate varies. It combines the benefits of the Index Time strategy, which allows you to select a specific number of days before the active write index is rotated, with the additional ability to restrict the size an index may reach before rotation, or the Index Size strategy.
When selected this strategy will attempt to rotate the desired index daily. It can, however, skip this rotation in order to achieve optimal sized indices by maintaining the shard size between 20 and 50 GB. The delay of the rotation occurs within a specific range of time configured by the user. So, if the index reaches the maximum amount of time allotted, it will be rotated regardless of its size, meaning that an index will never exceed its maximum lifetime parameter without rotation.
Index Time Size Optimizing Configuration
To enable this strategy for a new index set, select the following parameters when configuring a new set from the Indices & Index Sets menu in Graylog:
-
Index Rotation Configuration
-
Select rotation strategy: Index Time Size Optimizing
-
-
Index Retention Configuration
-
Select retention strategy:
- Archive: Automatically archive an index before closing or deleting it. (This feature is only available with Graylog Operations; see Archiving for more details.)
- Delete: Delete indices in Elasticsearch/OpenSearch to minimize resource consumption.
- Close: Close indices in Elasticsearch/OpenSearch to reduce resource consumption.
- Do nothing
-
Lifetime in days: Describes the minimum and maximum number of days that data is kept before the index is retained.
-
You may also opt to utilize this strategy for existing index sets. To do so select the specific index set from the Indices and Index Sets menu and click Edit. Now you can make your desired selections as described above.
Use Case Scenario
Let's look at an example of how this strategy might be implemented.
In this example the user has opted to implement Index Time Size Optimizing for this specific index set. Either depending on preferences or organizational requirements, they have determined that the minimum and maximum lifetime for indices in this set are 30 and 40 days respectively.
Now, Graylog will begin the process of determining whether or not to rotate the index depending on its size, and it will attempt to perform this process daily. As long as the maximum size (50 GB) is not reached, Graylog will not rotate index before the 30-day minimum has passed. And, as the user has set a 40-day maximum, Graylog will not allow the active write index to exceed a 40-day lifetime without being retained regardless of size. Therefore, this example configuration allows Graylog a 10-day period to determine the optimal time line for rotation while maintaining the shard size between 20 and 50 GB. While a specific day is not guaranteed for rotation to occur, the index will optimally be retained sometime between day 30 and day 40 after creation.
It is important to note that the larger the period that exists between the minimum and maximum lifetime, the more optimal shard size can be achieved. In this example the period allotted for optimization is 10 days; however, this can be adjusted based on a number of environmental/organizational factors and must be determined by the user.
Server.conf Configuration Parameters
New configuration parameters have been created for this index strategy and can be adjusted by administrators in the Graylog server.conf file. These parameters and their defaults are as follows:
-
time_size_optimizing_rotation_min_shard_size=20gIndex should be at least this big before rotating.
-
time_size_optimizing_rotation_max_shard_size=50gRotate if index exceeds this size limit.
