Index Set Templates

An index set template is a collection of preset configurations that dictate how an index set is created, used, and maintained. These configurations play a crucial role in defining how indices are structured and managed, affecting their efficiency, cost of data storage, data retrieval, and overall performance of your Graylog service.

Graylog provides three built-in index set templates that utilize data tiering to optimize your indexing preferences based on different requirements or use cases and also allows you create custom index set templates tailored to your environment. For example, an index set configuration used for compliance data might only require a template with a small hot tier since it is rarely searched, while a template appropriate for data that is regularly searched might instead have a large hot tier.

With the index set templates, you can:

Prerequisites

  • A valid Graylog Enterprise license is required to use a warm tier in index set templates.

  • You must utilize Graylog with OpenSearch 2.12+ or with Graylog Data Node.

Built-In Index Set Templates

When creating an index set, Graylog allows you to select built-in index set templates. These templates visually represent how long your data is securely stored in each tier (hot, warm, or cold) depending on your selections. Each template includes a descriptive use case suited to that profile, which can help you choose the appropriate index set template for your use.

There are three built-in index set templates:

  • 7 days Hot, 90 Days Total: This template has the lowest specification requirement for the search backend and uses the least hot tier storage. It is best suited for data that is rarely the subject of historical searches. This profile is pre-configured to store your data in a hot tier for 7 days with a minimum of 90 days in warm tier storage before deletion.

  • 14 days Hot, 90 Days Total: This template is pre-configured to store your data in a hot tier for 14 days with a minimum of 90 days in warm tier storage before deletion. This template is ideal in situations where data must be readily available for a short duration, followed by cost-effective storage needed for compliance or further analysis.

  • 30 days Hot, 90 Days Total: This template has the highest specification requirement for the search backend and uses the most hot tier storage. This template is ideal for high-value data that requires monitoring and is often utilized for in-depth historical analysis.

Apply a Built-In Index Set Template

To create an index set using a built-in index set template:

  1. Navigate to System >Indices and click the Create index set button.

  2. Select a template from the Built-in Templates displayed in the window and click the Apply template button. When the selected template is applied, the Create Index Set page is displayed with the fields pre-filled with the template values.

    Hint: The Warm Tier (Enterprise) check box is selected by default for Enterprise users but can be cleared if you choose to disable this feature.

  3. Fill out the following index set configuration fields:

    • Title: A descriptive name of the index set.

    • Description: A brief explanation of the index set's purpose.

    • Index prefix: A unique prefix used for indices managed by the index set. The prefix must start with a letter or number and can only contain letters, numbers, _, - and +. The index alias will be named accordingly, e. g. graylog_deflector if the index prefix was graylog.

  4. In the Rotation & Retention section, click the Create new warm storage repository button.

  5. In the resulting window, assign a unique name to your repository, select a location from the drop-down menu, and click Create. The available locations in the drop-down menu are detected from your search backendconfiguration file. Ensure you create at least one storage repository for saving snapshots. (Follow the instructions on preparing your environment for a warm tier for detailed instructions on creating a warm tier repository.)

  6. Click the Create index set button located at the bottom of the Create Index Set page to complete the index set creation using the selected built-in template.

Create Custom Templates

In situations where you have specific requirements for your environment beyond what is provided by the built-in templates, you have the option to create custom templates. These templates can be selected and applied when creating a new index set. A custom template is useful when you want to size your indices to match your hardware, address retention requirements not covered by the built-in templates, and configure advanced properties of index sets, such as shards and replica shards.

To create a custom template:

  1. Navigate to System >Indices and click on the Index Set Templates tab.

  2. Click on the Create template button

  3. On the resulting Create Index Set Template page, provide a template title and description. Adjust pre-filled configuration values to your custom settings.

  4. Navigate to the Rotation & Retention section and click the Create new warm storage repository button.

  5. In the resulting window, assign a unique name to your repository, select a location from the drop-down menu, and click Create. The available locations in the drop-down menu are detected from your search backendconfiguration file. Create a storage repository for saving snapshots.

  6. Click on the Create template button. Your new custom template has been created and will appear in the list of available index set templates when creating a new index set.

Create a Default Index Set Template

The default configuration for index sets defines how new index sets are created in Graylog. When a default template is configured, it is globally applied during index set creation as the configuration values in the default template are pre-filled and populated on the index set creation form. A default template is provided in the list of index set templates, which you can either use as-is or modify according to your requirements.

To configure a default index set template, follow these steps below:

  1. Navigate to System >Indices and click on the Index Set Templates tab.

  2. From the list of index set templates, a default template is provided. Click on the Edit button to modify the pre-filled values in the default template to create a custom default template then save by clicking the Update template button or use the default template provided.

  3. To apply the created default template, click on Indices & Index Sets > Create index set.

  4. Close the pop-up window that appears, which displays the built-in and custom index set templates. The configuration values from your default template are then pre-filled on the Create Index Set page.

  5. Alternatively, you can also toggle to Custom Templates on the Index Set Templates window, and select your custom default template from the dropdown menu.