Manage Warm Tier Repositories

Before you can move data to the warm tier, you must first set up a storage repository. This article explains how to create, view, and manage credentials for warm tier repositories.

Hint: Snapshot Archiving also reads from these same repositories. See Archiving for more information.

Prerequisites

Before proceeding, ensure that the following prerequisites are met:

You must have an Enterprise license.
You must be a Graylog administrator.
At least one node in your search cluster must have the search role assigned. For details, see Create a Warm Tier on Data Node or Create a Warm Tier on Self-Managed OpenSearch, depending on your installation.
Configure your storage backend outside the Graylog web interface, following Create a Warm Tier on Data Node or Create a Warm Tier on Self-Managed OpenSearch for your installation type. A backend appears as a selectable option here only after it is configured.

View Warm Tier Repositories

The Warm Tier Repositories tab at System > Indices is where you create and manage backend storage for warm tier. The tiles at the top provide useful information about system readiness as well as defined backends. If you have already configured backend storage, a table lists every repository, its type, location, and credential status.

If your cluster isn't ready for the warm tier yet or no backends have been defined, this page lists the remaining setup steps instead of showing the repository table.

Create a Repository

To create a repository:

Select Create repository.
Select a Warm Storage Repository Location. Only backend types already configured on your search cluster are listed.
For File System ▼
1. Enter a unique Name.
2. Select the file system Location from the dropdown.
3. Select Create.
For Amazon S3, OVH Object Storage, or SysEleven Object Storage ▼
1. Enter a unique Name.
2. Enter the Bucket in which snapshots will be stored.
  
  If using Amazon S3, note the following:
  
  Warning: Graylog does not support retrieval of data from the Glacier tier of AWS S3. For S3 backends, make sure the S3 bucket does not have lifecycle rules that move files into the Glacier tier.
3. Enter the required Base Path where snapshots should be stored within the bucket. You can use a single bucket for multiple purposes, for example the same bucket for a Data Lake backend and a warm tier snapshot backend, but use a different base path for each.
  
  Warning: Although you can use a bucket for multiple purposes, do not place multiple warm tier repositories in the same base path. You can create multiple warm tier repositories, but each should have a separate base path.
4. (Optional) Enter an S3 Endpoint URL to override the default endpoint.
5. Select an AWS Region, or type a custom region. A region is required for OVH Object Storage and SysEleven Object Storage; optional for Amazon S3.
6. Select an AWS Authentication Type, or leave the default of Automatic.
7. For Amazon S3 only, optionally enter an AWS Assume Role (ARN) and AWS External ID. These fields are not shown for OVH Object Storage or SysEleven Object Storage.
8. Select Create.
For Google Cloud Storage ▼
Warning: If you use self-managed OpenSearch, GCS repositories do not work with versions 3.3 through 3.7. To use GCS, run a supported version earlier than 3.3, or migrate to Data Node.
1. Enter a unique Name.
2. Enter the Bucket and Base Path where snapshots are stored. You can use a single bucket for multiple purposes, for example the same bucket for a Data Lake backend and a warm tier snapshot backend, but use a different base path for each.
  
  Warning: Although you can use a bucket for multiple purposes, do not place multiple warm tier repositories in the same base path. You can create multiple warm tier repositories, but each should have a separate base path.
3. Select Create.
For Azure Blob Storage ▼
1. Enter a unique Name.
2. Enter the Container and Base Path where snapshots are stored. You can use a single container for multiple purposes, for example the same container for a Data Lake backend and a warm tier snapshot backend, but use a different base path for each.
  
  Warning: Although you can use a container for multiple purposes, do not place multiple warm tier repositories in the same base path. You can create multiple warm tier repositories, but each should have a separate base path.
3. Select Create.
For Apache HDFS ▼
1. Enter a unique Name.
2. Enter the Uri of the nameserver.
3. In the HDFS cluster, create a directory for the repository. This directory must be owned by opensearch:superuser and have write permissions.
4. Enter the Path of that directory.
5. Leave Load defaults selected unless you need custom Hadoop configuration properties.
6. Select Create.

Edit Repository Credentials

Credential management through this page is available only for Amazon S3, OVH Object Storage, and SysEleven Object Storage repositories. To edit credentials:

Select More > Edit Credentials for the repository.
Enter the access key and secret, or select automatic authentication.
Select Save.

Important: Each repository needs its own credentials. Entering credentials for one S3-type repository does not apply them to any other.

For every other backend type, and for S3-type repositories used only by the warm tier, credentials still come from your OpenSearch or Data Node configuration.

Delete a Repository

To remove a repository you no longer need, select More > Delete for that repository. You must confirm this action in the dialog, and you cannot undo this action.