Archiving

The following article exclusively pertains to a Graylog Enterprise feature or functionality. To learn more about obtaining an Enterprise license, please contact the Graylog Sales team.

Storing extensive amounts of data in your search backend can be costly. Graylog allows you to store inactive data in the Graylog archive to help lower storage costs and maximize retention as needed. When archiving index sets, you can set a retention period based on a variety of factors. Archived indices are then deleted after the retention period is complete.

Graylog provides multiple types of backend storage for you to choose from, each with its own setup and configuration requirements. See Choose a Backend below for details.

Archived index sets are stored before retention cleaning begins so that no data is lost. In a multi-node Graylog deployment, only the leader node writes archive files.

Hint: Archived data is inactive and cannot be searched unless restored. If necessary, archived indices can be re-imported through the user interface. After the data is restored, you can search through and analyze this data via the web interface. See Restore an Archive for more information.

Prerequisites

Before proceeding, ensure that the following prerequisites are met:

  • You must have an Enterprise license.

  • You must be a Graylog administrator to configure archiving.

  • You must have appropriate credentials to configure your chosen storage backend.

  • To use the Snapshot strategy, at least one node in your search cluster must have the search role assigned.

Configure the Graylog Archive

You can configure the archive directly via the Graylog web interface:

  1. Navigate to Enterprise > Archives

  2. Select the Configuration tab.

  3. Select a Backend from the dropdown. If you haven't defined a backend, go to Choose a Backend to see the available backend types and how to configure them. You must create a backend before it is available to select for configuration.

  4. Select the Archiving Strategy.

  5. Enter additional configuration details.

  6. Select Update configuration.

Choose a Backend

Archived indices are stored in a backend. Choose a backend type based on your environment or preference:

File system is the default backend. Initial server start up triggers the creation of a backend, but you can adjust the backend location in your Graylog configuration .yml file.

You can use a backend to store archived data. Graylog supports only a single-file system backend type.

The archived indices are stored in the Output base path directory. This directory needs to exist and be writable for the Graylog server process to store the files.

The archiving process runs on the leader node, so only the leader node needs access to the output base path directory. We recommend housing the output base path directory on a separate disk or partition to avoid any negative impacts on message processing when the archive completely fills a disk.

You can edit archive backend configuration options via Enterprise > Archives under Manage Backends.

File System Backend Configuration Options

To configure a file system backend on your local network storage:

  1. Navigate to Enterprise > Archives, then select the Manage Backends tab.

  2. Click Create backend.

  3. Select File system from the Backend Type dropdown.

  4. Enter configuration details for your network storage backend:

    Title

    Enter a unique and descriptive name for the backend.

    Description

    Enter a description of the backend.

    Output Base Path

    Enter the base path where the archives should be stored. Use a simple directory path string, or you can create a template string to build dynamic paths, as described in Template Strings.

  5. Click Create backend.

  6. Proceed to Configure the Graylog Archive.

The S3 archiving backend is built to work with AWS and can be used to upload archives to an Amazon S3 object storage service. The S3 backend option is also compatible with other object storage implementations, such as MinIO, CEPH, and Digital Ocean Spaces.

Graylog explicitly supports two S3-compatible options, OVH Object Storage and SysEleven Object Storage, which are both described separately.

S3 Backend Configuration Options

To configure an S3 backend:

  1. Navigate to Enterprise > Archives, then select the Manage Backends tab.

  2. Click Create backend.

  3. Select S3 from the Backend Type dropdown.

  4. Enter the configuration options for your S3 backend:

    Title

    Enter a unique and descriptive name for the backend.

    Description

    Enter a description of the backend.

    S3 Endpoint URL

    (optional)

    Enter the URL of the S3 endpoint for the region that hosts your bucket (e.g. s3.us-east-1.amazonaws.com). This value is required only if you want to override the default endpoint. Leave this field blank to use the appropriate S3 endpoint based on the selected region.

    Warning: Graylog does not support retrieval of data from the Glacier tier of AWS S3. For S3 backends, make sure the S3 bucket does not have lifecycle rules that move files into the Glacier tier.

    AWS Authentication Type

    Select the authentication method for accessing Amazon S3. AWS recommends using role-based authentication with temporary credentials instead of long-term static access keys.

    AWS Assume Role (ARN)

    Enter the Amazon Resource Name (ARN) of the Identity and Access Management (IAM) role that Graylog will assume to access the S3 bucket. This method uses temporary credentials and is the preferred authentication approach. It also supports cross-account access.

    AWS External ID

    (optional)

    Enter the external ID for the assume role, if your role's trust policy requires one. This value confirms that only your intended service can assume the role, which prevents the role from being assumed by an unauthorized party. This field only applies when you provide an Assume Role ARN.

    S3 Bucket Name

    Enter the name of the S3 bucket in which logs will be stored.

    Spool directory

    Enter a directory where archiving data is stored temporarily before it is uploaded. Ensure that the directory is writable and has sufficient space for 10 times the Max Segment Size. Adjust the segment size on the Configuration page, if required.

    AWS Region

    Select an AWS region, or type a custom region.

    S3 Output Base Path

    Enter the base path where the archives should be stored within the S3 bucket. Use a simple directory path string, or you can create a template string to build dynamic paths, as described in Template Strings.

    You can use a single bucket for multiple purposes. For instance, you could use the same bucket for a Data Lake backend and a warm tier snapshot backend. However, if you do, it is important to use different sub folders for each specific use. The base path you set here determines the sub folder structure for this backend.

    If you are using a non-Amazon S3 bucket, complete the fields that best suit your choice of archive.

  5. Click Create backend.

  6. Proceed to Configure the Graylog Archive.

Apply AWS Security Permissions

When writing AWS security policies, make them as restrictive as possible. It is best practice to enable specific actions needed by the application rather than allowing all actions.

These permissions are required for Graylog to successfully make use of the S3 bucket:

Permission Description
CreateBucket Creates an S3 bucket.
HeadBucket Determines if an action is useful and if you have permission to access it.
PutObject Adds an object to a bucket.
CreateMultipartUpload Initiates a multipart upload and returns an upload ID.
CompleteMultipartUpload Completes a multipart upload by assembling previously uploaded parts.
UploadPart Uploads a part in a multipart upload.
AbortMultipartUpload Aborts a multipart upload.
GetObject Retrieves objects from Amazon S3.
HeadObject Retrieves metadata from an object without returning the object itself.
ListObjects Returns some or all (up to 1,000) of the objects in a bucket with each request.
DeleteObjects Enables you to delete multiple objects from a bucket using a single HTTP request.

The OVH Object Storage backend uses OVHcloud's S3-compatible object storage service to store archived index sets.

OVH Object Storage Backend Configuration Options

To configure an OVH Object Storage backend:

  1. Navigate to Enterprise > Archives, then select the Manage Backends tab.

  2. Click Create backend.

  3. Select OVH Object Storage from the Backend Type dropdown.

  4. Enter the configuration options for your OVH Object Storage backend:

    Title

    Enter a unique and descriptive name for the backend.

    Description

    Enter a description of the backend.

    OVH Endpoint URL

    Enter your OVH Object Storage endpoint URL (e.g. https://s3.gra.io.cloud.ovh.net).

    Authentication Type

    Select the authentication method for accessing OVH Object Storage.

    Access Key

    Enter your OVH Object Storage access key. This field appears only when you select Key & Secret as the Authentication Type.

    Secret Key

    Enter your OVH Object Storage secret key. This field appears only when you select Key & Secret as the Authentication Type.

    OVH Bucket Name

    Enter the name of your OVH Object Storage bucket.

    Spool directory

    Enter a directory where archiving data is stored temporarily before it is uploaded. Ensure that the directory is writable and has sufficient space for 10 times the Max Segment Size. Adjust the segment size on the Configuration page, if required.

    Region

    Select the region for your S3-compatible storage provider.

    OVH Output Base Path

    Enter the base path where the archives should be stored within the OVH bucket. Use a simple directory path string, or you can create a template string to build dynamic paths, as described in Template Strings.

    You can use a single bucket for multiple purposes. For instance, you could use the same bucket for a Data Lake backend and a warm tier snapshot backend. However, if you do, it is important to use different sub folders for each specific use. The base path you set here determines the sub folder structure for this backend.

  5. Click Create backend.

  6. Proceed to Configure the Graylog Archive.

The SysEleven Object Storage backend uses SysEleven's S3-compatible object storage service to store archived index sets.

SysEleven Object Storage Backend Configuration Options

To configure a SysEleven Object Storage backend:

  1. Navigate to Enterprise > Archives, then select the Manage Backends tab.

  2. Click Create backend.

  3. Select SysEleven Object Storage from the Backend Type dropdown.

  4. Enter the configuration options for your SysEleven Object Storage backend:

    Title

    Enter a unique and descriptive name for the backend.

    Description

    Enter a description of the backend.

    SysEleven Endpoint URL

    Enter your SysEleven S3-compatible storage endpoint URL.

    Authentication Type

    Select the authentication method for accessing SysEleven S3 Storage.

    Access Key

    Enter your SysEleven storage access key. This field appears only when you select Key & Secret as the Authentication Type.

    Secret Key

    Enter your SysEleven storage secret key. This field appears only when you select Key & Secret as the Authentication Type.

    SysEleven Bucket Name

    Enter the name of your SysEleven storage bucket.

    Spool directory

    Enter a directory where archiving data is stored temporarily before it is uploaded. Ensure that the directory is writable and has sufficient space for 10 times the Max Segment Size. Adjust the segment size on the Configuration page, if required.

    Region

    Select the region for your S3-compatible storage provider.

    SysEleven Output Base Path

    Enter the base path where the archives should be stored within the SysEleven bucket. Use a simple directory path string, or you can create a template string to build dynamic paths, as described in Template Strings.

    You can use a single bucket for multiple purposes. For instance, you could use the same bucket for a Data Lake backend and a warm tier snapshot backend. However, if you do, it is important to use different sub folders for each specific use. The base path you set here determines the sub folder structure for this backend.

  5. Click Create backend.

  6. Proceed to Configure the Graylog Archive.

Before you can establish Google Cloud Storage (GCS) as a backend, you must complete setup on your Google Cloud account.

  1. Create a GCS bucket. Follow Google's documentation on buckets to complete this process. Note the following:

    • To create a bucket, you must have the Storage Admin IAM role assigned for the project.

    • The bucket name must be globally unique, and you cannot change this name after the bucket is created. Make sure to note your bucket name as you need to provide it in the backend setup process in Graylog.

    • The default Standard storage class is recommended. However, depending on your use case, you might determine a different class is a better fit. Make sure that you understand cost implications with Google for each choice.

    • When setting access control and data protection and retention, be sure to follow your company guidelines and security best practices. Also, be aware that your choices can have cost implications from Google.

  2. Create a Google Cloud service account. Follow Google's documentation on service accounts to complete this process. Set permissions for this account such that it can read, write, and delete from the bucket.

  3. In Google Cloud, set up Application Default Credentials (ADC). Follow Google's documentation on ADC to complete this process. Depending on your environment, the steps might be as follows:

    1. Download your service account key file from the Google Cloud console.

    2. On every Graylog node, set the GOOGLE_APPLICATION_CREDENTIALS environment variable to point to the key file with a command like the following:

      Copy 
      export GOOGLE_APPLICATION_CREDENTIALS=/path/to/your/key.json

      Be sure to update the path in the above command to the location for your service account key file.

    3. To configure ADC with your Google account, run the following command in the Google Cloud CLI:

      Copy 
      gcloud auth application-default login

    Hint: You must complete ADC setup on all Graylog nodes in your environment! Google provides instructions for setting up ADC on multiple environment types, including development, on-premises, cloud, and containerized. Use the instructions that match your Graylog deployment.

GCS Backend Configuration Options

To configure a GCS backend:

  1. Navigate to Enterprise > Archives, then select the Manage Backends tab.

  2. Click Create backend.

  3. Select GCS from the Backend Type dropdown.

  4. Enter the configuration options for your GCS backend:

    Title

    Enter unique and descriptive name for the backend.

    Description

    Enter a description of the backend.

    GCS Bucket

    Enter the name of the GCS bucket in which logs will be stored.

    Project ID (optional)

    Enter your Google Cloud project’s unique identifier.

    The Google Cloud Project ID is a user-selected unique name that you can use to reference the Google project from Graylog. However, additional configuration is required, such as enabling the Google Cloud API, granting appropriate IAM roles, and additional authentication and authorization steps. Consult the Google Cloud documentation for complete information.

    Endpoint URI (optional)

    Enter a custom endpoint for accessing the storage. Leave this field blank to use the default value.

    Spool directory

    Enter a directory where archiving data is stored temporarily before it is uploaded. Ensure that the directory is writable and has sufficient space for 10 times the Max Segment Size. Adjust the segment size on the Configuration page, if required.

    GCS Output base path

    Enter the base path where the archives should be stored within the GCS bucket. Use a simple directory path string, or you can create a template string to build dynamic paths, as described in Template Strings.

    You can use a single bucket for multiple purposes. For instance, you could use the same bucket for a Data Lake backend and a warm tier snapshot backend. However, if you do, it is important to use different sub folders for each specific use. The base path you set here determines the sub folder structure for this backend.

  5. Click Create backend.

  6. Proceed to Configure the Graylog Archive.

Azure Blob Backend Configuration Options

To configure an Azure Blob backend:

  1. Navigate to Enterprise > Archives, then select the Manage Backends tab.

  2. Click Create backend.

  3. Select Azure Blob from the Backend Type dropdown.

  4. Enter the configuration options for your Azure Blob backend:

    Title

    Enter a unique and descriptive name for the backend.

    Description

    Enter a description of the backend.

    Azure Blob Endpoint URL

    (optional)

    Enter the URL for the Azure Blob endpoint. This value is required only if you want to override the default endpoint.

    Azure Blob Container Name

    Enter the name of the Azure Blob container in which logs will be stored.

    Azure account

    Enter the name of your Azure storage account.

    Azure account key

    Enter the account key for your Azure storage account.

    Spool directory

    Enter a directory where archiving data is stored temporarily before it is uploaded. Ensure that the directory is writable and has sufficient space for 10 times the Max Segment Size. Adjust the segment size on the Configuration page, if required.

    Azure Blob Output Base Path

    Enter the base path where the archives should be stored within the Azure Blob container. Use a simple directory path string, or you can create a template string to build dynamic paths, as described in Template Strings.

    You can use a single container for multiple purposes. For instance, you could use the same container for a Data Lake backend and a warm tier snapshot backend. However, if you do, it is important to use different sub folders for each specific use. The base path you set here determines the sub folder structure for this backend.

  5. Click Create backend.

  6. Proceed to Configure the Graylog Archive.

Template String for Storage

For the Output base path, you can define the path with a template string to help organize your data. For example, you can add the month, year, or day along with the index name to create subfolders based on these variables.

You can use the following variables to construct a dynamic value for each archive and give it structure:

Name

Description

${year}

Archival date year (e.g. 2026).

${month}

Archival date month (e.g. 04).

${day}

Archival date day (e.g. 01).

${hour}

Archival date hour (e.g. 23).

${minute}

Archival date minute (e.g. 24).

${second}

Archival date second (e.g. 59).

${index-name}

Name of the archived index (e.g. graylog_0).

The following example shows a defined template and what the resulting path would be:

# Template

/data/graylog-archive/${year}/${month}/${day}

# Result

/data/graylog-archive/2025/06/01/graylog_0

Select an Archiving Strategy

Archived indices are created using one of two strategies. The Scan strategy is the default method and requires no extra setup. The Snapshot strategy relies on OpenSearch snapshots, which can process large archiving backlogs faster, particularly on large clusters, but requires additional configuration.

The Scan strategy archives each index by reading its documents directly from the live index. This method requires no additional setup beyond your archive backend.

Complete the following configuration options for this strategy:

Name

Description

Enable multithreading

Provides faster archiving by using multiple threads per search node if enabled.

Restrict Restore to Leader Node

Runs archive restore jobs only on the leader node if enabled.

Max Segment Size

Sets the maximum size, in bytes, of archive segment files.

Compression Type

Sets the compression type used to compress the archives.

Checksum Type

Sets the algorithm used to calculate the checksum for archives.

Restore index batch size

Sets the OpenSearch batch size used when restoring archive files.

Failure notification threshold

Sets the number of consecutive failed attempts before Graylog shows an error notification.

Retention Time (days)

Sets how many days to keep archives before deleting them. Set the value to 0 to disable automatic deletion.

Streams to archive

Lists the streams you can select to include in the archive.

The Snapshot strategy builds the archive from an OpenSearch snapshot of the index. If a snapshot already exists through warm tier data routing, Graylog reuses it. Otherwise, Graylog takes a new snapshot for archiving.

Graylog downloads the created snapshot shard by shard to the spooling directory. Therefore, that directory must be sized large enough to hold the biggest shard plus some additional space.

Hint: The Snapshot strategy requires at least one OpenSearch node configured with the search role, the same requirement as the warm tier. See Prepare Your Environment for a Warm Tier for setup steps, and Manage Warm Tier Repositories to manage your snapshot repositories.

Complete the following configuration options for this strategy:

Name

Description

Snapshot Repository

Sets an S3 or file system repository used to create snapshots of hot indices.

If you already use warm tier data routing, Graylog reads snapshots from the existing warm tier repository instead of creating new ones. See Manage Warm Tier Repositories for information about the warm tier backend.

Legacy archive file system backends and backend types other than S3 and file system are not supported for the Snapshot strategy.

Path Override (optional)

Overrides the read path. Graylog uses the repository's registered location when this field is empty. The path must be readable by every Graylog node that may archive.

Restrict Creation to Leader Node

Runs archive creation jobs only on the leader node if enabled. This choice narrows the repository-access requirement to one node.

Processing Threads

Sets the number of threads used to convert each shard.

Restrict Restore to Leader Node

Runs archive restore jobs only on the leader node if enabled.

Max Segment Size

Sets the maximum size, in bytes, of archive segment files.

Compression Type

Sets the compression type used to compress the archives.

Checksum Type

Sets the algorithm used to calculate the checksum for archives.

Restore index batch size

Sets the OpenSearch batch size used when restoring archive files.

Failure notification threshold

Sets the number of consecutive failed attempts before Graylog shows an error notification.

Retention Time (days)

Sets how many days to keep archives before deleting them. Set the value to 0 to disable automatic deletion.

Streams to archive

Lists the streams you can select to include in the archive.

Enable Multithread Archiving

(Scan strategy only.) Multithread archiving allows for the use of more than one Java thread per search node. It is a faster option for creating archives because it expands write capacity by running a thread for each shard.

Multithread archiving is enabled by default.

Warning: Although selecting multithreading can increase archiving speed, it can also increase the load on your search cluster, which impacts the performance of your OpenSearch cluster.

Convert Shards with Processing Threads

(Snapshot strategy only.) The Processing Threads setting controls how many threads Graylog uses to convert each shard from an OpenSearch snapshot into an archive segment. As a best practice, start with a value about 25% of the archiving node's number of physical cores. More threads convert faster, but they also load the node harder, which can affect other functionality. A running job keeps its starting thread count even if you change this setting afterward.

Restrict Archive Restores to the Leader Node

The Restrict Restore to Leader Node setting limits archive restore jobs to run only on the leader node. Enable this setting when not all Graylog nodes can reach the backend storage.

Select a Max Segment Size

When you archive an index, the archive job writes the data into segments. The Max Segment Size setting sets the size limit for each of these data segments to control the size of the segment files and process them with tools with a file-size limit.

When the size limit is reached, a new segment file is started. For example:

Copy 
/path/to/archive/
  graylog_201/
    archive-metadata.json
    archive-segment-0.gz
    archive-segment-1.gz
    archive-segment-2.gz

Select a Compression Type

Archives are compressed with gzip by default, but you can switch to a different compression type.

The selected compression type significantly impacts the time it takes to archive an index. For example, gzip is generally slower, but the compression rate is more efficient. Snappy and LZ4 are quicker, but the archives will be larger.

Here is a general comparison between the available compression algorithms with select test data. Results will vary based on your data!

Type

Index Size

Archive Size

Duration

gzip

1 GB

134 MB

15 minutes, 23 seconds

Zstandard

1 GB

225.2 MB

5 minutes, 55 seconds

Snappy

1 GB

291 MB

2 minutes, 31 seconds

LZ4

1 GB

266 MB

2 minutes, 25 seconds

Warning: The current implementation of LZ4 is not compatible with LZ4 CLI tools, so it is currently impossible to decompress LZ4 archives outside of Graylog.

Select a Checksum Type

When Graylog writes archives, it also computes a CRC32 checksum over the files. Select a different option to use a different checksum algorithm if needed.

To find the most appropriate type of checksum, you might consider that CRC32 and MD5 are quick to compute and are a reasonable choice to detect damaged files, but neither are suitable for protection against malicious changes in the files. Graylog supports using SHA-1 or SHA-256 checksums, which can ensure the files were not modified as they are cryptographic hashes.

When selecting a checksum type, we recommend that you determine:

  • Whether the necessary system tools to compute them are installed (SHA-256 utility, for example).

  • The speed of checksum calculation for larger files.

  • Security considerations.

Set the Restore Index Batch Size

The Batch Size setting controls the batch size for re-indexing archive data into OpenSearch. When set to 1000, the restore job re-indexes the archived data in document batches of 1000.

Use this setting to control the speed of the restore process and the amount of load it generates on the OpenSearch cluster. The higher the batch size, the faster the restore progresses, and the more load is put on your OpenSearch cluster beyond the normal message processing.

Tune this setting carefully to avoid any negative impact on your message indexing throughput and search speed.

Set an Archive Retention Time

The Retention Time (days) setting controls how long Graylog keeps archive files before deleting them. When set to 0, automatic deletion is disabled and archives are kept indefinitely.

Hint: This setting controls how long the archive files themselves are kept after creation. It is separate from the retention settings on the index set, which determine when data gets archived in the first place. See Configure Index Retention with Data Tiering below for index-level retention.

Configure Index Retention with Data Tiering

Hint: Our model for rotation and retention of indices is referred to as data tiering. Data tiering is offered as an option to store and manage data in tiers for specific purposes. See Data Tiering for more information.

You can determine how long you want to retain the data in a new or existing index set. To do so:

  1. Navigate to the Rotation and Retention section on the index set configuration page.

  2. Toggle to the Data Tiering option.

  3. Select the minimum and maximum amount of days you want to store your data.

Data is deleted when the maximum amount of days are reached. If you wish to keep this data, you may select Archive before deletion.

If you have an Enterprise license, you may also select Enable Warm Tier. You can then enter the minimum amount of days you want the data to stay in the hot tier before moving it to warm. You can also designate a warm storage repository to store the index set in. If none exist, you can create a new warm storage repository in this menu.

Configure Index Retention (Legacy)

Warning: We strongly recommend utilizing data tiering for the rotation and retention of index sets. The following legacy strategies will be deprecated in the near future.

Graylog uses configurable index retention strategies to delete old indices. By default, indices can be closed or deleted if they exceed the configured limit.

The Graylog archive offers a separate index retention strategy that you can configure to automatically archive an index before closing or deleting it. Select Archive index to enable this feature. (See Index Model for details on these strategies.)

As with the other index retention strategies, you can configure a maximum number of OpenSearch indices. When there are more indices than the configured limit, the oldest indices are archived in the backend and closed or deleted. You can also choose to do nothing after archiving an index by selecting NONE. In that case, no cleanup of old indices will happen, and you will be able to manage the archive yourself.

Select Streams To Archive

The Streams to Archive setting is included in the archive and allows you to archive only important data as determined by your streams, rather than everything that is brought into Graylog.

Hint: New streams are archived automatically. If you create a new stream and don’t want it to be archived, disable it in this configuration dialog.

Next Steps

After you complete setup for Graylog archiving, you can archive data for a specified index set!

Copy 
/path/to/archive/
  graylog_201/
    archive-metadata.json
    archive-segment-0.gz
    archive-segment-1.gz
    archive-segment-2.gz