Introduced in Graylog 5.2 instant archiving offers an alternative method for archiving log data. The original method for archiving logs in Graylog remains supported and available; however, this new method bases the process of archiving on streams rather than on indices. The benefit to this method is that selected log data is archived as soon as it is imported into a Graylog stream based on a user's preferred configuration. Additionally, as the original Graylog method of archiving based on data that has been indexed is preserved, you can essentially archive data in parallel using both archiving methods depending on your needs and configuration settings. This data can be restored at any future point for searching and analyzing in Graylog or for exporting to Elasticsearch/OpenSearch if needed.
Prerequisites
-
This is a Graylog Cloud exclusive feature and is not currently available for self-managed Graylog instances.
Create A New Archive
As instant archiving is based on streams, to create a new archive set, you will need to determine which stream(s) your archive will be based on. Note that you may attach multiple streams to one archive as needed.
-
Navigate to Enterprise > Instant Archiving.
-
Select the Configuration tab and complete the following fields:
-
Archive Name: A unique name for the new archive. (Note that spaces will not be permitted.)
-
Streams: Select the stream(s) from which log data will be archived.
-
Archive Enabled: Select to enable archiving upon creation.
-
-
Then, click on Create.
-
Once you have created your archive, select the Overview tab to view and manage all your created archives.
Manage Your Archives
To view and manage your archives in the Enterprise > Instant Archiving menu, select the Overview tab. Here you will find a list of all created archives and their current status, whether active or inactive.
To view detailed information about an archive, select the archive name from this list. On this page you will be able to view information specific to each archive, including a total message count, streams attached to the archive, the time period in which the archive has been actively collecting data, and a detailed view of any restore operations for data in the archive.
Restore Log Data from an Archive
You can restore archived log data so that it may be utilized in Graylog for search or analysis purposes.
-
In the Enterprise > Instant Archiving menu, ensure that the Overview tab is selected.
-
For the archive from which you wish to restore data, locate the Actions column and select Restore.
-
From the resultant Restore Archives menu, select the time range for the data you wish to restore. You may limit your search by date or select a more specific time period within those dates by adjusting the hours, minutes, and seconds ranges.
-
You may also choose to include all streams within the archive or only a specific stream or subset of streams.
-
Note that the Estimation modal will provide you with an estimated message count and size for the range of data you have selected.
-
Once you have finished select Restore.
-
From the Overview menu you will see an Instant Archiving jobs tab appear while the restoration of your data is in progress.
-
Once the job has been completed, select the archive name from the list to see your newly completed restore operation.
-
Select Show messages from the Actions column for your recent restore. You will then be able to search and analyze your restored log data.
Hint: Indices containing this restored data will also be created once the restore is complete. In the Restore Operations menu, select the restored archive for a detailed view, and you will be able to view the new indices.
Delete Log Data from an Archive
Additionally, you may wish to delete all or a specific subset of data from an archive if it is no longer needed.
Warning: Please note that any data that is deleted from an archive CANNOT be retrieved by Graylog at a later date.
-
In the Enterprise > Instant Archiving menu, ensure that the Overview tab is selected.
-
For the archive from which you wish to delete data, locate the Actions column and select Delete.
-
From the resultant Restore Archives menu, you may choose to delete the entire archive of data, or you can opt to delete a portion of the data based on the time range in which the data was logged. (Please note that if you choose to delete all of the data within an archive, you will also be prompted to determine whether or not you want to delete any restored data from this archive as well.)
-
Once you have made your selections, click Delete to permanently delete the data.
Stop or Delete an Archive
In the Overview menu, you can stop an archive from collecting data by selecting the status in the Archive status column. If the status shows as “Stopped,” then log data for your selected streams are no longer being instantly archived. You can restart archiving at any time by selecting the status again so that it shows as “Running.”
Alternatively, you can also delete the archive configuration from the Configuration menu by selecting the checkbox next to the archive name and clicking Delete. Note that this will delete the configuration you have applied to the archive so that data will no longer be archived according to your specifications, but it will not delete the data that has been archived previously. Should you wish to restore your configuration settings at a later date, in the Overview menu select Restore from the Actions column for the specific archive you wish to enable.
