Backup Considerations

In order to ensure the availability and integrity of your Graylog system, it is crucial to have a proper backup and restore strategy. These backups should be validated and securely stored to provide a lifeline in case of unexpected data loss or system failure during an upgrade. This documentation provides general guidance on how to backup and restore MongoDB, OpenSearch (or Elasticsearch), and Graylog.

Warning: If you are unsure about the backup process,STOP and contact support for assistance.

Backup MongoDB

In MongoDB the tool mongodump creates a binary export of a database. Instructions on how to utilize this tool can be found in the MongoDB documentation.

As part of a Graylog deployment, a database typically named graylog is created and needs to be backed up; however, you need to know whether you have MongoDB deployed as a single host or as a replica set. To identify your deployment type, check the value of the mongodb_uri parameter in your Graylog node(s) server.conf configuration file. More information on that can be found in MongoDB's knowledge base.

In addition to backing up your Graylog MongoDB database, back up the configuration files of your MongoDB replica members. Here are their default locations listed by platform. If you have security enabled, consider backing up other files, such as certificates and keystore files.

Backup Procedure

To backup MongoDB:

1. Use the mongodump tool to create a dump of your database.

2. Ensure that the backup directory is not on the same disk or volume as the MongoDB data files for better protection against disk failures.

Restore Procedure

To restore MongoDB:

1. Restore your data to the same version used in backing up to prevent compatibility issues. For example, if you have a backup from MongoDB 4.4, you should restore it on MongoDB 4.4. You should not restore it on 5.0.

2. Make sure that the database you want to restore does not already exist. If it does, you can either drop it or restore to a different database.

3. Use the mongorestore command to restore the dump previously created.

Backup Elasticsearch

Elasticsearch supports snapshot creation for backup/restore purposes. These are incremental backups of your data and can contain everything from a single index to an entire cluster, which can be stored in a repository that is either local or remote to the cluster. Snapshots created can then be restored to the same or other Elasticsearch clusters.

Prepare for OpenSearch Migration

To prepare for migrating from Elasticsearch to OpenSearch, create a snapshot of your entire cluster. You must register a snapshot repository before you can create snapshots.

Snapshot repositories are used to store snapshots. Once the repository has been created and defined in the Elasticsearch cluster, you can create a snapshot of your cluster. Elasticsearch provides more information on this process in the following documentation:

• Elasticsearch 6.8 - Snapshot and Restore

• Elasticsearch 7.10 - Snapshot and Restore

In addition to creating a snapshot of your Elasticsearch cluster before upgrading to OpenSearch, you should also create backups of your Elasticsearch node(s) configuration files. You can find their default locations by platform here in our documentation. You may also need to consider backing up other files such as certificates and keystore files if you have security enabled.

Backup OpenSearch

Opensearch provides the ability to take snapshots of your running cluster that can then be restored into another running cluster via the restore API. An OpenSearch snapshot is a backup of an index taken from a running cluster.

Backup Procedure

To backup OpenSearch:

1. Create a snapshot repository by modifying the opensearch.yml file:

   Copy

   path.repo: /path/to/backup/repository

2. Restart the OpenSearch service for the changes to take effect.

3. Register the repository using the Snapshot API.

You can also automate snapshot creation by using the snapshot management feature.

Restore Procedure

To restore OpenSearch:

1. Restore your data to the same version used in backing up to prevent compatibility issues. For example, if you have a backup from OpenSearch 2.12, you should restore it on OpenSearch 2.12. You should not restore it on 2.13.

2. Ensure that OpenSearch is running.

3. Close the indices that you want to restore:

   Copy

   POST /<index>/_close

4. Restore the snapshot by using the OpenSearch restore API:

   Copy

   POST /_snapshot/<repository>/<snapshot>/_restore

5. Reopen the indices:

   Copy

   POST /<index>/_open

Backup Graylog

Default locations are listed by platform. Your Graylog configuration files would typically be found at /etc/graylog/server/server.conf. Similar to MongoDB and OpenSearch backups, consider backing up other files such as certificates and keystore files if custom certificates are in use.

Backup Procedure

To backup Graylog, you need to consider the following components:

• Graylog server configuration: Backup the /etc/graylog/server/server.conf file.

• Graylog JVM settings: Backup configuration parameters that control the Java Virtual Machine (JVM) environment in which Graylog runs. Here are the default file paths:

  • (Debian) /etc/default/graylog-server

  • (Redhat) /etc/sysconfig/graylog-server

• Plugins and customizations: Backup any additional plugins and customizations you have made to Graylog. The path to the plugins directory is as follows: /usr/share/graylog-server/plugin

• Graylog message data: Backup the message data stored in MongoDB (as explained in the MongoDB section above).

Restore Procedure

To restore Graylog, follow these general procedures:

1. Install Graylog with the same version as your backup.

2. Stop the Graylog services.

3. Restore the Graylog server configuration file.

4. Restore any plugins and customizations.

5. Restore the MongoDB data.

6. Update for new DNS names if required.

7. Start the Graylog services.

It is recommended to test your backup and restore procedures in a non-production environment to ensure their effectiveness and proper functioning before relying on them in a production environment.

Prepare for Disaster Recovery

In a disaster recovery situation, a solid backup and restore plan is an ideal solution. The best practice to safeguard against disaster is frequent MongoDB backups and OpenSearch snapshots taken and copied over to a disaster recovery site with a MongoDB replica set or OpenSearch cluster ready to restore the data.

In order to restore Graylog after a total system crash, first you would need the Graylog server.conf file, particularly the key you used for the password_secret in initial configuration. The second important part is MongoDB. This database contains all configurations. Follow MongoDB instructions on backup.

Copy data from one location to another that would not be affected by common causes of a service disruption to the original site. Typically, this would be another site or data center. If a network link or storage system goes down and you need to switch to the disaster recovery site, you can restore the MongoDB database and manually restore snapshots up to the most recent backup.

If you need to restore log data, you can do this using the Graylog Enterprise archiving feature or any other OpenSearch backup and restore option. Merely copying the data directories of your OpenSearch nodes might not work. You might not be able to restore from that.

Recommended Order of Operations

1. Restore OpenSearch data and configurations.

2. Restore MongoDB.

3. Install or copy configuration files for Graylog.

4. Update configuration files. Server names and IPs will likely be different in another location. You will need to update the MongoDB and Elasticsearch URIs to reflect those names.