Graylog 5.0 introduces support for OpenSearch 2.0+ along with several changes to the minimum versions of required software.

Prerequisites

Hint: It is important to be aware of the versions of indices that exist on the OpenSearch 1.x cluster. Any index whose version is ES 6.7 (or older) should be reindexed before upgrading OpenSearch software.

Indices on the OpenSearch 1.x cluster may need to be reindexed before upgrading to OpenSearch 2.x if these versions are not supported by OpenSearch 2.x:

  • The index [[logstash-index-000098/ka-F8tMiS-qJh8OBbv4pRA]] was created with version 6.7.0, but the minimum compatible version is OpenSearch 1.0.0 (or Elasticsearch 7.0.0). It should be re-indexed in OpenSearch 1.x (or Elasticsearch 7.x) before upgrading to 2.2.01.

OpenSearch 2.x is compatible with indices as old as Elasticsearch 7.0.0. The following command returns the names of indices whose version is older than or equal to ES 6.7:

Copy 
curl -X GET "http://localhost:9200/_settings?pretty=true" | jq '.[] | select(.settings.index.version.created <= 6700000) | [.settings.index.provided_name, .settings.index.version.created]'

This example uses the tool jq to parse the response from the Elasticsearch API at localhost port 9200.

To manually check the versions of each index, execute the following command to obtain a list of index versions:

Copy 
curl -X GET "http://localhost:9200/_settings?pretty=true" | jq '.[] | [.settings.index.provided_name, .settings.index.version.created]'

Upgrade to OpenSearch 2.x

OpenSearch’s documentation covers the upgrade process from end to end, which offers two paths to follow.

This guide will approach the upgrade process in several steps and will combine these two paths into one process. Make sure that you choose one approach to upgrading and stick with that process through the remainder of the upgrade--e.g. if you are starting a rolling-restart upgrade, then do not switch to a full-cluster restart upgrade in the middle.

It is a best practice to backup OpenSearch configuration files and create a new snapshot of your OpenSearch cluster before upgrading so that you have a known good point-in-time backup in the event of a failed upgrade. Moreover, filesystem backups of nodes' data directories can be created when the entire cluster is offline (e.g. this should not be attempted with a rolling-restart upgrade). This, however, is not a supported backup method and therefore is not recommended.

1. Pause message processing on all nodes in your Graylog deployment. Each node listed on the Nodes page within Graylog has a "More Actions" button which includes "Disable message processing."

2. Disable shard allocation to prevent OpenSearch from replicating shards as you shut down its nodes:

Copy 
curl -X PUT "http://localhost:9201/_cluster/settings?pretty" -H 'Content-Type: application/json' -d'
{
    "transient" : {
        "cluster.routing.allocation.enable" : "none"
    }
}
'

3. Shut down OpenSearch v1.3.5 on all nodes (full-cluster restart upgrade) or one node (rolling-restart upgrade).

  • In rolling-restart upgrades, all leader-ineligible nodes must be upgraded first before upgrading any leader-eligible nodes.

    • List leader ineligible nodes: GET /_nodes/_all,master:false

    • List leader eligible nodes: GET /_nodes/_all,master:true

4. Install OpenSearch 2.x software to upgrade the node(s).

Copy 
sudo systemctl stop opensearch
sudo mkdir /usr/share/bkup-opensearch

sudo mv /usr/share/opensearch /usr/share/bkup-opensearch
sudo dpkg -i opensearch-<version>-linux-x64.deb
sudo mv /usr/share/bkup-opensearch/opensearch/config/opensearch.yml /etc/opensearch/
sudo chown opensearch:opensearch /etc/opensearch/opensearch.yml 
sudo systemctl daemon-reload
sudo systemctl enable opensearch.service
sudo systemctl start opensearch
sudo systemctl status opensearch 

5. Confirm the OpenSearch 2.x cluster returns to a green state:

Copy 
curl -X GET http://localhost:9200/_cluster/health?pretty=true

6. (This step is for a rolling-restart upgrade.) Repeat steps 3–5 until all nodes are using OpenSearch. Remember, all leader-ineligible nodes must be upgraded first before upgrading any leader-eligible nodes.

7. Enable OpenSearch shard allocation:

Copy 
curl -X PUT "localhost:9201/_cluster/settings?pretty" -H 'Content-Type: application/json' -d'
{
    "transient" : {
        "cluster.routing.allocation.enable" : "all"
    }
}
'

8. Restart Graylog on all nodes.