TLS-Secured Data Node Migration

Conditions

Note that for this migration method, the following conditions must be true: 

• Your Data Node installation will be on the same hosts as existing OpenSearch nodes.

• You have deployed Graylog directly on a physical host, not within containers or on virtual hosts, using the same parameters outlined in the installation documentation.

• Your system has the ability to download and install the Graylog Data Node package from the remote repository.

• You must have a "healthy" OpenSearch cluster (including one or more nodes) that are configured to use TLS via the OpenSearch Security Plugin. In this instance, "healthy" is defined as the ability to curl the OpenSearch API without using the -k flag and return the proper output. For example:

  • Via certificate authentication:

    Copy

    curl --cert cert.pem --key key.pem https://opensearch:9200

  • Via basic authentication:

    Copy

    curl 'https://user:password@opensearch:9200'

• You have available (or can reproduce) the exact securityadmin.sh command and arguments used to initially secure the OpenSearch cluster.

Hint: You do not need to have your own CA or PKI set up for the new Data Nodes. We strongly recommend you use the internal Graylog CA and certificates generated and managed by Graylog during the migration process.

Prepare Data Node Configuration File

To begin the migration process, prepare the Graylog Data Node configuration file by installing the service and updating key settings to align with your existing OpenSearch and MongoDB environments.

1. First install the Data Node service. The supported process for installation is described in the individual installation guides depending on your operating system and whether or not you will be running a single- or multi-node Data Node cluster. For example, the instructions for installing Data Node on Ubuntu 22.04 or 24.04 as a single-node installation may be found here. For a list of all available guides, see Install Graylog.

2. During installation, you will be prompted to modify the Data Node configuration file at /etc/graylog/datanode/datanode.conf. Ensure that the following key configuration properties are set as indicated: 

   • Set the password_secret to the same value as the password_secret you set in Graylog’s server.conf.

   • Set the opensearch_data_location to the path.data value set in opensearch.yml.

   • Change the mongodb_uri to URL to connect to any existing MongoDB instance.

   • Leave the opensearch_http_port set to 9200 and opensearch_transport_port set to 9300. There should not be a port conflict with existing OpenSearch nodes as, at the first start of the Data Node service, Graylog can detect that new embedded OpenSearch is not configured yet, so only the Data Node API will be started on port 8999. It will not start the new embedded OpenSearch nodes until the migration wizard reaches a certain point.

3. As indicated in the installation guides, ensure you enable Data Node and start the service. For example: 

   Copy

   sudo systemctl daemon-reload
   sudo systemctl enable graylog-datanode.service
   sudo systemctl start graylog-datanode

Migrate to Data Node

Now, to begin the migration, log in to the Graylog web interface and complete the following steps:

1. Navigate to System > Data Nodes > Data Node Migration.

2. Wait for all Data Nodes to be discovered. They should be listed with a status of UNCONFIGURED.
   
   

3. Click Next  when all nodes are discovered.

4. Create a new CA when prompted by the interface.

5. Configure the renewal policy according to your preferences.

6. You should now be on the Migration Steps page. Carefully read the important information in the interface before proceeding.

Configure JWT Authentication

In order to prepare the indices from the existing OpenSearch cluster for migration, you must configure JWT (Java Web Token) authentication to give Data Node the proper authorization to import this data.

Warning: Only run these commands on the same server you used to configure the OpenSearch Security Plugin. Do not run these commands on any other OpenSearch node!

1. Create a base64-encoded version of your Graylog password_secret for use in setting up the JWT token:
   

   Copy

   echo -n $(grep password_secret /etc/graylog/datanode/datanode.conf | cut -d' ' -f3) | base64 -w0 && echo

2. Copy the JWT config below. Replace the value in signing_key with the base64-encoded string you just generated.

   Copy

   description: "Authenticate via Json Web Token"
           http_enabled: true
           transport_enabled: true
           order: 1
           http_authenticator:
             type: jwt
             challenge: false
             config:
               signing_key: "base64 encoded HMAC key or public RSA/ECDSA pem key"
               jwt_header: "Authorization"
               jwt_url_parameter: null
               roles_key: "os_roles"
               subject_key: null
           authentication_backend:
             type: noop
   

3. Copy the updated configuration block into $OPENSEARCH_CONFIG/opensearch-security/config.yml under the existing jwt_auth_domain configuration block.

   Hint: Copy the entire lines, including leading whitespaces. When pasting, place your cursor at the start of a new line to preserve correct indentation.

4. You must either disable the existing jwt_auth_domain configuration or replace it completely.

5. Run the securityadmin.sh command with the same arguments you used when initially setting up OpenSearch Security Plugin.

6. To verify the JWT token has been applied successfully, copy and paste the contents of thisjwt.sh script into a local jwt.sh file.

7. Then, run the following command:

   Copy

   curl -k -H "Authorization: Bearer $($(echo $SHELL) ./jwt.sh \
   $(grep password_secret /etc/graylog/datanode/datanode.conf | cut -d' ' -f3))" \
   https://localhost:9200
   

8. If successful, it will return the default output of the OpenSearch API root (/) endpoint, as shown below:

   Copy

   {
     "name" : "opensearch01.lab",
     "cluster_name" : "graylog",
     "cluster_uuid" : "7fHMidh5R36pTKE48a94Jw",
     "version" : {
       "distribution" : "opensearch",
       "number" : "2.15.0",
       "build_type" : "deb",
       "build_hash" : "61dbcd0795c9bfe9b81e5762175414bc38bbcadf",
       "build_date" : "2024-06-20T03:27:12.329408119Z",
       "build_snapshot" : false,
       "lucene_version" : "9.10.0",
       "minimum_wire_compatibility_version" : "7.10.0",
       "minimum_index_compatibility_version" : "7.0.0"
     },
     "tagline" : "The OpenSearch Project: https://opensearch.org/"
   }
   

Directory Compatibility Check

Verify your directory compatibility with the new Data Node directory.

1. Return to the Data Node Migration page in Graylog.

2. Pause message processing on all Graylog nodes by navigating to Cluster Configuration and selecting Pause message processing from the More actions menu.

3. Stop the legacy OpenSearch service on your OpenSearch node:
   

   Copy

    systemctl disable --now opensearch.service

4. Change ownership on the OpenSearch data path:
   

   Copy

   chown -R graylog-datanode:graylog-datanode /var/lib/opensearch/
   

5. Navigate again to the Data Node Migration page and select Directory Compatibility Check.

Provision Data Nodes with Certificates

To provision Data Nodes with certificates:

1. Start by expanding each Data Node in the Graylog interface by selecting the node to see per-index information and compatibility.

2. Click the Provision Data Nodes with certificates button.

You should see the following success message. Note that the Status field says Provisioned.

Journal Downtime Warning

Continuing in the Graylog migration interface, review and acknowledge the warning about message processing downtime and Graylog journal size. This information is essential for planning downtime and minimizing the risk of data loss.

Stop Message Processing

Confirm that message processing and all OpenSearch nodes are stopped.

1. Click Next to continue.

Update server.conf and Restart Graylog

1. Wait for all nodes to have a status of Available.
   

   Hint: This process may take several minutes. During this time, you might see log entries in datanode.log that appear unusual, or the Data Nodes list in the interface may show nodes temporarily dropping. This behavior is normal. Investigate further if the logs show continuous Java stack trace errors in the logs or the Data Nodes list shows no discovered nodes for more than 60 seconds.

2. Navigate to the Graylog configuration file on each Graylog node and comment out the elasticsearch_hosts property.

3. Restart the Graylog service:

   Copy

   sudo systemctl restart graylog-server.service

4. Navigate back to theData Nodes page in the Graylog interface and verify that the nodes are green, indicating that the nodes are available.

The migration is now complete!

Verify Migration Success

Verify that your Data Node migration has completed successfully:

1. Open the Graylog Search page and confirm you see the expected results, including data from before the migration.

2. If you see this error message: OpenSearch exception [type=index_not_found_exception, reason=no such index []], then navigate to System > Indices and select Maintenance > Cleanup & recalculate all index ranges in the top-right.

3. If  there are no errors, navigate to System > Overview and confirm:

   • There are no red notifications (relevant to migration).

   • Your OpenSearch cluster is green.

   • There are no processing or indexing failures.

4. Navigate to System > Nodes and verify that all nodes are processing messages and that their journal utilization is decreasing or at pre-migration levels.

5. In System > Indices, verify all existing index sets are still listed, as well as total shards is the same or greater than before. (These may be greater as index sets may have rotated during migration.)

• Graylog Data Node

• Data Node Migration

• Data Node Configuration Settings Reference

• Initial Configuration Settings