Securing OpenSearch is a multi-faceted process necessary to ensure the confidentiality and integrity of your data. This process can involve various measures, such as enabling SSL/TLS, implementing Role-Based Access Control (RBAC), and restricting network access.

Graylog communicates with OpenSearch using HTTPS transport (TCP 9200) and can be configured to use HTTP-basic authentication. For HTTP transport, either certificate authentication or HTTP basic authentication is required. When configured, only the specified nodes allowed in the plugins.security.nodes_dn property in the opensearch.yml file can communicate with the OpenSearch cluster.

For comprehensive guidance on securing OpenSearch, consult the OpenSearch documentation.

Overview

This article will generally advise you on the process of configuring OpenSearch with the security plugin and setting up permissions for a Graylog user in the following steps:

1. Generate Certificates: Create the necessary SSL/TLS certificates for secure communication.

2. Configure OpenSearch Security Plugin: Enable and configure the security plugin in OpenSearch, including setting up roles and permissions.

3. Configure Graylog: Apply the required settings in Graylog to establish secure communication with OpenSearch.

Prerequisites

• OpenSearch without TLS configured

• DNS records configured and used for the dependencies listed above, foor example: opensearch1.example.org

• A general understanding of both X.509 and PKI is recommended

Generate Certificates

Certificate generation varies based on your specific environment and requirements due to the diversity of PKI solutions available. For detailed guidance, refer to the OpenSearch documentation on certificate generation. You will need to generate two types of certificates:

1. Internode Communication Certificate: used for secure communication between OpenSearch nodes

2. HTTP Communication Certificate: used for secure HTTP communication (TCP 9200), which Graylog uses

While this document does not cover the specific certificate generation process, you can consult the OpenSearch documentation for more information.

Configure TLS Certificates

Follow the steps below to apply the security configuration:

1. Open your opensearch.yml file in a text editor.

2. Change plugins.security.disabled: true to plugins.security.disabled: false.

3. Modify the following and paste into your opensearch.yml file.

   Copy

   plugins.security.ssl.transport.pemcert_filepath: /etc/opensearch/hostname.domain.tld.pem
   plugins.security.ssl.transport.pemkey_filepath: /etc/opensearch/hostname.domain.tld.key
   plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/opensearch/CA.pem 
   plugins.security.ssl.transport.enforce_hostname_verification: false plugins.security.ssl.http.enabled: true
   plugins.security.ssl.http.pemcert_filepath: /etc/opensearch/hostname.domain.tld.pem
   plugins.security.ssl.http.pemkey_filepath: /etc/opensearch/hostname.domain.tld.key
   plugins.security.ssl.http.pemtrustedcas_filepath: /etc/opensearch/CA.pem

   Hint: Replace hostname.domain.tld with the real value of your certificate. If your file names are different, change the file names to match your certificates. It is important that .pemtrustedcas_filepath contains the public key of the certificate (or certificate chain) that signed the certificate you specify for .pemcert_filepath.

   Note that transport and HTTP certificates can be different or the same depending on your requirements.

4. Update the configuration with correct certificate names and subject Distinguished Names (DNs). Modify these to reflect values specific to your certificates and paste into your opensearch.yml file, for example:

   Copy

   plugins.security.nodes_dn:
      - 'emailAddress=jd@jalogis.ch,CN=hostname.domain.tld,OU=Support shadowCA,O=shadowCA,L=Herne,ST=NRW,C=DE'
   plugins.security.authcz.admin_dn:
      - 'emailAddress=jd@jalogis.ch,CN=admin,OU=Support shadowCA,O=shadowCA,L=Herne,ST=NRW,C=DE'

   Note that this value must match the DN of the certificate. If unsure, you can use openssl to output this value by using the following command:

   Copy

   sudo openssl x509 -subject -nameopt RFC2253 -noout -in /etc/opensearch/hostname.domain.tld.crt | sed 's/subject=//'

5. When completed, your opensearch.yml file should look like this:

   Copy

   plugins.security.disabled: false
   
   plugins.security.ssl.transport.pemcert_filepath: /etc/opensearch/hostname.domain.tld.pem
   plugins.security.ssl.transport.pemkey_filepath: /etc/opensearch/hostname.domain.tld.key
   plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/opensearch/shadowCA.pem
   plugins.security.ssl.transport.enforce_hostname_verification: false
   
   plugins.security.ssl.http.enabled: true
   plugins.security.ssl.http.pemcert_filepath: /etc/opensearch/hostname.domain.tld.pem
   plugins.security.ssl.http.pemkey_filepath: /etc/opensearch/hostname.domain.tld.key
   plugins.security.ssl.http.pemtrustedcas_filepath: /etc/opensearch/shadowCA.pem
   
   plugins.security.nodes_dn:
      - 'CN=hostname.domain.tld,OU=Support,O=ExampleOrg,L=Houston,ST=Texas,C=US'
   plugins.security.authcz.admin_dn:
      - 'CN=admin,OU=Support,O=ExampleOrg,L=Houston,ST=Texas,C=US'

6. Restart OpenSearch.

   Copy

   sudo systemctl restart opensearch

   Hint: At this point, if you look in the OpenSearch log, you will see numerous SSL/TLS errors. This is normal and expected because the security plugin configuration has not been completed.

Configure OpenSearch Security Plugin

1. Open the internal_users.yml file in a text editor. This file is located in a sub-folder titled opensearch-security inside of the OpenSearch configuration path.

   • /etc/opensearch for OpenSearch configuration path

   • /etc/opensearch/opensearch-security for internal_users.yml

2. Generate a password hash that you will use to configure the Graylog user. When deciding on a password, it is recommended to use alphanumeric characters as special characters may cause syntax errors in Graylog's server.conf file.

   Copy

    export OPENSEARCH_JAVA_HOME=/usr/share/opensearch/jdk
   /usr/share/opensearch/plugins/opensearch-security/tools/hash.sh -p your_password_here

3. Using the output from hash.sh, change the hash for the Graylog user and then paste at the bottom of the internal_users.yml file.

   Copy

   graylog:
     hash: "<hash>"
     reserved: false
     backend_roles:
     - "admin"
     description: "Graylog Admin user"

Update tenants.yml File

This file is located in a sub-folder called opensearch-security inside of the OpenSearch configuration path.

• /etc/opensearch for OpenSearch configuration path

• /etc/opensearch/opensearch-security for tenants.yml

To update the file:

1. Open tenants.yml in a text editor.

2. Comment out for the values beneath ## Demo tenants. For example:

   Copy

   ## Demo tenants
   #admin_tenant:
   #  reserved: false
   #  description: "Demo tenant for admin user"

Run OpenSearch Security Admin Script

1. Replace the following values to match your environment:

Argument	Description
-cn	Cluster name as defined in opensearch.yml. For our purposes this is graylog.
-cacert	Path to CA cert (chain)
-cert	Path to admin cert
-key	Path to admin cert key
-h	Hostname to connect to
-cd	Configuration directory (yml file location)

2. Execute the following command:

Copy

export OPENSEARCH_JAVA_HOME=/usr/share/opensearch/jdk
sudo -u opensearch \
    -E /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh \
    -cn graylog \
    -nhnv \
    -cacert /etc/opensearch/shadowCA.pem \
    -cert /etc/opensearch/admin.pem \
    -key /etc/opensearch/admin.key \
    -h hostname.domain.tld \
    -cd /etc/opensearch/opensearch-security

If everything works as intended, the returned output should be as follows:

Copy

Security Admin v7
Will connect to hostname.domain.tld:9200 ... done
Connected as "<admin cert full dn>"
OpenSearch Version: 2.9.0
Contacting opensearch cluster 'graylog' and wait for YELLOW clusterstate ...
Clustername: graylog
Clusterstate: YELLOW
Number of nodes: 1
Number of data nodes: 1
.opendistro_security index does not exists, attempt to create it ... done (0-all replicas)
Populate config from /etc/opensearch/opensearch-security/
Will update '/config' with /etc/opensearch/opensearch-security/config.yml 
   SUCC: Configuration for 'config' created or updated
Will update '/roles' with /etc/opensearch/opensearch-security/roles.yml 
   SUCC: Configuration for 'roles' created or updated
Will update '/rolesmapping' with /etc/opensearch/opensearch-security/roles_mapping.yml 
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '/internalusers' with /etc/opensearch/opensearch-security/internal_users.yml 
   SUCC: Configuration for 'internalusers' created or updated
Will update '/actiongroups' with /etc/opensearch/opensearch-security/action_groups.yml 
   SUCC: Configuration for 'actiongroups' created or updated
Will update '/tenants' with /etc/opensearch/opensearch-security/tenants.yml 
   SUCC: Configuration for 'tenants' created or updated
Will update '/nodesdn' with /etc/opensearch/opensearch-security/nodes_dn.yml 
   SUCC: Configuration for 'nodesdn' created or updated
Will update '/whitelist' with /etc/opensearch/opensearch-security/whitelist.yml 
   SUCC: Configuration for 'whitelist' created or updated
Will update '/audit' with /etc/opensearch/opensearch-security/audit.yml 
   SUCC: Configuration for 'audit' created or updated
Will update '/allowlist' with /etc/opensearch/opensearch-security/allowlist.yml 
   SUCC: Configuration for 'allowlist' created or updated
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
Done with success

Test and Validate

At this point, your OpenSearch cluster should be fully configured with TLS, both for internet transport and HTTP(S).

You can validate success by using curl.

• To test HTTP basic authorization:

# Graylog User
curl https://hostname.domain.tld:9200/_cluster/health?pretty -k -u graylog:password

# Admin User
curl https://hostname.domain.tld:9200/_cluster/health?pretty -k -u admin:password

• Test using certificate authorization:

curl https://hostname.domain.tld:9200/_cluster/health?pretty -k --cert admin.crt --key admin.key

Configure Graylog

In order for Graylog to successfully connect to your newly secured OpenSearch cluster, it must be able to trust the certificate that OpenSearch is using. We can accomplish this by configuring Graylog to use a custom Java Key Store (JKS) and adding your certificate chain to the JKS.

Configure Graylog to use a custom Java Key Store (JKS)

1. Make a copy of the default cacerts JKS:

   Copy

   sudo cp /usr/share/graylog-server/jvm/lib/security/cacerts /etc/graylog/graylog.jks

2. Change the file owner to graylog:

   Copy

   sudo chown graylog:graylog /etc/graylog/graylog.jks

3. Prepare your certificate trust chain file. Typically this will be the root certificate authority and intermediate certificates placed in a single text file.

   Warning: This certificate chain must not include the certificate configured for OpenSearch. For example, if you configure /etc/opensearch/hostname.domain.tld.pem, that certificate should not be contained in the certificate imported into the JKS.

4. Ensure that OpenJDK is installed, as Keytool requires it. For Graylog version 5.0 and above installed via package (which includes the bundled JVM/JDK), run the following command (in this example, for Ubuntu):

   Copy

   sudo apt install -y openjdk-17-jre-headless

5. Import the certificate chain into JKS:

   Copy

   sudo keytool -importcert -keystore /etc/graylog/graylog.jks -noprompt -storepass changeit -alias opensearch_chain -file  /etc/graylog/trustchain.pem

6. Edit the JVM settings using a text editor, such as Vim or Nano. Use sudo when launching your text editor to ensure you can save changes to the file.

   • For Debian/Ubuntu open the Graylog server settings configuration file /etc/default/graylog-server.

   • For Red Hat/CentOS open the Graylog server settings configuration file /etc/sysconfig/graylog-server.

7. Add the following to the first line starting with GRAYLOG_SERVER_JAVA_OPTS:

Copy

-Djavax.net.ssl.trustStore=/etc/graylog/graylog.jks

For example:

Copy

GRAYLOG_SERVER_JAVA_OPTS="-Xms8g -Xmx8g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djavax.net.ssl.trustStore=/etc/graylog/graylog.jks"

See Default File Locations. for other configuration file paths.

Configure OpenSearch Nodes via server.conf

1. Open server.conf file in a text editor. By default, this file is located at /etc/graylog/server/server.conf.

2. Set the value of elasticsearch_hosts to match your information, for example:

   Copy

   elasticsearch_hosts = https://graylog:password@hostname.domain.tld:9200

3. To configure more than one OpenSearch node, use a comma separated list, for example:

   Copy

   elasticsearch_hosts = https://graylog:password@hostname1.domain.tld:9200,https://graylog:password@hostname2.domain.tld:9200

Restart Graylog Server

• Restart Graylog to implement your configuration changes:

  Copy

  sudo systemctl restart graylog-server