SUSE Installation: Multiple Graylog Nodes

This installation guide covers installing Graylog on SUSE Linux Enterprise Server 12 or 15 on a multi-node cluster consisting of more than one Graylog and MongoDB node and more than one Data Node. A generally recommended cluster design that supports most medium-to-large scale deployments is our Conventional model, which is composed of three Graylog and MongoDB nodes and three Data Nodes; however, this installation guide can generally be followed at any scale.

Hint: If you are instead deploying a single-node Graylog instance, see SUSE Installation: Single Graylog Node.

This article will guide you through installing three services that comprise the Graylog stack: MongoDB, Graylog Data Node, and Graylog.

Warning: This guide does not cover security settings! The server administrator must make sure the Graylog server is not publicly exposed and is following security best practices.

Prerequisites

Before proceeding, ensure that the following prerequisites are met:

Verify that all required ports are open. We assume in the following steps that firewalls are disabled and traffic can flow across all necessary ports.
A load balancer with a DNS record assigned for Graylog is encouraged for this deployment type.
We strongly recommend the use of an XFS file system for storage.
Review the Conventional architecture diagram and our recommendations for resource allocation before you begin installation.

Server Timezone

To set a specific time zone on a Graylog server, you can use the following command. (For more information on setting a time zone, we recommend this blog post.)

Copy

sudo timedatectl set-timezone UTC

Configuration and Service Files

Before you begin the process of installing Graylog and its required components, it is important to note that there are multiple configuration files and service files included in the Graylog and MongoDB operating system packages that should be adjusted during the initial deployment process to optimize your system performance. For your reference, the files mentioned in this installation guide for each service are as follows:

MongoDB: The MongoDB configuration file. For extensive information on the configuration properties included in this file, we recommend you review the MongoDB documentation. We offer some additional guidance on the use and maintenance of this file in Additional Configuration.
Data Node: The Graylog Data Node has both a configuration file and a service file that should be adjusted during deployment. A full reference for the Data Node configuration file can be found in Data Node Configuration Settings Reference. Also, the Data Node service file must be adjusted for heap settings; additional information on this file may be found in Additional Configuration.
Graylog: Graylog also has both a configuration file and a service file that should be adjusted during deployment. We recommend you review Initial Configuration Settings first. A full reference for the Graylog server configuration file can also be found in Graylog Server Configuration Settings Reference. Also, the Graylog service file must be adjusted for heap settings; additional information on this file may be found in Additional Configuration.

Install MongoDB

MongoDB is a NoSQL database designed to store and manage data in a flexible, scalable, and high-performance manner. As a part of the Graylog stack, MongoDB serves as the metadata database for the system.

The official MongoDB documentation provides a helpful tutorial on installation with SUSE. Additionally, we recommend you review their guide for setting up a MongoDB cluster. For information on installing MongoDB for use with Graylog, we recommend you generally follow the process below.

Warning: This guide assumes you are installing MongoDB 7.0. Review the compatibility matrix for versions of MongoDB supported both by your chosen Graylog and operating system versions.

1. Begin on a selected node in your Graylog/MongoDB cluster and import the MongoDB public key:

Copy

sudo rpm --import https://www.mongodb.org/static/pgp/server-7.0.asc

2. Add the MongoDB repository.

For SUSE 12 ↓

Copy

sudo zypper addrepo --gpgcheck "https://repo.mongodb.org/zypper/suse/12/mongodb-org/7.0/x86_64/" mongodb

For SUSE 15 ↓

Copy

sudo zypper addrepo --gpgcheck "https://repo.mongodb.org/zypper/suse/15/mongodb-org/7.0/x86_64/" mongodb

3. Install the latest stable version of MongoDB:

Copy

sudo zypper -n install mongodb-org

4. Hold the currently installed version of the MongoDB package to prevent it from being automatically upgraded to a newer version when updates are installed:

Copy

sudo zypper addlock mongodb-org

5. Open the MongoDB configuration file:

Copy

sudo nano /etc/mongod.conf

6. By default, MongoDB only listens locally. You must modify either the bindIp or bindIpAll setting in the MongoDB configuration file so that the port binds to an interface address or a host name or listens on all interfaces. See the example configurations below.

To listen on all interfaces:

Copy

net:
  port: 27017
  bindIpAll: true

To bind to a specific interface, like 192.168.50.71:

Copy

net:
  port: 27017
  bindIp: 192.168.50.71

To bind to a host name, like graylog01:

Copy

net:
  port: 27017
  bindIp: graylog01

7. Create the MongoDB replica set in the MongoDB configuration file:

Copy

replSetName: "rs0"

8. On the leader node only, initiate the replica set. In the following example command, graylog01, graylog02, and graylog03 represent the host names of the Graylog/MongoDB nodes. You must update them to reflect the host names or IP addresses of your Graylog/MongoDB nodes:

Copy

sudo mongosh
rs.initiate( { _id : "rs0", members: [ { _id: 0, host: "graylog01:27017" }, { _id: 1, host:"graylog02:27017" }, { _id: 2, host: "graylog03:27017" } ] })

9. Enable MongoDB and start the service:

Copy

sudo systemctl daemon-reload
sudo systemctl enable mongod.service
sudo systemctl start mongod.service

If this step is successful, the node should return:

Copy

{ ok: 1 }

10. Repeat all of the above steps for your additional MongoDB nodes.

Warning: You must complete the installation on all nodes to ensure proper functionality. The deployment will remain incomplete until all nodes have the service installed!

Install Data Node

Data Node is a component in Graylog's architecture designed to handle log ingestion, processing, and indexing, and manage communication with OpenSearch for storing and querying logs efficiently. As noted in the Conventional architecture, the Data Node service should be maintained on its own cluster, separate from Graylog and MongoDB.

To install Data Node, we recommend you follow the process below:

1. Begin on a selected node in your Data Node cluster and download the Graylog GPG key:

Copy

sudo wget https://downloads.graylog.org/repo/el/stable/GPG-KEY-graylog -O /etc/pki/GPG-KEY-graylog

2. Add the following configuration content to the repository file at /etc/zypp/repos.d/graylog.repo:

Copy

[graylog]
name=graylog
baseurl=https://packages.graylog2.org/repo/el/stable/6.1/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/GPG-KEY-graylog
repo_gpgcheck=0

3. Refresh the zypper repositories:

Copy

sudo zypper --gpg-auto-import-keys ref

4. Install the Data Node package:

Copy

sudo zypper install graylog-datanode

5. Ensure that the Linux setting vm.max_map_count is set to at least 262144. To check the current value:

Copy

cat /proc/sys/vm/max_map_count

To increase the value:

Copy

echo 'vm.max_map_count=262144' | sudo tee -a /etc/sysctl.d/99-graylog-datanode.conf
sudo sysctl --system
cat /proc/sys/vm/max_map_count

6. Create your password_secret. This is a secure, randomly generated string used to encrypt sensitive data within the system:

Copy

< /dev/urandom tr -dc A-Z-a-z-0-9 | head -c${1:-96};echo;

7. Open the Data Node configuration file:

Copy

sudo nano /etc/graylog/datanode/datanode.conf

8. Add the password_secret value to the Data Node configuration file. Note that you must add this same value to the Graylog server configuration files in a later step since it is crucial that this value be the same for all nodes that are members of the Graylog cluster.

9. Configure the MongoDB connection string in the Data Node configuration file. In the following example command, graylog01, graylog02, and graylog03 represent the host names of the MongoDB nodes. You must update them to reflect the host names or IP addresses of your MongoDB nodes:

Copy

mongodb_uri = mongodb://graylog01:27017,graylog02:27017,graylog03:27017/graylog?replicaSet=rs0

10. Ensure the heap settings are set to half your system memory, up to a max of 31 GB. To do so, you will need to add the following configuration property to your Data Node configuration file. For example, if you have 16 GB of RAM, you would set it to 8 GB, like in the example below:

Copy

opensearch_heap = 8g

Warning: This configuration property is not included in the datanode.conf by default! It must be added to the file manually.

11. Now, open the datanode file found in the default directory at /etc/graylog/datanode/:

Copy

sudo nano /etc/graylog/datanode

12. In this file, you must adjust the heap settings in jvm.options for the Data Node service, as noted in Additional Configurations. This service needs very little memory and can be set at 1 or 2 GB, depending on your requirements. It is recommended to set both values to the same size, so setting a minimum of 2 GB for Xms2g and a maximum of 2 GB for Xmx2g would look like the following:

Copy

-Xms2g
-Xmx2g

13. Enable Data Node and start the service:

Copy

sudo systemctl daemon-reload
sudo systemctl enable graylog-datanode.service
sudo systemctl start graylog-datanode

14. Follow all of the above steps for each additional Data Node with the exception of generating the password secret. The value you first generated and configured must be the same for each additional node.

Warning: You must complete the installation on all nodes to ensure proper functionality. The deployment will remain incomplete until all nodes have the service installed!

Install Graylog

After Data Node installation is complete, you must install the core Graylog service. Our recommended cluster architecture denotes that the Graylog and MongoDB services are installed on the same nodes.

To install Graylog, we recommend you follow the process below:

1. Install the Graylog package on a select node in your Graylog/MongoDB cluster.

For Graylog Open ↓

Copy

sudo zypper install graylog-server

For Graylog Enterprise ↓

Copy

sudo zypper install graylog-enterprise

For Graylog Security ↓

Copy

sudo zypper install graylog-enterprise

2. Use the following command to create your root_password_sha2. This is the password for the root administrator account for the Graylog interface. Make sure you record this password as you will need it to log into the Graylog interface after preflight configuration:

Copy

echo -n "Enter Password: " && head -1 </dev/stdin | tr -d '\n' | sha256sum | cut -d" " -f1

Warning: Do NOT attempt to log into Graylog for the first time using the password you generated in the steps above! You must complete the preflight login with the credentials found in the log file after first starting the Graylog service. You will be able to login to Graylog with your generated password after the preflight is complete. See The Web Interface for more information.

3. Open the Graylog server configuration file:

Copy

sudo nano /etc/graylog/server/server.conf

4. Enter the value you created for the root_password_sha2 in the previous step into the Graylog configuration file.

5. Retrieve the password secret from the Data Node configuration file as indicated in steps 6-8 of Install Data Node and add it to the password_secret property in the Graylog configuration file.

6. Make the following additional adjustments to your Graylog configuration file to optimize the performance and integration of your Graylog cluster.

Set the http_bind_address value to the public host name or a public IP address on which the Graylog web and API server will listen for incoming HTTP requests:
Copy
```
http_bind_address = 0.0.0.0:9000
```
Hint: More information about these settings can be found in The Web Interface.
Configure the MongoDB connection string. In the following example command, graylog01, graylog02, and graylog03 represent the host names of the MongoDB nodes. You must update them to reflect the host names or IP addresses of your MongoDB nodes:
Copy
```
mongodb_uri = mongodb://graylog01:27017,graylog02:27017,graylog03:27017/graylog?replicaSet=rs0 
```
Configure the external URI. The http://graylog.user.com/ value equals the DNS record set on your load balancer:
Copy
```
http_external_uri = http://graylog.user.com/
```
Adjust your journal settings. We recommend you configure your journal settings' max age to 72 hours and the size to your expected total log volume over a 72-hour period divided by the number of nodes in your cluster. So, if your expected daily log volume is 30 GB, your max size over 72 hours would be 90 GB. And if there are three Graylog nodes in your cluster, then you would divide 90 GB by 3, resulting in 30 GB, as in the example below:
Copy
```
message_journal_max_age = 72h
message_journal_max_size = 90gb
```
On the follower nodes only, ensure that the setting is_leader is set to false.
Copy
```
is_leader = false
```

Hint: The setting is_leader is set to true by default!

7. Now, open the graylog-server file found by default at /etc/graylog/:

Copy

sudo nano /etc/graylog/graylog-server

8. In this file, adjust your heap settings to half the system memory up to a max of 16 GB for the Graylog service, as recommended in Additional Configuration. It is recommended that you set the minimum and maximum values to be the same, so if you are setting your minimum to 2 GB, like shown in -Xms2g, and your maximum to 2 GB, like shown in -Xmx2g, the setting may look like:

Copy

GRAYLOG_SERVER_JAVA_OPTS="-Xms2g -Xmx2g -server -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow"

9. Enable Graylog and start the service:

Copy

sudo systemctl daemon-reload
sudo systemctl enable graylog-server.service
sudo systemctl start graylog-server.service

10. Follow all of the above steps for each additional Graylog node with the exception of generating the root administrator password. The value you first generated and configured must be the same for each additional node.

Warning: You must complete the installation on all nodes to ensure proper functionality. The deployment will remain incomplete until all nodes have the service installed!

Once installation is complete, proceed to the following section on the preflight login to access the Graylog web interface!

Preflight Login

After installation is complete, immediately proceed to The Web Interface for information on completing the preflight process and logging into Graylog for the first time. This preflight login will require the initial login credentials found in the log file after first starting the Graylog service!

Troubleshooting and Common Issues

The following section outlines troubleshooting steps for common issues to assist you in resolving potential challenges you may encounter.

Issue: Network Connectivity Issue During MongoDB Installation

It is possible you may receive a network connectivity issue while you are initiating the replica set for MongoDB. This may be related to your DNS and/or your TCP port or network configuration.

Solution: Modify Firewall and/or DNS Settings

If you encounter this issue, we recommend you test whether the required port TCP/27017 is open. You can use telnet or nc (Netcat) to test:

Copy

nc -vz <node_ip> 27017