Debian Installation: Multiple Graylog Nodes

This installation guide covers installing Graylog on Debian Linux 10 (Buster), 11 (Bullseye), or 12 (Bookworm) on a multi-node cluster consisting of more than one Graylog and MongoDB node and more than one Data Node. A generally recommended cluster design that supports most medium-to-large scale deployments is our Conventional model, which is composed of three Graylog and MongoDB nodes and three Data Nodes; however, this installation guide can generally be followed at any scale.

sudo timedatectl set-timezone UTC

The official MongoDB documentation provides a helpful tutorial on installation with Debian. Additionally, we recommend you review their guide for setting up a MongoDB cluster. For information on installing MongoDB for use with Graylog, we recommend you generally follow the process below.

1. Begin on a selected node in your Graylog/MongoDB cluster and install gnupg:

sudo apt-get install gnupg curl

2. Import the MongoDB public key:

curl -fsSL https://www.mongodb.org/static/pgp/server-7.0.asc | \
   sudo gpg -o /usr/share/keyrings/mongodb-server-7.0.gpg \
   --dearmor

3. Create a list file for MongoDB:

echo "deb [signed-by=/usr/share/keyrings/mongodb-server-7.0.gpg] https://repo.mongodb.org/apt/debian bookworm/mongodb-org/7.0 main" | sudo tee /etc/apt/sources.list.d/mongodb-org-7.0.list

4. Reload the local package database:

sudo apt-get update

5. Install the latest stable version of MongoDB:

sudo apt-get install -y mongodb-org

6. When managing cryptographic keys, we recommended using a keyserver approach via a widget to incorporate proxies and other non-free environments instead of directly importing the keys. The following is an example of a script-based automation method that simplifies key management; however, you may use your preferred method:

wget -qO- 'http://keyserver.debian.com/pks/lookup?op=get&search=0xf5679a222c647c87527c2f8cb00a0bd1e2c63c11' | sudo apt-key add -

7. Hold the currently installed version of the MongoDB package to prevent it from being automatically upgraded to a newer version when updates are installed:

sudo apt-mark hold mongodb-org

8. Open the MongoDB configuration file:

sudo nano /etc/mongod.conf

9. By default, MongoDB only listens locally. You must modify either the bindIp or bindIpAll setting in the MongoDB configuration file so that the port binds to an interface address or a host name or listens on all interfaces. See the example configurations below.

To listen on all interfaces:

net:
  port: 27017
  bindIpAll: true

To bind to a specific interface, like 192.168.50.71:

net:
  port: 27017
  bindIp: 192.168.50.71

To bind to a host name, like graylog01:

net:
  port: 27017
  bindIp: graylog01

10. Create the MongoDB replica set in the MongoDB configuration file:

replSetName: "rs0"

11. On the leader node only, initiate the replica set. In the following example command, graylog01, graylog02, and graylog03 represent the host names of the Graylog/MongoDB nodes. You must update them to reflect the host names or IP addresses of your Graylog/MongoDB nodes:

sudo mongosh
rs.initiate( { _id : "rs0", members: [ { _id: 0, host: "graylog01:27017" }, { _id: 1, host:"graylog02:27017" }, { _id: 2, host: "graylog03:27017" } ] })

12. Enable MongoDB and start the service:

sudo systemctl daemon-reload
sudo systemctl enable mongod.service
sudo systemctl start mongod.service

If this step is successful, the node should return:

{ ok: 1 }

13. Repeat all of the above steps for your additional MongoDB nodes.

1. Install the Data Node package on a select node in your Data Node cluster:

wget https://packages.graylog2.org/repo/packages/graylog-6.1-repository_latest.deb
sudo dpkg -i graylog-6.1-repository_latest.deb
sudo apt-get update
sudo apt-get install graylog-datanode

2. Ensure that the Linux setting vm.max_map_count is set to at least 262144. To check the current value:

cat /proc/sys/vm/max_map_count

To increase the value:

echo 'vm.max_map_count=262144' | sudo tee -a /etc/sysctl.d/99-graylog-datanode.conf
sudo sysctl --system
cat /proc/sys/vm/max_map_count

3. Create your password_secret. This is a secure, randomly generated string used to encrypt sensitive data within the system:

< /dev/urandom tr -dc A-Z-a-z-0-9 | head -c${1:-96};echo;

4. Open the Data Node configuration file:

sudo nano /etc/graylog/datanode/datanode.conf

5. Add the password_secret value to the Data Node configuration file. Note that you must add this same value to the Graylog server configuration files in a later step since it is crucial that this value be the same for all nodes that are members of the Graylog cluster.

6. Configure the MongoDB connection string in the Data Node configuration file. In the following example command, graylog01, graylog02, and graylog03 represent the host names of the MongoDB nodes. You must update them to reflect the host names or IP addresses of your MongoDB nodes:

mongodb_uri = mongodb://graylog01:27017,graylog02:27017,graylog03:27017/graylog?replicaSet=rs0 

7. Ensure the heap settings are set to half your system memory, up to a max of 31 GB. To do so, you will need to add the following configuration property to your Data Node configuration file. For example, if you have 16 GB of RAM, you would set it to 8 GB, like in the example below:

opensearch_heap = 8g

8. Now, open the datanode file found in the default directory at /etc/graylog/datanode/:

sudo nano /etc/graylog/datanode

9. In this file, you must adjust the heap settings in jvm.options for the Data Node service, as noted in Additional Configurations. This service needs very little memory and can be set at 1 or 2 GB, depending on your requirements. It is recommended to set both values to the same size, so setting a minimum of 2 GB for Xms2g and a maximum of 2 GB for Xmx2g would look like the following: 

-Xms2g
-Xmx2g

10. Enable Data Node and start the service:

sudo systemctl daemon-reload
sudo systemctl enable graylog-datanode.service
sudo systemctl start graylog-datanode

11. Follow all of the above steps for each additional Data Node with the exception of generating the password secret. The value you first generated and configured must be the same for each additional node.

1. Install the Graylog package on a select node in your Graylog/MongoDB cluster.

sudo apt-get install graylog-server

sudo apt-get install graylog-enterprise

sudo apt-get install graylog-enterprise

2. Use the following command to create your root_password_sha2. This is the password for the root administrator account for the Graylog interface. Make sure you record this password as you will need it to log into the Graylog interface after preflight configuration: 

echo -n "Enter Password: " && head -1 </dev/stdin | tr -d '\n' | sha256sum | cut -d" " -f1

3. Open the Graylog server configuration file:

sudo nano /etc/graylog/server/server.conf

4. Enter the value you created for the root_password_sha2 in the previous step into the Graylog configuration file.

5. Retrieve the password secret from the Data Node configuration file as indicated in steps 3-5 of Install Data Node and add it to the password_secret property in the Graylog configuration file.

6. Make the following additional adjustments to your Graylog configuration file to optimize the performance and integration of your Graylog cluster.

1. Set the http_bind_address value to the public host name or a public IP address on which the Graylog web and API server will listen for incoming HTTP requests:

   Copy

   http_bind_address = 0.0.0.0:9000

   Hint: More information about these settings can be found in The Web Interface.

2. Configure the MongoDB connection string. In the following example command, graylog01, graylog02, and graylog03 represent the host names of the MongoDB nodes. You must update them to reflect the host names or IP addresses of your MongoDB nodes:

   Copy

   mongodb_uri = mongodb://graylog01:27017,graylog02:27017,graylog03:27017/graylog?replicaSet=rs0 

3. Configure the external URI. The http://graylog.user.com/ value equals the DNS record set on your load balancer:

   Copy

   http_external_uri = http://graylog.user.com/

4. Adjust your journal settings. We recommend you configure your journal settings' max age to 72 hours and the size to your expected total log volume over a 72-hour period divided by the number of nodes in your cluster. So, if your expected daily log volume is 30 GB, your max size over 72 hours would be 90 GB. And if there are three Graylog nodes in your cluster, then you would divide 90 GB by 3, resulting in 30 GB, as in the example below:

   Copy

   message_journal_max_age = 72h
   message_journal_max_size = 30gb

5. On the follower nodes only, ensure that the setting is_leader is set to false.

   Copy

   is_leader = false

Hint: The setting is_leader is set to true by default!

7. Now, open the graylog-server file found by default at /etc/graylog/:

sudo nano /etc/graylog/graylog-server

8. In this file, adjust your heap settings to half the system memory up to a max of 16 GB for the Graylog service, as recommended in Additional Configuration. It is recommended that you set the minimum and maximum values to be the same, so if you are setting your minimum to 2 GB, like shown in -Xms2g, and your maximum to 2 GB, like shown in -Xmx2g, the setting may look like: 

GRAYLOG_SERVER_JAVA_OPTS="-Xms2g -Xmx2g -server -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow"

9. Enable Graylog and start the service:

sudo systemctl daemon-reload
sudo systemctl enable graylog-server.service
sudo systemctl start graylog-server.service

10. Follow all of the above steps for each additional Graylog node with the exception of generating the root administrator password. The value you first generated and configured must be the same for each additional node.

Once installation is complete, proceed to the following section on the preflight login to access the Graylog web interface! 

Troubleshooting and Common Issues

The following section outlines troubleshooting steps for common issues to assist you in resolving potential challenges you may encounter.

Issue: Network Connectivity Issue During MongoDB Installation

It is possible you may receive a network connectivity issue while you are initiating the replica set for MongoDB. This may be related to your DNS and/or your TCP port or network configuration.

Solution: Modify Firewall and/or DNS Settings

If you encounter this issue, we recommend you test whether the required port TCP/27017 is open. You can use telnet or nc (Netcat) to test:

nc -vz <node_ip> 27017