Graylog and Docker

Deploying Graylog in a containerized environment is supported for use with Docker Compose. This guide provides an overview of the fundamental principles for creating and using Docker Compose files to run Graylog. It includes a detailed walkthrough of our example Compose files, along with insights into customization options to tailor your Graylog container to your specific needs.

Prerequisites

Before proceeding, ensure that the following prerequisites are met:

• Familiarity with Docker and Docker Compose is required, including how Compose files are built and maintained. We recommend you review the official Docker documentation before you continue!

Highlights

The following highlights provide a summary of the key takeaways from this article:

• Graylog can be deployed efficiently in a containerized environment using Docker Compose.

• Graylog provides an example docker-compose.yml file, covering the essential services, networks, and volumes, each with attributes that control container behavior.

• Critical environment variables for Graylog and Data Node must be individually defined, including those required for security and connectivity.

• Only expose the ports necessary for your use case to reduce security risks and resource consumption.

• We recommend you configure external volumes for MongoDB, Data Node, and Graylog to persist data across container restarts.

Docker Compose

Docker Compose is a tool used to define and manage multi-container Docker applications. It uses a YAML file (docker-compose.yml) to specify the configuration of services, networks, and volumes required for an application. You can create, start, stop, and scale all containers defined in the file, which simplifies the management of complex applications relying on multiple containers, such as those with databases, web servers, and backend services.

Graylog and Docker Compose

To run Graylog with Docker Compose, you must utilize a docker-compose.yml file. This Compose file is a configuration file written in YAML that defines how Docker containers should be built, configured, and run in a multi-container application.

The Compose file for deploying Graylog includes the required services that constitute the Graylog stack: MongoDB, Data Node, and Graylog.

Graylog Example Compose File

Graylog provides an example YAML file for use with Docker Compose. However, understanding the structure and management of Compose files is crucial, as you will likely need to modify this file to suit your specific Graylog use case. Additionally, note that this Compose file is tailored to support a single Graylog node deployment.

The following sections will review the most important elements and attributes of the Compose file, including their syntax, behavior, and options for customization.

Top-Level Elements

Essentially, the Graylog example Compose file first consists of top-level elements, which are the main categories that define the configuration of the application. Each top-level element is broken down into specific attributes further defining individual properties or configurations that control how containers, networks, and volumes behave.

The following key top-level elements are used to build the Graylog service in Docker Compose:

• Services: Defines the individual services (i.e. containers) that make up your application.

• Networks: Defines custom networks for inter-container communication.

• Volumes: Defines named volumes for persistent data storage.

Now, let’s review the essential attributes needed to define configurations for each of these elements.

Services Attributes

Services are the core building blocks that define the individual containers you want to run as part of your application. Each service corresponds to a container and has several attributes that determine its behavior:

• Image: Specifies the Docker image to use for the service, i.e. Graylog, Data Node, and MongoDB.

• Hostname: Declares a hostname to use for the service container.

• Environment: Defines specific environment variables set in the container based on the configuration settings from the Graylog and Data Node server configuration files.

  Hint: See the following section on important environment attributes recommended for Graylog.

• Ulimits: Overrides the default user limits for a container. User limits manage and restrict the resources available to a shell session or a process started by the shell.

• Ports: Sets the port mappings between the host machine and the containers. See Default Ports for a list of ports required for use with Graylog.

  Hint: See the following section on default ports configured in the example Compose file.

• Restart: Configures the restart policy for the container.

• Networks: Defines the networks to which the service containers are attached.

• Volumes: Determines the mounted host paths or named volumes that are accessible by service containers.

• Depends_on: Specifies service dependencies, indicating the order in which services should start.

• Entrypoint: Declares the default entry point for the service container.

Networks Attributes

The networks element allows you to define custom Docker networks for your application. Each network can include specific attributes to customize its configuration. These attributes are useful for controlling network isolation, drivers, and options. In the example Compose file, one network is created and one driver configuration specified: 

• Graylog: Creates the graylog network to which all your individual containers should be attached.

• Driver: Specifies which driver should be used for this network. The bridge driver is the default network driver in Docker.

Volumes Attributes

The volumes section is used to define data volumes, which allow containers to persist data or share data between multiple containers. For Graylog, the following volumes have been defined in the example Compose file; however, these can be customized to map to your individual services:

• mongodb_data

• mongodb_config

• graylog-datanode

• graylog_data

Graylog Environment Variables

Within the services element, there are many variables that you can set within the Compose file. These variables are based on the configuration settings used in the Graylog server and Data Node configuration files. Several key environment variables are included in the example Compose file, but many more may be defined by you.

Key Environment Variables

Key environment variables are defined for both Data Node and Graylog services in the example Compose file. These are the variables that we recommend you set at before installing Graylog in Docker Compose, though note that some are listed as required since the Graylog service will not start without them! There are two services for which key variables must be defined: Data Node and Graylog.

Data Node Environment Variables

The following Data Node environment variables are included in the example Compose file:

Graylog Environment Variables

The following Graylog environment variables are included in the example Compose file:

Custom Environment Variables

Additionally, you can add environment variables to the Compose file based on any of the configuration settings listed in the server configuration file, allowing you to adjust your Graylog service to suit your needs.

To do so, note that the syntax used in the Graylog configuration file must be adjusted for use with Docker Compose by adding the prefix GRAYLOG_ to the Graylog service settings or GRAYLOG_DATANODE_ to the Data Node service settings. Additionally, ensure that you capitalize the variable name in its entirety.

For example, to adjust the default timeout for outgoing HTTP connections, the original configuration parameter in the Graylog server configuration file would read:

http_connect_timeout

However, for use in the Compose file under the Graylog service element, this parameter should read:

GRAYLOG_HTTP_CONNECT_TIMEOUT

Example Custom SMTP Configuration for Alerts

Custom variables can be added to perform more complex configuration settings in the Compose file, like setting up the SMTP configuration for sending Graylog alerts via email. Using the properties found in the Graylog server configuration file as a base, you could add variables to the docker-compose.yml that may look something like this:

GRAYLOG_TRANSPORT_EMAIL_ENABLED: "true"

GRAYLOG_TRANSPORT_EMAIL_HOSTNAME: smtp

GRAYLOG_TRANSPORT_EMAIL_PORT: 25

GRAYLOG_TRANSPORT_EMAIL_USE_AUTH: "false"

GRAYLOG_TRANSPORT_EMAIL_USE_TLS: "false"

GRAYLOG_TRANSPORT_EMAIL_USE_SSL: "false"

Configuration via the Web Interface

It is possible to manage most of your essential Graylog configuration settings via the web interface when running Graylog in Docker Compose. See Initial Configuration Settings for more information on this process.

Ports Configuration

The example Compose file includes several ports that need to be mapped from the container to the host system to enable access to various services. For example, ports like 9000 are used for the Graylog web interface, while others, such as 5140, are commonly used for receiving syslog messages over TCP or UDP. When configuring these ports in your Compose file, it is important to only expose the ports that are essential for your specific use case. For a list of all default ports for use with Graylog, see Default Ports.

The following is a list of the ports included by default in the example Compose file. Note that some ports are commented out and need to be included manually if required.

MongoDB Ports

• 27017:27017/tcp: Sets the port mappings between the host machine and the containers

Data Node Ports

• 8999:8999/tcp: Required for access to the Data Node API.

• 9200:9200/tcp: Required for access to the OpenSearch API.

• 9300:9300/tcp: Required for OpenSearch cluster communication. (Thus port is only required if you use self-managed OpenSearch.)

Graylog Ports

• 5044:5044/tcp: Used for Beats inputs.

• 5140:5140/tcp: Used for Syslog TCP inputs.

• 5140:5140/udp: Used for Syslog UDP inputs.

• 5555:5555/tcp: Used for RAW TCP inputs.

• 5555:5555/udp: Used for RAW UDP inputs.

• 9000:9000/tcp: Required for the Graylog REST API.

• 2201:12201/tcp: Used for GELF TCP inputs.

• 12201:12201/ud: Used for GELF UDP inputs.

• 10000:10000/tcp: Used for custom TCP inputs. (Note this is commented out by default.)

• 10000:10000/udp: Used for custom UDP inputs. (Note this is commented out by default.)

• 13301:13301/tcp: Required for use with the Graylog Forwarder.

• 13302:13302/tcp: Sets the mapping for Graylog Forwarder configuration.

Persisting Data

Docker containers are ephemeral by default; any data written inside the container is lost when the container stops or is deleted. To persist data, you will need to use external volumes for data storage. In the case of a container restart, this will simply re-use the existing data from the former instances.

Persisting Data Examples

Using Docker volumes for MongoDB, Data Node, and Graylog, the docker-compose.yml may be modified as follows for each service when utilizing external volumes:

MongoDB

volumes:
  - "/your/local/path:/data/db"
  - "/your/local/path:/data/configdb"

Data Node

volumes:
  - "/your/local/path:/var/lib/graylog-datanode/opensearch/data"

Graylog

volumes:
  - "/your/local/path:/usr/share/graylog/data"

Manage the Compose File

The Compose file can be stored and managed in various ways depending on your unique requirements.

Local File Storage

This is the most common method and the one generally recommended by Graylog. You can create and store the docker-compose.yml file on your local filesystem. For information on creating and accessing this file during deployment, see Run Graylog in Docker.

Remote Access

It is also possible to maintain the docker-compose.yml in a central repository, like GitHub, by overwriting the bundled configuration files. Note that the bundled configuration files are stored in /usr/share/graylog/data/config/ inside the Docker container.

Remote Access Example

If you prefer to manage the Compose file in GitHub, you will need to create the new configuration directory next to the docker-compose.yml file and copy the default files from GitHub, using a command like the following:

mkdir -p ./graylog/config
cd ./graylog/config
wget https://raw.githubusercontent.com/Graylog2/graylog-docker/6.1/config/graylog.conf
wget https://raw.githubusercontent.com/Graylog2/graylog-docker/6.1/config/log4j2.xml

You will then need to mount this newly created directory ./graylog/config/ into the Graylog Docker container. This can be done by adding an entry to the volumes section of the docker-compose.yml, like seen below:

services:
  mongodb:
    image: "mongo:6.0.18"
# Other settings [...]
  datanode:
    image: "graylog/graylog-datanode:6.1"
# Other settings [...]
  graylog:
    image: "graylog/graylog-enterprise:6.1"
# Other settings [...]
volumes:
# Mount local configuration directory into Docker container
- ./graylog/config:/usr/share/graylog/data/config

Docker Configurations (Swarm Mode)

In a Docker Swarm setup, you can also store the YAML file as part of the Swarm configuration. This has the added benefit of being able to utilize Docker secrets if desired. For more information, see the Docker documentation for Swarm mode.

Pre-Release Versions of Graylog

You can use Docker to test pre-release versions of Graylog. The pre-releases are tagged in the graylog/graylog or graylog/graylog-datanode Docker images. Select from the available tags for the Data Node image on Docker Hub. For example:

datanode:
    image: "graylog/graylog-datanode:6.1.0-beta.1-1"
graylog:
    hostname: "server"
    image: "graylog/graylog-enterprise:6.1.0-beta.1-1"

Further Reading

Explore the following additional resources and recommended readings to expand your knowledge on related topics: