Graylog Sidecar is a lightweight configuration management system for different log collectors, also called Backends. The Graylog node(s) acts as a centralized hub containing the configurations of log collectors. On supported message-producing devices/hosts, Sidecar can run as a service (Windows host) or daemon (Linux host).

The log collector configurations are centrally managed through the Graylog web interface. Periodically, the Sidecar daemon will fetch all relevant configurations for the target, using the REST API. On its first run or when a configuration change has been detected, Sidecar will generate (render) relevant backend configuration files. Then it will start, or restart, those reconfigured log collectors.

Installation

You can get .deb and .rpm packages for Graylog Sidecar in our package repository. For Windows, you can download the installer from here.

Please follow the version matrix to pick the right package:

All following commands should be executed on the remote machine which is where you want to collect log data from.

Install the Sidecar

Ubuntu

Install the Graylog Sidecar repository configuration and Graylog Sidecar itself with the following commands:

wget https://packages.graylog2.org/repo/packages/graylog-sidecar-repository_1-2_all.deb
sudo dpkg -i graylog-sidecar-repository_1-2_all.deb
sudo apt-get update && sudo apt-get install graylog-sidecar

Edit the configuration (see Configuration) and activate the Sidecar as a system service:

vi /etc/graylog/sidecar/sidecar.yml

sudo graylog-sidecar -service install

# Ubuntu 14.04 with Upstart
sudo start graylog-sidecar

# Ubuntu 16.04 and later with systemd
sudo systemctl enable graylog-sidecar
sudo systemctl start graylog-sidecar

CentOS

Install the Graylog Sidecar repository configuration and Graylog Sidecar itself with the following commands:

sudo rpm -Uvh https://packages.graylog2.org/repo/packages/graylog-sidecar-repository-1-2.noarch.rpm
sudo yum install graylog-sidecar

Edit the configuration (see Configuration) and activate the Sidecar as a system service:

vi /etc/graylog/sidecar/sidecar.yml

sudo graylog-sidecar -service install

sudo systemctl enable graylog-sidecar
sudo systemctl start graylog-sidecar

Windows (Preparation)

To get started with Sidecar on Windows be sure these requirements are met:

• When choosing what version of Sidecar to install, you must choose the version based on the Graylog version you are currently using. You can find the Windows Sidecar install link on our Github page.
• Graylog Sidecar service is successfully installed inside your Windows OS.
• Your Graylog instance is up and running alongside your Windows OS.

Before starting the procedure to set up Sidecar on Windows, configure your input to receive Windows Sidecar logs on port 5044.

1. Navigate to System > Inputs.
2. Select an input from the first dropdown menu on the Inputs screen.
3. Select Beats.
4. Click the Launch new input button to prompt a new form.
5. Check the Global box.
6. Ensure the port field is set to 5044.

Installing Collectors

Graylog contains default collector configurations for Filebeat, Winlogbeat (mentioned above), and NXLog.

Next up, you can decide which collectors you want to use with your Sidecar and install them. We only cover the installation of the most common ones here, but you are free to use other collectors as well. But since you’re able to define your own collector backends, you could run e.g. sysmon, auditd, or packetbeat.

Beats on Linux

Install Filebeat or another Beats package by following the instructions on the official Filebeat download page.

NXLog on Ubuntu

Install the NXLog package from the official NXLog download page. Because the Sidecar takes control of stopping and starting NXlog it is necessary to stop all running instances of NXlog and deconfigure the default system service:

sudo /etc/init.d/nxlog stop
sudo update-rc.d -f nxlog remove
sudo gpasswd -a nxlog adm
sudo chown -R nxlog.nxlog /var/spool/nxlog

NXLog on CentOS

The same on a RedHat based system:

sudo service nxlog stop
sudo chkconfig --del nxlog
sudo gpasswd -a nxlog root
sudo chown -R nxlog.nxlog /var/spool/nxlog

NXlog on Windows

Install the NXLog package from the official download page and deactivate the system service. We just need the binaries installed on the system:

"C:\Program Files (x86)\nxlog\nxlog" -u

(Prefix the commands with & when using PowerShell)

Sidecar Configuration

On the command line you can provide a path to the configuration file with the -c switch. The default configuration path on Linux systems is /etc/graylog/sidecar/sidecar.yml and C:\\Program Files\\Graylog\\sidecar\\sidecar.yml on Windows.

Most configuration parameters come with built-in defaults. The only parameters that need adjustment are server_url and server_api_token. You can get your API token by following the link on the Sidecars Overview page.

sidecar.yml Reference

First Start

Once you have installed the Sidecar package and start the service for the first time, you can verify that it shows up in the Sidecars Overview page. A new sidecar instance will not have any configurations assigned yet. Take the Step-by-step guide to create your first configuration.

Mode of Operation

When the Sidecar is assigned a configuration via the Graylog web interface, it will write a configuration file into the collector_configuration_directory for each collector backend, e.g. if you assigned a Filebeat collector, then you will find a filebeat.yml file in that directory. All changes have to be made in the Graylog web interface.

Every time the Sidecar detects an update to its configuration it will rewrite the corresponding collector configuration file. Manually editing these files is not recommended.

Every time a collector configuration file is changed the collector process is restarted. The Sidecar takes care of the collector processes and reports the status back to the web interface.

Sidecar Status

Each Sidecar instance is able to send status information back to Graylog. By enabling the option send_status metrics like load or the IP address of the host Sidecar is running on are sent. Also metrics that are relevant for a stable operation e.g. disk volumes with over 75% utilization are included.

Additionally with the list_log_files option a directory listing is displayed in the Graylog web interface. This way an administrator can see which files are available for collecting. The list is periodically updated and files with write access are highlighted for easy identification. After enabling send_status or send_status + list_log_files go to the collector overview and click on one of them, a status page with the configured information will be displayed.

Step-by-Step Guide

We have prepared an example on how to configure the Sidecar using the Graylog web interface. The example describes a winlogbeats configuration.

This assumes you have a Beats input already configured.

Create an API Token

1. Navigate to System > Sidecars, and click the explainer text: Create or reuse a token for the graylog-sidecar user.

2. Enter your choice name into the Token Name field.
3. Click the Create Token button.

4. Save the API server token in a safe yet accessible location in case you need to retrieve it again.

Configure the Sidecar Service on WindowsOS

Now that you have access to an API Token, you need to run the Windows Sidecar installer.

Complete the following steps:

1. Enter the URL in your Graylog API, it should be pre configured as (https://127.0.0.1:9000/api).
2. Name your Sidecar instance.
3. Enter your server API token that you created earlier.
4. Click Install to close the installer.

Once finished, you can change or configure your sidecar.yml file, which should be located in C:\\Program Files\\Graylog\\sidecar\\sidecar.yml.

Configuring the Winlogbeat Collector

1. Navigate back to your Graylog instance.
2. Go to System > Sidecars within your Graylog instance and select the configuration tab in the left hand corner, then click the Create Configuration tab.
3. Select winlogbeat on Windows from the Collector dropdown menu.

4. Enter this configuration script in the Configuration field:

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}
output.logstash:
  hosts: ["<your_graylog_ip>:5044"]
path:
  data: C:\Program Files\Graylog\sidecar\cache\winlogbeat\data
  logs: C:\Program Files\Graylog\sidecar\logs
tags:
  - windows
winlogbeat:
  event_logs:
    - name: Application
    - name: System
    - name: Security

This is a configuration Graylog pre-builds for you.
5. Add a name, and (optionally) apply a custom color to the configuration.
6. Click Create.

7. Verify the new Sidecar/winlogbeat configuration you created is listed in the Configurations menu.

Install and Start the Service

Now, open a command prompt window using administrator rights. Then, perform the following steps:

1. Run these commands: (Prefix the commands with & when using PowerShell)

"C:\Program Files\graylog\sidecar\graylog-sidecar.exe" -service install
"C:\Program Files\graylog\sidecar\graylog-sidecar.exe" -service start

2. Navigate back to the Graylog UI.
3. Locate your Windows device on the Sidecar sub-menu (under System).

5. Select the winlogbeat collector underneath the Windows Sidecar machine on the left-side, and on the Configure drop down on the right-hand side select the windows_sidecar configuration that we set up earlier.

6. Click the Process drop down on the right-hand side and select Start after choosing your configuration.

Your Graylog instance should now be successfully started and collecting logs from your Windows machine.

Using Configuration Variables

Configuration variables can contain arbitrary strings like the IP address of your Graylog server or the port of an input. The variables can then be used in multiple collector configurations, which avoids duplication and simplifies management.
To create a configuration variable go any Collector Configuration page:

On the right you’ll find a box Collector Configuration Reference which contains and. Click on Variables and then Create Variable
to receive the following modal:

In this example we replace the hard coded IP and Port from our Beats input with a new variable named ${user.BeatsInput}:

We can now use this variable in all our configurations. If we ever need to change the IP/port of our input, we just change the variable.

Runtime Variables

Runtime variables contain runtime informations from each Sidecar that is requesting this configuration. An important example is the ${sidecar.nodeId} variable. The collector configuration should contain an instruction to fill that variable in an extra field. This allows Graylog to relate messages to the Sidecar that produced them. (This is what makes the Show messages button on the Sidecars overview page work.)

Secure Sidecar Communication

The Communication between Sidecar and Graylog will be secured if your API uses SSL . To secure the communication between the Collector and Graylog you just need to mark Enable TLS in your Beats Input. Without giving additional Information, Graylog will now create a self-signed certificate for this Input. Now in the Sidecar Beats Output Configuration you just mark Enable TLS Support and Insecure TLS connection. After this is saved, the communication between Beats and Graylog will use TLS.

Certificate Based Client Authentication

If you want Graylog to only accept data from authenticated Collectors please follow the steps at Secured Graylog and Beats input.

Run Sidecar as Non-Root User

The default is that the Sidecar is started with the root user to allow access to all log files. But this is not mandatory. If you like to start it with a daemon user, proceed like the following:

• Create a daemon user, e.g. sidecar. 

The Sidecar itself is accessing the following files and directories:

• sidecar.yml - /etc/graylog/sidecar/sidecar.yml
• collector_configuration_directory - /var/lib/graylog-sidecar/generated/
• node_id - /etc/graylog/sidecar/node-id
• cache_path - /var/cache/graylog-sidecar/
• log_path - /var/log/graylog-sidecar/

chown -R sidecar /etc/graylog/sidecar
• chown -R sidecar /var/cache/graylog-sidecar
• chown -R sidecar /var/lib/graylog-sidecar
• chown -R sidecar /var/log/graylog-sidecar

You can change all paths to different places in the file system. If you prefer to store all Sidecar data in the home directory of the sidecar user, just change the paths accordingly.

Now systemd needs to know that the Sidecar should be started with a non-root user. Open /etc/systemd/system/collector-sidecar.service with an editor and navigate to the [Service] section, add:

User = sidecar
Group = sidecar 

To make use of these settings reload systemd:

sudo systemctl daemon-reload
sudo systemctl restart graylog-sidecar

Check the log files in /var/log/graylog-sidecar for any errors. Understand that not only the Sidecar but also all backends, like filebeat, will be started as sidecar user after these changes. So all log files that the backend should observe also need to be readable by the sidecar user. Depending on the Linux distribution there is usually an administrator group which has access to most log files. By adding the sidecar user to that group you can grant access fairly easy. For example on Debian/Ubuntu systems this group is called adm (see System Groups in Debian Wiki or Security/Privileges - Monitor system logs in Ubuntu wiki).

Upgrading from the Collector Sidecar

This guide describes how you can perform an upgrade from the deprecated Collector Sidecars (0.1.x) to the new Sidecars (1.x).
One major difference between the old and the new Sidecars, is that we replaced the UI based collector configuration approach with one where you can manage the plain text configuration of the collectors directly. This might seem like an inconvenience at first, but gives you the flexibility to configure any collector backend you want.

Additionally, the new Sidecars don’t assign configurations based on tags anymore. Instead you have to assign configurations explicitly (see Step-by-Step guide).

1. Install New Sidecar

The new Sidecar has different paths and executable names, so it can coexist with the old one. Install the new Sidecar by following the Installation instructions and have your Sidecar running as described in First Start.

2. Migrate Configuration

Next, we need to migrate the configuration that was previously rendered on each host by the Collector Sidecar, to a new Collector Configuration.
We recommend to use the Sidecar Configuration Migrator. However, retrieving the old configuration can also be done manually by fetching it from your host at the /etc/graylog/collector-sidecar/generated/ directory.

3. Adopt Configuration to Graylog 3.0

There are a few things that might need attention after an upgrade:

• Use Runtime variables for static fields
  The imported configuration contains instructions that add static fields which allows Graylog to relate messages to a Sidecar. You should replace the hardcoded values of gl2_source_collector and collector_node_id with runtime variables.

In case of a Beats collector this would be:

fields.gl2_source_collector: ${sidecar.nodeId}
fields.collector_node_id: ${sidecar.nodeName}

• Migrate to the new Beats input
  Graylog 3.0 comes with a new Beats input. The former one was renamed to Beats(deprecated). The new input handles fields a little different. Therefore you should define fields_under_root: true for the new input to get the Graylog fields work.

4. Switch Over to the New Sidecar

Once you’re done creating a new configuration, you can assign it to your Sidecar (see Step-by-Step guide). If everything works as expected, make sure to uninstall the old Collector Sidecar to avoid collecting your logs twice.

Sidecar Configuration Migrator

The task of the Sidecar configuration migrator is to extract the configuration from existing Collector Sidecars and convert it into new Sidecar configurations.

This feature needs a Collector Sidecar with version 0.1.8 or greater. Please upgrade the instance you want to import configurations from, if necessary.

• Navigate to the Collectors (legacy) overview. In your Graylog web interface click on System / Collectors (legacy).

• Click on the name of the Collector you want to import configurations from.

• Click the Import Configuration button on a backend to import a configuration. If the import was successful, follow the link to create a new Sidecar configuration:

• After clicking on Create Configuration use the Migrate button underneath the configuration editor:

• A window opens up and lets you pick already imported configurations. Clicking Apply will paste the configuration into the editor. Afterwards, you can edit and save the configuration as usual.

Testing Sidecar

To ensure that our Graylog instance is collecting our Windows logs:

1. Go to the Overview tab underneath Sidecars.
2. Select the Show messages button on the right-hand side.

Now, it shows the logs that are coming in from our Windows Machine.

The image below shows a more detailed example of what your search page should look like when you view your incoming logs from your Sidecar.

Sidecar Glossary

The following section is an explanation of terms that refer to different parts of the Graylog Sidecar.

Configuration

A configuration is the representation of a log collector configuration file in the Graylog web interface. A configuration can be assigned to Sidecars, which also assigns the corresponding collector. You can have multiple configurations for a single log collector. However, you can not assign the same collector twice to a Sidecar.

Inputs

Collectors ingest data through inputs. An input can be a log file that the collector should continuously read or a connection to the Windows event system that emits log events. An input is connected to an output, otherwise there would be no way of sending the data to the next hop. So first create an output and then associate one or many inputs with it.

Debug

The Sidecar writes log files to the directory configured in log_path. There is one file for each backend. You can check for general issues like file permissions or log transmission problems. The Sidecar itself writes to sidecar.log. Problems like failed connection to the Graylog API can be found here.

You can also start the Sidecar in the foreground and monitor the output of the process:

graylog-sidecar -debug

Uninstall

In Linux, just uninstall the package, to perform an uninstall in Windows run:

"C:\Program Files\Graylog\graylog-sidecar.exe" -service stop 
"C:\Program Files\Graylog\graylog-sidecar.exe" -service uninstall

(Prefix the commands with & when using PowerShell)

Known Problems

Currently we know of two problems with NXLog:

• Since version 2.9.17, timestamps are transmitted without millisecond precision.
• In Windows machines NXlog is not able to store its collector state so features like file tailing don’t work correctly in combination with Sidecar. Use Sidecar version 0.1.0-alpha.1 or newer.

A known issue if you use a loadbalancer or firewall in front of Graylog’s API:

• The Sidecar uses a persistent connection for API requests. Therefore it logs 408 Request Time-out if the load balancer session or timeout is lower than the configured update_interval.