Graylog Sidecar is a lightweight configuration management system for log collectors, which are also called backends. These collectors ingest data through inputs. An input can be a log file that the collector continuously reads or a connection to the Windows event system that emits log events. The Graylog node(s) acts as a centralized hub containing the configurations of log collectors. On supported message-producing devices/hosts, sidecar can run as a service (Windows host) or daemon (Linux host).

Log collector configurations are centrally managed through the Graylog web interface. Periodically, the sidecar daemon will fetch all relevant configurations for the target, using the REST API. On its first run or when a configuration change has been detected, sidecar will generate (render) relevant backend configuration files. Then it will start or restart those reconfigured log collectors.

The following guide describes the Graylog sidecar on-premise configuration. For information regarding the sidecar in Graylog Cloud, see the related article.

Sidecar Installation

Prerequisites

Before installing sidecar, make sure that you have a Beats input in order to receive data from the Beats collector. Configure your input to receive sidecar logs on port 5044. See the following instructions:

• Navigate to System > Inputs.
• Select an input from the first drop down menu on the Inputs screen.
• Select Beats.
• Click the Launch new input button to prompt a new form.
• Check the Global box.
• Ensure the port field is set to5044.
• The variable ${user.graylog_host}should match the IP address.

Sidecar Version

You can find .deb and .rpm packages for Graylog sidecar in our package repository.

Please follow the version matrix to pick the right package which you can download from here.

Installation by OS

Ubuntu & Debian Installation

Install the Graylog sidecar repository configuration and Graylog sidecar itself with the following commands:

wget https://packages.graylog2.org/repo/packages/graylog-sidecar-repository_1-5_all.deb
sudo dpkg -i graylog-sidecar-repository_1-5_all.deb
sudo apt update && sudo apt install graylog-sidecar            

Edit the configuration (see Configuration ) and activate the sidecar as a system service:

sudo vi /etc/graylog/sidecar/sidecar.yml

sudo graylog-sidecar -service install

# Ubuntu 14.04 with Upstart
sudo start graylog-sidecar

# Ubuntu 16.04 and later with systemd
sudo systemctl enable graylog-sidecar
sudo systemctl start graylog-sidecar

CentOS & RedHat Installation

Install the Graylog sidecar repository configuration and Graylog sidecar itself with the following commands:

sudo rpm -Uvh https://packages.graylog2.org/repo/packages/graylog-sidecar-repository-1-5.noarch.rpm
sudo yum install graylog-sidecar
            

Edit the configuration (see Sidecar Configuration) and activate the sidecar as a system service:

vi /etc/graylog/sidecar/sidecar.yml

sudo graylog-sidecar -service install

sudo systemctl enable graylog-sidecar
sudo systemctl start graylog-sidecar        

SUSE Installation

sudo rpm -Uvh https://packages.graylog2.org/repo/packages/graylog-sidecar-repository-1-5.noarch.rpm
sudo mv /etc/yum.repos.d/* /etc/zypp/repos.d/
sudo zypper up
sudo zypper install graylog-sidecar
Continue? y
Do you want to reject the key, or trust always? [r/a/?] (r): a

Edit the configuration (see Sidecar Configuration) and activate the sidecar as a system service:

sudo vi /etc/graylog/sidecar/sidecar.yml
sudo graylog-sidecar -service install

sudo systemctl enable graylog-sidecar
sudo systemctl start graylog-sidecar

Windows Installation

Install Windows sidecar using the installation link on our Github page. The Windows installer can be run interactively by running:

$ graylog_sidecar_installer_1.4.0-1.exe

Or you could run it in silent mode using:

$ graylog_sidecar_installer_1.4.0-1.exe /S -SERVERURL=http://your Graylog IP address or DNS name/api -APITOKEN=yourapitoken

The Windows Installer supports additional options in silent mode. Examples of these parameters are: -TAGS=["example","IIS"] -NODENAME=mynodename -NODEID=1234 -SENDSTATUS=false -TLSSKIPVERIFY=true -UPDATEINTERVAL=10s.

Installing Collectors

Graylog contains default collector configurations for Filebeat, Winlogbeat, and Auditbeat. You can also decide which collectors you want to use with your sidecar and install them. We provide information about installing NXLog here, but you are free to install other collectors as well. Since you are able to define your own collector backends, you could run sysmon, auditd, packetbeat, etc.

NXLog on Ubuntu

Install the NXLog package from the official NXLog download page. Because the sidecar takes control of stopping and starting NXlog, it is necessary to stop all running instances of NXlog and unconfigure the default system service:

sudo /etc/init.d/nxlog stop
sudo update-rc.d -f nxlog remove
sudo gpasswd -a nxlog adm
sudo chown -R nxlog.nxlog /var/spool/nxlog    

NXLog on CentOS

Stop all running instances of NXlog and unconfigure the default system service on a RedHat based system:

sudo service nxlog stop
sudo chkconfig --del nxlog
sudo gpasswd -a nxlog root
sudo chown -R nxlog.nxlog /var/spool/nxlog            

NXlog on Windows

Install the NXLog package from the official download page and deactivate the system service. We just need the binaries installed on the system:

"C:\Program Files (x86)\nxlog\nxlog" -u
            

Sidecar Configuration

On the command line you can provide a path to the configuration file with the-c switch. The default configuration path on Linux systems is /etc/graylog/sidecar/sidecar.yml and C:\\Program Files\\Graylog\\sidecar\\sidecar.yml on Windows.

Most configuration parameters come with built-in defaults. The only parameters that need adjustment are server_url and server_api_token. You can get your API token by following the link in Graylog on the System>Sidecars page. Remember to save the API server token as you may need it during the installation process.

sidecar.yml Reference

Upgrading from the Collector Sidecar

This guide describes how you can perform an upgrade from the deprecated Collector Sidecars (0.1.x) to the new Sidecars (1.x). We still support the old Collector Sidecarswhich can be found in the System / Collectors (legacy) menu entry. We encourage users to migrate to the new sidecar which is covered by this document.
One major difference between the old and the new sidecars, is that we replaced the UI based collector configuration approach with one where you can manage the plain text configuration of the collectors directly. This might seem like an inconvenience at first, but gives you the flexibility to configure any collector backend you want.

1. Install the New Sidecar

The new sidecar has different paths and executable names, so it can co-exist with the old one. Install the new sidecar by following the installation instructions and have your sidecar running as described in First Start.

2. Migrate the Configuration

Next, we need to migrate the configuration that was previously rendered on each host by the Collector Sidecar, to a new Collector Configuration.
We recommend using the Sidecar Configuration Migrator. However, retrieving the old configuration can also be done manually by fetching it from your host in the /etc/graylog/collector-sidecar/generated/directory.

3. Adopt a Configuration

There are a few things that might need attention after an upgrade to 3.0:

• Use Runtime variables for static fields:
  The imported configuration contains instructions that add static fields which allow Graylog to relate messages to the sidecar. You should replace the hardcoded values of gl2_source_collector and collector_node_id with runtime variables.

For a Beats collector this would be:

fields.gl2_source_collector: ${sidecar.nodeId}
fields.collector_node_id: ${sidecar.nodeName}

• Migrate to the new Beats input:
  As of Graylog 3.0, we have a new Beats input. The former one was renamed Beats(deprecated). The new input handles fields differently. Therefore you should define fields_under_root: true for the new input in order for Graylog fields to work.

4. Switch Over to the New Sidecar

Once you’re done creating a new configuration, you can assign it to your sidecar (see Step-by-Step guide). Make sure to uninstall the old Collector Sidecar to avoid collecting your logs twice.

Ready-to-Use Sidecar Configurations

Several default collector configurations have been added to the sidecar in Graylog 5.2. These default configurations are ready to use and will be automatically assigned once the sidecar is installed. They will immediately start collecting data such as event logs or audit framework data.

Go to System > Sidecars and click on the Overview tab. Click on your sidecar to view the collectors that will be running by default as seen below.

Previously you needed to install collectors before being able to fully utilize the sidecar. Sidecar version 1.5 ships with the collectors shown below:

You may edit the configuration file according to your preferences. Please refer to the section on Sidecar Configuration for further details. If you wish to terminate the default configurations, delete the default tag in the configuration file. This way, the configuration will not apply to the sidecar.

You may view and edit all default sidecar configurations on the System > Sidecars page under Collector Configurations.

Using Sidecar

First Start

Once you have installed the sidecar package and start the service for the first time, verify that it shows up in the Sidecars Overview page. A new sidecar instance will have default collector configurations assigned. Follow the Step-by-Step Guide below to configure your instance.

Mode of Operation

• When the sidecar is assigned a configuration via the Graylog web interface, it will write a configuration file into the collector_configuration_directory for each collector backend. For example, if you assigned a Filebeat collector, you will find a filebeat.yml file in that directory. All changes have to be made in the Graylog web interface.

• Every time the sidecar detects an update to its configuration, it will rewrite the corresponding collector configuration file. Manually editing these files is not recommended.

• Every time a collector configuration file is changed, the collector process is restarted. The sidecar takes care of the collector processes and reports the status back to the web interface.

Sidecar Status

Each sidecar instance is able to send status information back to Graylog. By enabling the option send_status, metrics like load or the IP address of the host sidecar is running on are sent. Also metrics that are relevant for a stable operation, e.g. disk volumes with over 75% utilization are included.

Additionally with the list_log_ files option, a directory listing is displayed in the Graylog web interface. This way an administrator can see which files are available for collecting. The list is periodically updated and files with write access are highlighted for easy identification. After enabling send_status or send_status + list_log_, files go to the Collector Overview. If you click on one of them, a status page with the configured information will be displayed.

Failure Tracking

If you are managing a large deployment with numerous sidecars, it may be a daunting task to analyze the reason for each collector’s failure individually. This is why we have incorporated a Failure Tracking page into sidecars.

This searchable and sortable page will display the name, status, error message, and verbose error message of each collector.

Step-by-Step Guide

We have prepared an example of how to configure sidecar using the Graylog web interface. The example describes a Winlogbeat configuration.

This assumes you have a Beats input already configured.

Create an API Token

1. Navigate to System > Sidecars, and click on Create or reuse a token for the graylog-sidecar user.

2. Enter your choice name into the Token Name field.
3. Click the Create Token button.

4. Save the API server token in a safe yet accessible location in case you need to retrieve it again.

Configure the Sidecar Service on Windows OS

Now that you have access to an API Token, you need to run the Windows sidecar installer.

Complete the following steps:

1. Enter the URL in your Graylog API, it should be pre-configured as (https://127.0.0.1:9000/api).
2. Name your sidecar instance.
3. Enter your server API token that you created earlier.
4. Click Install to close the installer.

Once finished, you can change or configure your sidecar.yml file, which should be located in C:\\Program Files\\Graylog\\sidecar\\sidecar.yml.

Configuring the Winlogbeat Collector

1. Navigate back to your Graylog instance.
2. Go to System > Sidecars within your Graylog instance and select the configuration tab in the left hand corner, then click the Create Configuration tab.
3. Select winlogbeat on Windows from the Collector drop-down menu.

4. Enter this configuration script in the Configuration field:

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

output.logstash:
 hosts: ["<your_graylog_ip>:5044"]
path:
 data: C:\Program Files\Graylog\sidecar\cache\winlogbeat\data
 logs: C:\Program Files\Graylog\sidecar\logs
tags:
 - windows
winlogbeat:
 event_logs:
  - name: Application
  - name: System
  - name: Security        

This is a configuration Graylog pre-builds for you.

5. Add a name, and (optionally) apply a custom color to the configuration.
6. Click Create.

7. Verify that the new Sidecar/winlogbeat configuration is listed in the Configurations menu.

Install and Start the Service

Now, open a command prompt window using administrator rights. Then, perform the following steps:

1. Run these commands (prefix the commands with&if you are using PowerShell):

"C:\Program Files\graylog\sidecar\graylog-sidecar.exe" -service install
"C:\Program Files\graylog\sidecar\graylog-sidecar.exe" -service start
            

2. Navigate back to the Graylog UI.
3. Locate your Windows device on the Sidecar sub-menu (under System).
4. Select the winlogbeat collector which is under the Windows Sidecar machine on the left. Select the windows_sidecar configuration (which you had set up earlier), which can be found under the Configure drop-down menu, on the right-hand side.

6. Click the Process drop-down menu on the right-hand side, choose a configuration and select Start.

Your Graylog instance will now start and collect logs from your Windows machine.

Using Configuration Variables

Configuration variables can contain arbitrary strings like the IP address of your Graylog server or the port of an input. The variables can then be used in multiple collector configurations, which avoids duplication and simplifies management.

1. Go to the Collector Configuration page to create a configuration variable.

2. On the right you’ll find a box named Collector Configuration Reference.

3. Click on Variables and then on Create Variable.

4. You will see the following modal:

In this example we replace the hard coded IP and Port from our Beats input with a new variable named ${user.BeatsInput}:

We can now use this variable in all our configurations. If we ever need to change the IP/port of our input, we just change the variable.

Runtime Variables

Runtime variables contain runtime information from each sidecar that is requesting this configuration. An important example is the ${sidecar.nodeId}variable. The collector configuration should contain an instruction to fill that variable in an extra field. This allows Graylog to relay messages to the Sidecar that produced them. (This is what makes the Show messages button on the Sidecars overview page work.

Secure Sidecar Communication

The Communication between sidecar and Graylog will be secured if your API uses SSL . To secure the communication between the Collector and Graylog select Enable TLS in your Beats Input. Graylog will create a self-signed certificate for this Input. In the Sidecar Beats Output Configuration select Enable TLS Support and Insecure TLS connection. After this is saved, the communication between Beats and Graylog will use TLS.

Certificate Based Client Authentication

If you want Graylog to only accept data from authenticated Collectors please follow the steps at Secured Graylog and Beats input.

Run Sidecar as Non-Root User

Sidecar is initiated using the root user by default in order to provide access to all log files, but this is not mandatory. If you choose to start it with a daemon user, proceed as so:

• Create a daemon user, e.g.sidecar. 

Sidecar itself can access the following files and directories:

• sidecar.yml- /etc/graylog/sidecar/sidecar.yml
• collector_configuration_directory- /var/lib/graylog-sidecar/generated/
• node_id- /etc/graylog/sidecar/node-id
• cache_path- /var/cache/graylog-sidecar/
• log_path- /var/log/graylog-sidecar/

So to make these directories readable for the sidecar user, use:

• chown -R sidecar /etc/graylog/sidecar
• chown -R sidecar /var/cache/graylog-sidecar
• chown -R sidecar /var/lib/graylog-sidecar
• chown -R sidecar /var/log/graylog-sidecar

You can change all paths to different places in the file system. If you prefer to store all sidecar data in the home directory of the sidecar user. Please change the paths accordingly if you choose to do so.

Next, systemd needs to know that the sidecar should be started with a non-root user. Open /etc/systemd/system/collector-sidecar.service with an editor and navigate to the [Service] section, add:

User = sidecar
Group = sidecar        

To make use of these settings reload systemd:

sudo systemctl daemon-reload
sudo systemctl restart graylog-sidecar        

Check the log files in /var/log/graylog-sidecar for any errors. Understand that not only the sidecar but also all backends, like filebeat, will be started as sidecar user after these changes. So all log files that the backend should observe also need to be readable by the sidecar user. Depending on the Linux distribution there is usually an administrator group which has access to most log files. By adding the sidecar user to that group you can grant access fairly easy. For example on Debian/Ubuntu systems this group is called adm (see System Groups in Debian Wiki or Security/Privileges - Monitor system logs in Ubuntu wiki).

fields.gl2_source_collector: ${sidecar.nodeId}
fields.collector_node_id: ${sidecar.nodeName}

Sidecar Configuration Migrator

The task of the Sidecar Configuration Migrator is to extract the configuration from existing Collector Sidecars and convert it into new Sidecar configurations.

This feature needs a Collector Sidecar with version 0.1.8 or greater. Please upgrade the instance you want to import configurations from, if necessary.

• Navigate to the Collectors (Legacy) overview. In your Graylog web interface click onSystem /Collectors (legacy).

• Click on the name of the Collector you want to import configurations from.

• Click theImport Configurationbutton on a backend to import a configuration. If the import was successful, follow the link to create a new sidecar configuration.

• After clicking onCreate Configurationclick on theMigratebutton underneath the configuration editor.

• A window opens up and lets you pick previously imported configurations. ClickingApplywill paste the configuration into the editor. Afterward, you can edit and save the configuration as usual.

Testing Sidecar

To ensure that our Graylog instance is collecting our Windows logs:

1. Go to the Overview tab underneath Sidecars.
2. Select the Show messages button on the right-hand side.

Now, it shows the logs that are coming in from our Windows Machine.

The image below shows a more detailed example of what your search page should look like when you view your incoming logs from your sidecar.

Assigning Tags

You can assign configurations based on tags. Tags are used to define which configurations the host should receive. For example a user can create a configuration for Apache access log files. In this case, the configuration gets the tag apache.

The sidecar can also be started on all web servers running the Apache daemon with the apache tag to fetch this configuration and to collect web access log files.

This is the specific configuration section of a configuration YAML file that would typically exist in an endpoint computer:

# A list of tags to assign to this sidecar. Collector configuration matching any of these tags will automatically be
# applied to the sidecar.
# Example:
# tags:
# - apache-logs
# - dns-logs

Once the system has the tag applied to the endpoint, it automatically starts collecting logs and bringing them into Graylog. The tag feature frees users/administrators from the burden of managing the collector configurations on all PCs or other endpoints registering to Graylog. Tags automatically assign configurations to new clients as long as their YAML configuration file has the correct configuration in it. This obviously saves time and effort in the management of endpoints.

Sidecar tags can be stored in the endpoint client such as a Windows Server and Windows Workstation or any other Linux Servers or Workstations. They are also built so you can have them attached to different collectors for Winlogbeat, Filebeat, Metricbeat, and NX log clients.

The new sidecar allows you to have multiple collector configurations assigned to one endpoint. For example, you could have a configuration for a Winlogbeat tag and another for a Sysmon tag layered on the same Windows client.

Sidecar Glossary

The following section is an explanation of terms that refer to different parts of the Graylog sidecar.

Configuration

A configuration is the representation of a log collector configuration file in the Graylog web interface. A configuration can be assigned to sidecars, which also assigns the corresponding collector. You can have multiple configurations for a single log collector. However, you can not assign the same collector twice to a sidecar.

Inputs

Collectors ingest data through inputs. An input can be a log file that the collector should continuously read or a connection to the Windows event system that emits log events. An input is connected to an output, otherwise there would be no way of sending the data to the next hop. So first create an output and then associate one or many inputs with it.

Debug

The sidecar writes log files to the directory configured in log_path. There is one file for each backend. You can check for general issues like file permissions or log transmission problems. The sidecar itself writes to sidecar.log. Problems like failed connection to the Graylog API can be found here.

You can also start the sidecar in the foreground and monitor the output of the process:

graylog-sidecar -debug

Uninstall

In Linux, just uninstall the package, to perform an uninstall in Windows run:

"C:\Program Files\Graylog\graylog-sidecar.exe" -service stop 
"C:\Program Files\Graylog\graylog-sidecar.exe" -service uninstall

(Prefix the commands with & when using PowerShell)

Known Problems

Currently we know of two problems with NXLog:

• Since version 2.9.17, timestamps are transmitted without millisecond precision.
• In Windows machines NXlog is not able to store its collector state so features like file tailing don’t work correctly in combination with sidecar. Use sidecar version 0.1.0-alpha.1 or newer.

A known issue if you use a load balancer or firewall in front of the Graylog API:

• The sidecar uses a persistent connection for API requests. Therefore it logs408 Request Time-outif the load balancer session or timeout is lower than the configuredupdate_interval.

sudo rpm -Uvh https://packages.graylog2.org/repo/packages/graylog-sidecar-repository-1-5.noarch.rpm
sudo yum install graylog-sidecar
            

vi /etc/graylog/sidecar/sidecar.yml

sudo graylog-sidecar -service install

sudo systemctl enable graylog-sidecar
sudo systemctl start graylog-sidecar        

sudo /etc/init.d/nxlog stop
sudo update-rc.d -f nxlog remove
sudo gpasswd -a nxlog adm
sudo chown -R nxlog.nxlog /var/spool/nxlog    

sudo service nxlog stop
sudo chkconfig --del nxlog
sudo gpasswd -a nxlog root
sudo chown -R nxlog.nxlog /var/spool/nxlog            

"C:\Program Files (x86)\nxlog\nxlog" -u
            

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

output.logstash:
 hosts: ["<your_graylog_ip>:5044"]
path:
 data: C:\Program Files\Graylog\sidecar\cache\winlogbeat\data
 logs: C:\Program Files\Graylog\sidecar\logs
tags:
 - windows
winlogbeat:
 event_logs:
  - name: Application
  - name: System
  - name: Security        

"C:\Program Files\graylog\sidecar\graylog-sidecar.exe" -service install
"C:\Program Files\graylog\sidecar\graylog-sidecar.exe" -service start
            

User = sidecar
Group = sidecar        

sudo systemctl daemon-reload
sudo systemctl restart graylog-sidecar        

fields.gl2_source_collector: ${sidecar.nodeId}
fields.collector_node_id: ${sidecar.nodeName}

# A list of tags to assign to this sidecar. Collector configuration matching any of these tags will automatically be
# applied to the sidecar.
# Example:
# tags:
# - apache-logs
# - dns-logs

graylog-sidecar -debug

"C:\Program Files\Graylog\graylog-sidecar.exe" -service stop 
"C:\Program Files\Graylog\graylog-sidecar.exe" -service uninstall