Graylog Sidecar is a lightweight configuration management system for different log
collectors, also called Backends. The Graylog node(s) acts as a centralized hub containing the
configurations of log collectors. On supported message-producing devices/hosts, Sidecar can run as a service
(Windows host) or daemon (Linux
host).
The
log collector configurations are centrally managed through the Graylog web interface. Periodically, the Sidecar
daemon will fetch all relevant configurations for the target, using the REST API. On
its first run or when a configuration change has been detected, Sidecar will generate (render) relevant
backend configuration files. Then it will start, or restart, those reconfigured log collectors.
Installation
You can get .deb
and
.rpm
packages for Graylog Sidecar in our package repository. For Windows, you can
download the installer from here.
Please follow the version matrix to pick the right package:
Sidecar Version | Graylog Server Version |
---|---|
1.2.x | 3.2.5 or higher |
1.1.x | 3.2.5 or higher |
1.0.x | 3.0 or higher |
0.1.x | 2.2.x, 2.3.x, 2.4.x, 2.5.x, 3.0.x, 4.0.x |
0.0.9 | 2.1.x |
All following commands should be executed on the remote machine which is where you want to collect log data from.
Install the Sidecar
Ubuntu
Install the Graylog Sidecar repository configuration and Graylog Sidecar itself with the following commands:
wget https://packages.graylog2.org/repo/packages/graylog-sidecar-repository_1-2_all.deb
sudo dpkg -i graylog-sidecar-repository_1-2_all.deb
sudo apt-get update && sudo apt-get install graylog-sidecar
Edit the configuration (see Configuration) and activate the Sidecar as a system service:
vi /etc/graylog/sidecar/sidecar.yml
sudo graylog-sidecar -service install
# Ubuntu 14.04 with Upstart
sudo start graylog-sidecar
# Ubuntu 16.04 and later with systemd
sudo systemctl enable graylog-sidecar
sudo systemctl start graylog-sidecar
CentOS
Install the Graylog Sidecar repository configuration and Graylog Sidecar itself with the following commands:
sudo rpm -Uvh https://packages.graylog2.org/repo/packages/graylog-sidecar-repository-1-2.noarch.rpm
sudo yum install graylog-sidecar
Edit the configuration (see Configuration) and activate the Sidecar as a system service:
vi /etc/graylog/sidecar/sidecar.yml
sudo graylog-sidecar -service install
sudo systemctl enable graylog-sidecar
sudo systemctl start graylog-sidecar
Windows (Preparation)
To get started with Sidecar on Windows be sure these requirements are met:
- When choosing what version of Sidecar to install, you must choose the version based on the Graylog version you are currently using. You can find the Windows Sidecar install link on our Github page.
- Graylog Sidecar service is successfully installed inside your Windows OS.
- Your Graylog instance is up and running alongside your Windows OS.
Sidecar Version | Graylog Version |
---|---|
1.2.x | 3.2.5 or higher |
1.1.x | 3.2.5 or higher |
1.0.x | 3.0 or higher |
Before starting the procedure to set up Sidecar on Windows, configure your input to receive Windows Sidecar
logs on port 5044
.
- Navigate to System > Inputs.
- Select an input from the first dropdown menu on the Inputs screen.
- Select Beats.
- Click the Launch new input button to prompt a new form.
- Check the Global box.
- Ensure the port field is set to
5044
.
Installing Collectors
Graylog contains default collector configurations for Filebeat, Winlogbeat (mentioned above), and NXLog.
Next up, you can decide which collectors you want to use with your Sidecar and install them. We only cover the installation of the most common ones here, but you are free to use other collectors as well. But since you’re able to define your own collector backends, you could run e.g. sysmon, auditd, or packetbeat.
Beats on Linux
Install Filebeat or another Beats package by following the instructions on the official Filebeat download page.
NXLog on Ubuntu
Install the NXLog package from the official NXLog download page. Because the Sidecar takes control of stopping and starting NXlog it is necessary to stop all running instances of NXlog and deconfigure the default system service:
sudo /etc/init.d/nxlog stop
sudo update-rc.d -f nxlog remove
sudo gpasswd -a nxlog adm
sudo chown -R nxlog.nxlog /var/spool/nxlog
NXLog on CentOS
The same on a RedHat based system:
sudo service nxlog stop
sudo chkconfig --del nxlog
sudo gpasswd -a nxlog root
sudo chown -R nxlog.nxlog /var/spool/nxlog
NXlog on Windows
Install the NXLog package from the official download page and deactivate the system service. We just need the binaries installed on the system:
"C:\Program Files (x86)\nxlog\nxlog" -u
(Prefix the
commands with &
when using PowerShell)
Sidecar Configuration
On
the command line you can provide a path to the configuration file with the -c
switch. The default configuration path on Linux systems is
/etc/graylog/sidecar/sidecar.yml
and C:\\Program
Files\\Graylog\\sidecar\\sidecar.yml
on Windows.
Most configuration parameters come with built-in
defaults. The only parameters that need adjustment are server_url
and
server_api_token
. You can get your API token by following the link on the Sidecars
Overview page.
sidecar.yml
Reference
Parameter |
Description |
---|---|
server_url |
URL to the Graylog API, e.g.
|
server_api_token |
The API token to use to authenticate against the Graylog server
API. |
node_id |
The node ID of the sidecar. This can be a path to a file or an ID
string. |
node_name |
Name of the Sidecar instance, will also show up in the web
interface. |
update_interval |
The interval where the sidecar will fetch new configurations from
the Graylog server in seconds |
tls_skip_verify |
This configures if the sidecar should skip the verification of
TLS connections. Default:
|
send_status |
This controls the transmission of detailed sidecar information
like collector status, |
list_log_files |
Send a directory listing to Graylog and display it on the host
status page, |
cache_path |
The directory where the sidecar stores internal data. Default:
|
collector_configuration_directory |
The directory where the sidecar generates configurations for
collectors. |
log_path |
The directory where the sidecar stores its logs. Default:
|
log_rotate_max_file_size |
The maximum size of the log file before it gets rotated. Default:
|
log_rotate_keep_files |
The maximum number of old log files to retain. |
collector_binaries_accesslist |
A list of binaries which are allowed to be executed by the
Sidecar. |
First Start
Once you have installed the Sidecar package and start the service for the first time, you can verify that it shows up in the Sidecars Overview page. A new sidecar instance will not have any configurations assigned yet. Take the Step-by-step guide to create your first configuration.
Mode of Operation
When
the Sidecar is assigned a configuration via the Graylog web interface, it will write a configuration file into the
collector_configuration_directory
for each collector backend, e.g. if you assigned a
Filebeat collector, then you will find a filebeat.yml
file in that directory. All changes
have to be made in the Graylog web interface.
Every time the Sidecar detects an update to its configuration it will rewrite the corresponding collector configuration file. Manually editing these files is not recommended.
Every time a collector configuration file is changed the collector process is restarted. The Sidecar takes care of the collector processes and reports the status back to the web interface.
Sidecar Status
Each Sidecar
instance is able to send status information back to Graylog. By enabling the option send_status
metrics like load or the IP address of the host Sidecar is running on are sent. Also metrics that are relevant for a
stable operation e.g. disk volumes with over 75% utilization are included.
Additionally with the
list_log_files
option a directory listing is displayed in the Graylog web interface.
This way an administrator can see which files are available for collecting. The list is periodically updated and
files with write access are highlighted for easy identification. After enabling
send_status
or send_status
+
list_log_files
go to the collector overview and click on one of them, a status page
with the configured information will be displayed.
Step-by-Step Guide
We have prepared an example on how to configure the Sidecar using the Graylog web interface. The example describes a winlogbeats configuration.
This assumes you have a Beats input already configured.
Create an API Token
- Navigate to System > Sidecars, and click the explainer text: Create or reuse a token for the graylog-sidecar user.
- Enter your choice name into the Token Name field.
- Click the Create Token button.
- Save the API server token in a safe yet accessible location in case you need to retrieve it again.
Configure the Sidecar Service on WindowsOS
Now that you have access to an API Token, you need to run the Windows Sidecar installer.
Complete the following steps:
- Enter the URL in your Graylog API, it should be pre configured as (https://127.0.0.1:9000/api).
- Name your Sidecar instance.
- Enter your server API token that you created earlier.
- Click Install to close the installer.
Once finished, you can change or configure your sidecar.yml
file, which should
be located in C:
.\\Program Files\\Graylog\\sidecar\\sidecar.yml
Configuring the Winlogbeat Collector
- Navigate back to your Graylog instance.
- Go to System > Sidecars within your Graylog instance and select the configuration tab in the left hand corner, then click the Create Configuration tab.
- Select winlogbeat on Windows from the Collector dropdown menu.
- Enter this configuration script in the Configuration field:
# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}
output.logstash:
hosts: ["<your_graylog_ip>:5044"]
path:
data: C:\Program Files\Graylog\sidecar\cache\winlogbeat\data
logs: C:\Program Files\Graylog\sidecar\logs
tags:
- windows
winlogbeat:
event_logs:
- name: Application
- name: System
- name: Security
This is a configuration Graylog pre-builds for you.
5. Add a name, and (optionally) apply a custom color to the
configuration.
6. Click Create.
- Verify the new Sidecar/winlogbeat configuration you created is listed in the Configurations menu.
Install and Start the Service
Now, open a command prompt window using administrator rights. Then, perform the following steps:
- Run these commands: (Prefix the commands with
&
when using PowerShell)
"C:\Program Files\graylog\sidecar\graylog-sidecar.exe" -service install
"C:\Program Files\graylog\sidecar\graylog-sidecar.exe" -service start
- Navigate back to the Graylog UI.
- Locate your Windows device on the Sidecar sub-menu (under System).
- Select the winlogbeat collector underneath the Windows Sidecar machine on the left-side, and on the Configure drop down on the right-hand side select the windows_sidecar configuration that we set up earlier.
- Click the Process drop down on the right-hand side and select Start after choosing your configuration.
Your Graylog instance should now be successfully started and collecting logs from your Windows machine.
Using Configuration Variables
Configuration variables can contain arbitrary strings like the IP address
of your Graylog server or the port of an input. The variables can then be used in multiple collector configurations,
which avoids duplication and simplifies management.
To create a configuration variable go any
Collector
Configuration
page:
On
the right you’ll find a box Collector Configuration Reference
which contains and.
Click on Variables
and then Create Variable
to
receive the following
modal:
In
this example we replace the hard coded IP and Port from our Beats input with a new variable named
${user.BeatsInput}
:
We can now use this variable in all our configurations. If we ever need to change the IP/port of our input, we just change the variable.
Runtime Variables
Runtime variables contain runtime informations from each
Sidecar that is requesting this configuration. An important example is the
variable.
The collector configuration should contain an instruction to fill that variable in an extra field. This allows
Graylog to relate messages to the Sidecar that produced them. (This is what makes the ${sidecar.nodeId}
button on the Sidecars overview page work.)Show
messages
Secure Sidecar Communication
The
Communication between Sidecar and Graylog will be secured if your API uses SSL .
To secure the communication between the Collector and Graylog you just need to mark Enable
TLS
in your Beats Input. Without giving additional Information, Graylog will now create a self-signed
certificate for this Input. Now in the Sidecar Beats Output Configuration you just mark Enable
TLS Support
and Insecure TLS connection
. After this is saved, the
communication between Beats and Graylog will use TLS.
Certificate Based Client Authentication
If you want Graylog to only accept data from authenticated Collectors please follow the steps at Secured Graylog and Beats input.
Run Sidecar as Non-Root User
The default is that the Sidecar is started with the root user to allow access to all log files. But this is not mandatory. If you like to start it with a daemon user, proceed like the following:
- Create a daemon user, e.g.
sidecar
.
The Sidecar itself is accessing the following files and directories:
sidecar.yml
- /etc/graylog/sidecar/sidecar.ymlcollector_configuration_directory
- /var/lib/graylog-sidecar/generated/node_id
- /etc/graylog/sidecar/node-idcache_path
- /var/cache/graylog-sidecar/log_path
- /var/log/graylog-sidecar/
sidecar
user, use:chown -R sidecar /var/cache/graylog-sidecar
chown -R sidecar /var/lib/graylog-sidecar
chown -R sidecar /var/log/graylog-sidecar
chown -R sidecar /etc/graylog/sidecar
You can change all paths to different places in the file system. If you prefer to store all Sidecar data in the
home directory of the sidecar
user, just change the paths accordingly.
Now
systemd
needs to know that the Sidecar should be started with a non-root user. Open
/etc/systemd/system/collector-sidecar.service
with an editor and navigate to the
[Service]
section, add:
User = sidecar
Group = sidecar
To make use of these settings reload systemd:
sudo systemctl daemon-reload
sudo systemctl restart graylog-sidecar
Check the log files in /var/log/graylog-sidecar
for any errors. Understand that not
only the Sidecar but also all backends, like filebeat
, will be started as
sidecar
user after these changes. So all log files that the backend should observe
also need to be readable by the sidecar
user. Depending on the Linux distribution
there is usually an administrator group which has access to most log files. By adding the sidecar
user to that group you can grant access fairly easy. For example on Debian/Ubuntu systems this group is called
(see System Groups in
Debian
Wiki or Security/Privileges -
Monitor system logs in Ubuntu wiki).adm
Upgrading from the Collector Sidecar
This guide describes how you can perform an upgrade from the deprecated Collector Sidecars (0.1.x)
to the new Sidecars (1.x).
One major difference between the old and the new Sidecars, is that we
replaced the UI based collector configuration approach with one where you can manage the plain text configuration of
the collectors directly. This might seem like an inconvenience at first, but gives you the flexibility to configure
any collector backend you want.
Additionally, the new Sidecars don’t assign configurations based on tags anymore. Instead you have to assign configurations explicitly (see Step-by-Step guide).
1. Install New Sidecar
The new Sidecar has different paths and executable names, so it can coexist with the old one. Install the new Sidecar by following the Installation instructions and have your Sidecar running as described in First Start.
2. Migrate Configuration
Next, we need to migrate the configuration that was previously rendered on
each host by the Collector Sidecar, to a new Collector Configuration.
We
recommend to use the Sidecar Configuration Migrator. However, retrieving the old configuration can also be done
manually by fetching it from your host at the
/etc/graylog/collector-sidecar/generated/
directory.
3. Adopt Configuration to Graylog 3.0
There are a few things that might need attention after an upgrade:
- Use Runtime variables for static fields
The imported configuration contains instructions that add static fields which allows Graylog to relate messages to a Sidecar. You should replace the hardcoded values ofgl2_source_collector
andcollector_node_id
with runtime variables.
In case of a Beats collector this would be:
fields.gl2_source_collector: ${sidecar.nodeId}
fields.collector_node_id: ${sidecar.nodeName}
- Migrate to the new Beats input
Graylog 3.0 comes with a new Beats input. The former one was renamed toBeats(deprecated)
. The new input handles fields a little different. Therefore you should definefields_under_root: true
for the new input to get the Graylog fields work.
4. Switch Over to the New Sidecar
Once you’re done creating a new configuration, you can assign it to your Sidecar (see Step-by-Step guide). If everything works as expected, make sure to uninstall the old Collector Sidecar to avoid collecting your logs twice.
Sidecar Configuration Migrator
The task of the Sidecar configuration migrator is to extract the configuration from existing Collector Sidecars and convert it into new Sidecar configurations.
This feature needs a Collector Sidecar with version 0.1.8 or greater. Please upgrade the instance you want to import configurations from, if necessary.
- Navigate to the Collectors (legacy) overview. In your Graylog web interface click on
System / Collectors (legacy)
.
- Click on the name of the Collector you want to import configurations from.
- Click the
Import Configuration
button on a backend to import a configuration. If the import was successful, follow the link to create a new Sidecar configuration:
- After clicking on
Create Configuration
use theMigrate
button underneath the configuration editor:
- A window opens up and lets you pick already imported configurations. Clicking
Apply
will paste the configuration into the editor. Afterwards, you can edit and save the configuration as usual.
Testing Sidecar
To ensure that our Graylog instance is collecting our Windows logs:
- Go to the Overview tab underneath Sidecars.
- Select the Show messages button on the right-hand side.
Now, it shows the logs that are coming in from our Windows Machine.
The image below shows a more detailed
example of what your search page should look like when you view your incoming logs from your
Sidecar.
Sidecar Glossary
The following section is an explanation of terms that refer to different parts of the Graylog Sidecar.
Configuration
A configuration is the representation of a log collector configuration file in the Graylog web interface. A configuration can be assigned to Sidecars, which also assigns the corresponding collector. You can have multiple configurations for a single log collector. However, you can not assign the same collector twice to a Sidecar.
Inputs
Collectors ingest data through inputs. An input can be a log file that the collector should continuously read or a connection to the Windows event system that emits log events. An input is connected to an output, otherwise there would be no way of sending the data to the next hop. So first create an output and then associate one or many inputs with it.
Debug
The Sidecar writes log files to
the directory configured in log_path
. There is one file for each backend. You can
check for general issues like file permissions or log transmission problems. The Sidecar itself writes to
sidecar.log
. Problems like failed connection to the Graylog API can be found here.
You can also start the Sidecar in the foreground and monitor the output of the process:
graylog-sidecar -debug
Uninstall
In Linux, just uninstall the package, to perform an uninstall in Windows run:
"C:\Program Files\Graylog\graylog-sidecar.exe" -service stop
"C:\Program Files\Graylog\graylog-sidecar.exe" -service uninstall
(Prefix the commands with &
when using PowerShell)
Known Problems
Currently we know of two problems with NXLog:
- Since version 2.9.17, timestamps are transmitted without millisecond precision.
- In Windows machines NXlog is not able to store its collector state so features like file tailing don’t work correctly in combination with Sidecar. Use Sidecar version 0.1.0-alpha.1 or newer.
A known issue if you use a loadbalancer or firewall in front of Graylog’s API:
- The Sidecar uses a persistent connection for API requests. Therefore it logs
408 Request Time-out
if the load balancer session or timeout is lower than the configuredupdate_interval
.