Along with 5.0 tags can be used for organization and automation making the configuration process much faster. See Assigning Tags for details.
Graylog Sidecar is a lightweight configuration management system for different log collectors, also called Backends. The Graylog node(s) acts as a centralized hub containing the configurations of log collectors. On supported message-producing devices/hosts, Sidecar can run as a service (Windows host) or daemon (Linux host).
The log collector configurations are centrally managed through the Graylog web interface. Periodically, the Sidecar daemon will fetch all relevant configurations for the target, using the REST API. On its first run or when a configuration change has been detected, Sidecar will generate (render) relevant backend configuration files. Then it will start, or restart, those reconfigured log collectors.
The following guide describes the Graylog Sidecar on-premise configuration. For information regarding the Sidecar in Graylog Cloud, see the related article.
Installation
You can get .deb and .rpm packages for Graylog Sidecar in our package repository. For Windows, you can download the installer from here.
Please follow the version matrix to pick the right package:
Sidecar Version | Graylog Server Version |
---|---|
1.4.x |
5.0.x |
1.3.x | 5.0.x |
1.2.x | 3.2.5 or higher |
1.1.x | 3.2.5 or higher |
1.0.x | 3.0 or higher |
0.1.x | 2.2.x, 2.3.x, 2.4.x, 2.5.x, 3.0.x, 4.0.x |
0.0.9 | 2.1.x |
All following commands should be executed on the remote machine which is where you want to collect log data from.
Install the Sidecar
Ubuntu
Install the Graylog Sidecar repository configuration and Graylog Sidecar itself with the following commands:
wget https://packages.graylog2.org/repo/packages/graylog-sidecar-repository_1-5_all.deb
sudo dpkg -i graylog-sidecar-repository_1-5_all.deb
sudo apt-get update && sudo apt-get install graylog-sidecar
Edit the configuration (see Configuration ) and activate the Sidecar as a system service:
sudo vi /etc/graylog/sidecar/sidecar.yml
sudo graylog-sidecar -service install
# Ubuntu 14.04 with Upstart
sudo start graylog-sidecar
# Ubuntu 16.04 and later with systemd
sudo systemctl enable graylog-sidecar
sudo systemctl start graylog-sidecar
CentOS
Install the Graylog Sidecar repository configuration and Graylog Sidecar itself with the following commands:
sudo rpm -Uvh https://packages.graylog2.org/repo/packages/graylog-sidecar-repository-1-5.noarch.rpm
sudo yum install graylog-sidecar
Edit the configuration (see Configuration) and activate the Sidecar as a system service:
vi /etc/graylog/sidecar/sidecar.yml
sudo graylog-sidecar -service install
sudo systemctl enable graylog-sidecar
sudo systemctl start graylog-sidecar
Windows (Prerequisites)
To get started with Sidecar on Windows be sure these requirements are met:
-
When choosing what version of Sidecar to install, you must choose the version based on the Graylog version you are currently using. You can find the Windows Sidecar install link on our Github page.
The Windows installer can be run interactively by running:
$ graylog_sidecar_installer_1.4.0-1.exe.
Or you could run it in silent mode using:
$ graylog_sidecar_installer_1.4.0-1.exe /S -SERVERURL=http://your Graylog IP address
orDNS name/api -APITOKEN=yourapitoken
Hint: The Windows Installer supports additional options in silent mode. Examples of these parameters are: -TAGS=["example","IIS"] -NODENAME=mynodename -NODEID=1234 -SENDSTATUS=false -TLSSKIPVERIFY=true -UPDATEINTERVAL=10s - Graylog Sidecar service is successfully installed inside your Windows OS.
- Your Graylog instance is up and running alongside your Windows OS.
Sidecar Version | Graylog Version |
---|---|
1.4.x |
5.0.x |
1.3.x | 5.0.x |
1.2.x | 3.2.5 or higher |
1.1.x | 3.2.5 or higher |
1.0.x | 3.0 or higher |
Before starting the procedure to set up Sidecar on Windows, configure your input to receive Windows Sidecar logs on port 5044.
- Navigate to System > Inputs.
- Select an input from the first dropdown menu on the Inputs screen.
- Select Beats.
- Click the Launch new input button to prompt a new form.
- Check the Global box.
- Ensure the port field is set to
5044
.
Installing Collectors
Graylog contains default collector configurations for Filebeat, Winlogbeat (mentioned above), and NXLog.
Next up, you can decide which collectors you want to use with your Sidecar and install them. We only cover the installation of the most common ones here, but you are free to use other collectors as well. But since you’re able to define your own collector backends, you could run e.g. sysmon, auditd, or packetbeat.
Beats on Linux
Install Filebeat or another Beats package by following the instructions on the official Filebeat download page.
NXLog on Ubuntu
Install the NXLog package from the official NXLog download page. Because the Sidecar takes control of stopping and starting NXlog it is necessary to stop all running instances of NXlog and unconfigure the default system service:
sudo /etc/init.d/nxlog stop
sudo update-rc.d -f nxlog remove
sudo gpasswd -a nxlog adm
sudo chown -R nxlog.nxlog /var/spool/nxlog
NXLog on CentOS
The same on a RedHat based system:
sudo service nxlog stop
sudo chkconfig --del nxlog
sudo gpasswd -a nxlog root
sudo chown -R nxlog.nxlog /var/spool/nxlog
NXlog on Windows
Install the NXLog package from the official download page and deactivate the system service. We just need the binaries installed on the system:
"C:\Program Files (x86)\nxlog\nxlog" -u
(Prefix the commands with & when using PowerShell)
Sidecar Configuration
On the command line you can provide a path to the configuration file with the -c switch. The default configuration path on Linux systems is /etc/graylog/sidecar/sidecar.yml and C:\\Program Files\\Graylog\\sidecar\\sidecar.yml on Windows.
Most configuration parameters come with built-in defaults. The only parameters that need adjustment are server_url
and server_api_token
. You can get your API token by following the link on the Sidecars page which is located under the System menu. Please remember to save the API server token as you may need it during the installation process.
sidecar.yml Reference
Parameter |
Description |
---|---|
server_url |
URL to the Graylog API, e.g. https://192.168.1.1:9000/api/ |
server_api_token |
The API token to use to authenticate against the Graylog server API. |
node_id |
The node ID of the sidecar. This can be a path to a file or an ID string. |
node_name |
Name of the Sidecar instance, will also show up in the web interface. |
update_interval |
The interval where the sidecar will fetch new configurations from the Graylog server in seconds The Graylog server considers all sidecars that frequently perform these updates "active". In order to globally configure a threshold which determines when a sidecar should be considered "inactive," please navigate to "Sidecars System" under the System/Configuration menu. |
tls_skip_verify |
This configures if the sidecar should skip the verification of TLS connections. Default: false |
send_status |
This controls the transmission of detailed sidecar information like collector status, |
list_log_files |
Send a directory listing to Graylog and display it on the host status page, |
cache_path |
The directory where the sidecar stores internal data. Default: /var/cache/graylog-sidecar |
collector_configuration_directory |
The directory where the sidecar generates configurations for collectors. |
log_path |
The directory where the sidecar stores its logs. Default: /var/log/graylog-sidecar |
log_rotate_max_file_size |
The maximum size of the log file before it gets rotated. Default: 10MiB |
log_rotate_keep_files |
The maximum number of old log files to retain. |
collector_binaries_accesslist |
A list of binaries which are allowed to be executed by the Sidecar. |
tags | List of configuration tags. All configurations on the server side that match the tag list will be fetched and merged by this instance. |
First Start
Once you have installed the Sidecar package and start the service for the first time, you can verify that it shows up in the Sidecars Overview page. A new sidecar instance will not have any configurations assigned yet. Take the Step-by-step guide to create your first configuration.
Mode of Operation
When the Sidecar is assigned a configuration via the Graylog web interface, it will write a configuration file into the collector_configuration_directory
for each collector backend. E.g. if you assigned a Filebeat collector you will find a filebeat.yml
file in that directory. All changes have to be made in the Graylog web interface.
Every time the Sidecar detects an update to its configuration it will rewrite the corresponding collector configuration file. Manually editing these files is not recommended.
Every time a collector configuration file is changed the collector process is restarted. The Sidecar takes care of the collector processes and reports the status back to the web interface.
Sidecar Status
Each Sidecar instance is able to send status information back to Graylog. By enabling the option send_status
metrics like load or the IP address of the host Sidecar is running on are sent. Also metrics that are relevant for a stable operation e.g. disk volumes with over 75% utilization are included.
Additionally with the list_log_files
option a directory listing is displayed in the Graylog web interface. This way an administrator can see which files are available for collecting. The list is periodically updated and files with write access are highlighted for easy identification. After enabling send_status
or send_status + list_log_files
go to the collector overview and click on one of them, a status page with the configured information will be displayed.
Step-by-Step Guide
We have prepared an example on how to configure the Sidecar using the Graylog web interface. The example describes a winlogbeats configuration.
This assumes you have a Beats input already configured.
Create an API Token
- Navigate to System > Sidecars, and click the explainer text: Create or reuse a token for the graylog-sidecar user.
- Enter your choice name into the Token Name field.
- Click the Create Token button.
- Save the API server token in a safe yet accessible location in case you need to retrieve it again.
Configure the Sidecar Service on WindowsOS
Now that you have access to an API Token, you need to run the Windows Sidecar installer.
Complete the following steps:
- Enter the URL in your Graylog API, it should be pre configured as (https://127.0.0.1:9000/api).
- Name your Sidecar instance.
- Enter your server API token that you created earlier.
- Click Install to close the installer.
Once finished, you can change or configure your sidecar.yml
file, which should be located in C:\\Program Files\\Graylog\\sidecar\\sidecar.yml
.
Configuring the Winlogbeat Collector
- Navigate back to your Graylog instance.
- Go to System > Sidecars within your Graylog instance and select the configuration tab in the left hand corner, then click the Create Configuration tab.
- Select winlogbeat on Windows from the Collector dropdown menu.
- Enter this configuration script in the Configuration field:
# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}
output.logstash:
hosts: ["<your_graylog_ip>:5044"]
path:
data: C:\Program Files\Graylog\sidecar\cache\winlogbeat\data
logs: C:\Program Files\Graylog\sidecar\logs
tags:
- windows
winlogbeat:
event_logs:
- name: Application
- name: System
- name: Security
This is a configuration Graylog pre-builds for you.
5. Add a name, and (optionally) apply a custom color to the configuration.
6. Click Create.
7. Verify the new Sidecar/winlogbeat configuration you created is listed in the Configurations menu.
Install and Start the Service
Now, open a command prompt window using administrator rights. Then, perform the following steps:
- Run these commands: (Prefix the commands with
&
when using PowerShell)
"C:\Program Files\graylog\sidecar\graylog-sidecar.exe" -service install
"C:\Program Files\graylog\sidecar\graylog-sidecar.exe" -service start
- Navigate back to the Graylog UI.
- Locate your Windows device on the Sidecar sub-menu (under System).
- Select the winlogbeat collector underneath the Windows Sidecar machine on the left-side, and on the Configure drop down on the right-hand side select the windows_sidecar configuration that we set up earlier.
- Click the Process drop down on the right-hand side and select Start, after choosing your configuration.
Your Graylog instance should now be successfully started and collecting logs from your Windows machine.
Using Configuration Variables
Configuration variables can contain arbitrary strings like the IP address of your Graylog server or the port of an input. The variables can then be used in multiple collector configurations, which avoids duplication and simplifies management.
To create a configuration variable go any Collector Configuration page. On the right you’ll find a box named Collector Configuration Reference . Click on Variables and then Create Variable to receive the following modal:
In this example we replace the hard coded IP and Port from our Beats input with a new variable named ${user.BeatsInput}
:
We can now use this variable in all our configurations. If we ever need to change the IP/port of our input, we just change the variable.
Runtime Variables
Runtime variables contain runtime informations from each Sidecar that is requesting this configuration. An important example is the ${sidecar.nodeId}
variable. The collector configuration should contain an instruction to fill that variable in an extra field. This allows Graylog to relate messages to the Sidecar that produced them. (This is what makes the Show messages
button on the Sidecars overview page work.)
Secure Sidecar Communication
The Communication between Sidecar and Graylog will be secured if your API uses SSL . To secure the communication between the Collector and Graylog you just need to mark Enable TLS in your Beats Input. Without giving additional Information, Graylog will now create a self-signed certificate for this Input. Now in the Sidecar Beats Output Configuration you just mark Enable TLS Support
and Insecure TLS connection
. After this is saved, the communication between Beats and Graylog will use TLS.
Certificate Based Client Authentication
If you want Graylog to only accept data from authenticated Collectors please follow the steps at Secured Graylog and Beats input.
Run Sidecar as Non-Root User
The default is that the Sidecar is started with the root user to allow access to all log files. But this is not mandatory. If you like to start it with a daemon user, proceed like the following:
- Create a daemon user, e.g.
sidecar
.
The Sidecar itself is accessing the following files and directories:
sidecar.yml
- /etc/graylog/sidecar/sidecar.ymlcollector_configuration_directory
- /var/lib/graylog-sidecar/generated/node_id
- /etc/graylog/sidecar/node-idcache_path
- /var/cache/graylog-sidecar/log_path
- /var/log/graylog-sidecar/
So to make these directories readable for thesidecar user, use:
chown -R sidecar /etc/graylog/sidecar
chown -R sidecar /var/cache/graylog-sidecar
chown -R sidecar /var/lib/graylog-sidecar
chown -R sidecar /var/log/graylog-sidecar
You can change all paths to different places in the file system. If you prefer to store all Sidecar data in the home directory of the sidecar user, just change the paths accordingly.
Now systemd needs to know that the Sidecar should be started with a non-root user. Open /etc/systemd/system/collector-sidecar.service with an editor and navigate to the [Service] section, add:
User = sidecar
Group = sidecar
To make use of these settings reload systemd:
sudo systemctl daemon-reload
sudo systemctl restart graylog-sidecar
Check the log files in /var/log/graylog-sidecar
for any errors. Understand that not only the Sidecar but also all backends, like filebeat
, will be started as sidecar user after these changes. So all log files that the backend should observe also need to be readable by the sidecar user. Depending on the Linux distribution there is usually an administrator group which has access to most log files. By adding the sidecar user to that group you can grant access fairly easy. For example on Debian/Ubuntu systems this group is called adm (see System Groups in Debian Wiki or Security/Privileges - Monitor system logs in Ubuntu wiki).
Upgrading from the Collector Sidecar
This guide describes how you can perform an upgrade from the deprecated Collector Sidecars (0.1.x) to the new Sidecars (1.x). We still support the old Collector Sidecars which can be found in the System / Collectors (legacy) menu entry. We encourage users to migrate to the new Sidecar which is covered by this document.
One major difference between the old and the new Sidecars, is that we replaced the UI based collector configuration approach with one where you can manage the plain text configuration of the collectors directly. This might seem like an inconvenience at first, but gives you the flexibility to configure any collector backend you want.
1. Install the New Sidecar
The new Sidecar has different paths and executable names, so it can coexist with the old one. Install the new Sidecar by following the Installation instructions and have your Sidecar running as described in First Start.
Hint: In case you were using filebeat on Linux, please make sure to also install the official collector package, since the filebeat binary is not part of the Sidecar package anymore.
2. Migrate Configuration
Next, we need to migrate the configuration that was previously rendered on each host by the Collector Sidecar, to a new Collector Configuration.
We recommend to use the Sidecar Configuration Migrator. However, retrieving the old configuration can also be done manually by fetching it from your host at the /etc/graylog/collector-sidecar/generated/
directory.
3. Adopt Configuration
There are a few things that might need attention after an upgrade to 3.0:
- Use Runtime variables for static fields
The imported configuration contains instructions that add static fields which allows Graylog to relate messages to a Sidecar. You should replace the hardcoded values ofgl2_source_collector
andcollector_node_id
with runtime variables.
In case of a Beats collector this would be:
fields.gl2_source_collector: ${sidecar.nodeId}
fields.collector_node_id: ${sidecar.nodeName}
- Migrate to the new Beats input
As of Graylog 3.0 we have a new Beats input. The former one was renamed asBeats(deprecated)
. The new input handles fields differently. Therefore you should definefields_under_root: true
for the new input to get the Graylog fields work.
4. Switch Over to the New Sidecar
Once you’re done creating a new configuration, you can assign it to your Sidecar (see Step-by-Step guide). Make sure to uninstall the old Collector Sidecar to avoid collecting your logs twice.
Sidecar Configuration Migrator
The task of the Sidecar configuration migrator is to extract the configuration from existing Collector Sidecars and convert it into new Sidecar configurations.
This feature needs a Collector Sidecar with version 0.1.8 or greater. Please upgrade the instance you want to import configurations from, if necessary.
- Navigate to the Collectors (Legacy) overview. In your Graylog web interface click on
System / Collectors (legacy)
.
- Click on the name of the Collector you want to import configurations from.
- Click the
Import Configuration
button on a backend to import a configuration. If the import was successful, follow the link to create a new Sidecar configuration.
- After clicking on
Create Configuration
click on theMigrate
button underneath the configuration editor.
- A window opens up and lets you pick previously imported configurations. Clicking
Apply
will paste the configuration into the editor. Afterwards, you can edit and save the configuration as usual.
Testing Sidecar
To ensure that our Graylog instance is collecting our Windows logs:
- Go to the Overview tab underneath Sidecars.
- Select the Show messages button on the right-hand side.
Now, it shows the logs that are coming in from our Windows Machine.
The image below shows a more detailed example of what your search page should look like when you view your incoming logs from your Sidecar.
Assigning Tags
With Graylog 5.0 you can now assign configurations based on tags. Tags are used to define which configurations the host should receive. For example a user can create a configuration for Apache access log files. In this case, the configuration gets the tag apache.
The Sidecar can also be started on all web servers running the Apache daemon with the apache
tag to fetch this configuration and to collect web access log files.
This is the specific configuration section of a configuration YAML file that would typically exist in an endpoint computer:
# A list of tags to assign to this sidecar. Collector configuration matching any of these tags will automatically be
# applied to the sidecar.
# Example:
# tags:
# - apache-logs
# - dns-logs
Once the system has the tag applied to the endpoint, it automatically starts collecting logs and bringing them into Graylog. The tag feature frees users/administrators from the burden of managing the collector configurations on all PCs or other endpoints registering to Graylog. Tags automatically assign configurations to new clients as long as their YAML configuration file has the correct configuration in it. This obviously saves time and effort in the management of endpoints.
Sidecar tags can be stored in the endpoint client such as a Windows Server and Windows Workstation or any other Linux Servers or Workstations. They are also built so you can have them attached to different collectors for Winlogbeat, Filebeat, Metricbeat, and NX log clients.
The new Sidecar allows you to have multiple collector configurations assigned to one endpoint. For example, you could have a configuration for a Winlogbeat tag and another for a Sysmon tag layered on the same Windows client.
Sidecar Glossary
The following section is an explanation of terms that refer to different parts of the Graylog Sidecar.
Configuration
A configuration is the representation of a log collector configuration file in the Graylog web interface. A configuration can be assigned to Sidecars, which also assigns the corresponding collector. You can have multiple configurations for a single log collector. However, you can not assign the same collector twice to a Sidecar.
Inputs
Collectors ingest data through inputs. An input can be a log file that the collector should continuously read or a connection to the Windows event system that emits log events. An input is connected to an output, otherwise there would be no way of sending the data to the next hop. So first create an output and then associate one or many inputs with it.
Debug
The Sidecar writes log files to the directory configured in log_path
. There is one file for each backend. You can check for general issues like file permissions or log transmission problems. The Sidecar itself writes to sidecar.log
. Problems like failed connection to the Graylog API can be found here.
You can also start the Sidecar in the foreground and monitor the output of the process:
graylog-sidecar -debug
Uninstall
In Linux, just uninstall the package, to perform an uninstall in Windows run:
"C:\Program Files\Graylog\graylog-sidecar.exe" -service stop
"C:\Program Files\Graylog\graylog-sidecar.exe" -service uninstall
(Prefix the commands with & when using PowerShell)
Known Problems
Currently we know of two problems with NXLog:
- Since version 2.9.17, timestamps are transmitted without millisecond precision.
- In Windows machines NXlog is not able to store its collector state so features like file tailing don’t work correctly in combination with Sidecar. Use Sidecar version 0.1.0-alpha.1 or newer.
A known issue if you use a load balancer or firewall in front of Graylog’s API:
- The Sidecar uses a persistent connection for API requests. Therefore it logs
408 Request Time-out
if the load balancer session or timeout is lower than the configuredupdate_interval
.