Graylog Forwarder

The Forwarder is a feature that is exclusively available for Graylog Cloud, Graylog Security, and Graylog Enterprise. To learn more about Graylog licenses, please contact the Graylog Sales team.

The Graylog Forwarder is a standalone agent that sends log data to Graylog Cloud or an on-premise Graylog Server cluster. The Forwarder is typically run as a service to continuously stream data to the destination Graylog cluster.

Security

The Forwarder connects to Graylog Cloud over TLS. TLS is highly recommended for on-premise installations to ensure data moves securely. The Forwarder also uses API Token authentication to ensure only authorized forwarders can connect to your environment.

Assign forwarders and configure them with a token to authenticate with the destination Graylog cluster or Cloud environment. To set up a new forwarder, use the Forwarder Wizard to help you create an API token. Any tokens used by the forwarder must belong to the user Forwarder System User (built-in). Please see API token in the documentation for additional information.

Installation

The Forwarder is distributed in similar packaging and installation methods as the Graylog server. You can choose between operating system packages, Docker, and binary installation methods to install the forwarder. If you plan to run the tool on an OS package or binary, ensure that Java is installed on your operating system. See the Forwarder Installation page for more information.

Setup

The same forwarder agent can be used for both Graylog Cloud and on-premise Graylog installations, but the required setup is different for each environment.

Hint:  It is recommended to have at least 2 CPU cores (3GHz) and 4GB of RAM to ensure optimal performance.

Create Forwarder Input (on-premise only)

For on-premise forwarder setup, create a forwarder input on the System > Inputs page. Skip this step if you are using Graylog Cloud. This special forwarder input allows your Graylog nodes to accept connections from forwarders. Only create the special forwarder input once with the Global option checked to ensure that the input runs on all Graylog nodes within the cluster.

The default values are appropriate for most environments. We highly recommend that you enable TLS, especially if the forwarder traffic will route over the internet. The process is similar to Enable TLS in Graylog Server. You must provide your own TLS certificate and key for the input, and provide the certificate later when you configure the forwarder agent.

Once you have created the input, verify that it is RUNNING.Please see the Graylog server log for troubleshooting instructions.

Setup Wizard

Graylog ships with a Forwarder Setup Wizard to set up forwarders in both the Cloud and on-premise environments.

The Forwarder Setup Wizard can be found in the following locations:

  • Graylog on-premise: Enterprise > Forwarders
  • Graylog Cloud: System > Forwarders

From the main Forwarders page, launch the wizard by clicking the New Forwarder or Get Started buttons. Once launched, the wizard will guide you through the appropriate configuration steps for your environment.

Complete the wizard by following the sections below:

Install Forwarder

  1. Download and install the Forwarder.
  2. Click Continue to navigate to the next step in the accordion.

Create an API Token

We recommend a unique API token for each forwarder to ensure that you can revoke tokens for unused individual forwarders.

  1. Enter your Token Name in the available field.
  2. Click the Create Token button to create a new name.

Configure Local Forwarder Agent

The wizard now displays the required configuration to input in the forwarder.conf file. This file provides all of the environment-specific configurations needed for the forwarder to connect successfully.

The required configuration is different for Graylog Cloud and on-premise. The wizard automatically provides the needed configuration for your particular setup. You can then copy and paste it into your forwarder.conf file.

See the Forwarder Configuration Options page for a list of all supported configuration options.

Example Graylog Cloud configuration file:

Copy 
forwarder_server_hostname = ingest-<your-account-url>.graylog.cloud
forwarder_grpc_api_token = <your-api-token> 

Example Graylog on-premise configuration file:

Copy 
forwarder_server_hostname = (required) The Graylog server hostname where the Forwarder should connect to.
forwarder_grpc_api_token = <your-api-token>
forwarder_configuration_port = 13302
forwarder_message_transmission_port = 13301
forwarder_grpc_enable_tls = true
forwarder_grpc_tls_trust_chain_cert_file = <path to cert.pem>

Start Forwarder

Once configured, start the forwarder, so the setup wizard can find it on the next step.

If you use the Forwarder OS packages or Docker, follow the instructions on the Installation page.

If you use the forwarder binaries, use these instructions to start the forwarder:

  1. Open the terminal.
  2. Locate the startup script found in the application directory.
  3. Run the startup script with the command ./bin/graylog-forwarder run --configfile forwarder.conf.
  4. Return to the wizard after the forwarder successfully runs.

Once the Forwarder starts successfully, the console will display the following message:

Copy 
INFO: Forwarder Service started successfully. 

Select Forwarder

When the forwarder establishes a connection to Graylog, it will register automatically and you will see it listed on the next step. Both the hostname of the machine, where the forwarder is started, and the node ID are shown for each forwarder.

  1. Click the radio button to select the new forwarder.
  2. Click Configure selected Forwarder to navigate to the next menu: Configure Forwarder.

Configure Forwarder

In this section, you can customize the identity of your forwarder.

  1. Add a title.
  2. Enter a long-form description in case you want to distinguish it from existing forwarders.
  3. Click Add Forwarder inputs to complete this section.

Add Inputs

In this section, select an Input Profile. An input profile contains a collection of inputs that multiple forwarders can use. In this case, create one:

  1. Click Create Input Profile.
  2. Add a name in the Title field.
  3. Enter a description that corresponds to the Title.
  4. Click Add Inputs to complete the form.
  5. Select an Input Type from the dropdown menu.
  6. Fill in the details for your input in the form.
  7. Click Create Input. Save the configuration.

Summary

Review the summary, and then select Exit configuration. You will see the new forwarder on the "Forwarder" page.

After installation, configuration, and startup, the forwarder instance will register with Graylog and appear on the "Forwarders" page in Graylog. Each forwarder has a Configure button to begin the configuration process. If the forwarder does not appear, click on New Forwarder for information on how to configure and start it.

Input Profile

Input Profiles are a set of inputs you can configure to run on one or more forwarders. Input profiles provide a gateway to move data to Graylog.

Input Profiles allow you to assign a set of previously designed and tested inputs to all forwarders. If you want to collect the same kind of logs in different parts of your infrastructure or maintain a more redundant setup, input profiles allow you to do so.

As you create the forwarder, you must create an Input Profile from the wizard, as explained in Add Inputs. Provide a descriptive name for the Input Profile, a short description of its intention, and then create all inputs needed for the forwarder.

Monitoring Forwarder Activity and Health

After you connect the forwarder to Graylog, explore methods to access metrics and other information on your forwarder(s) and corresponding input(s). Here are a few methods to analyze and extract details on forwarder activity:

  • Review active forwarder(s) on your Graylog Cloud instance.
  • Call forwarder REST endpoints to consume information on health and a list of inputs.
  • Export Forwarder metrics from Prometheus, a third-party monitoring tool.

Forwarder Overview

One place to review forwarder agent connectivity is the Forwarders screen under the Systems menu. This page provides a summary of all forwarders. Identify the green Connected badge in the Status column. The Connected badge indicates that a forwarder is actively sending messages to your cloud instance. Active message rates in the Metrics column also indicate a functioning forwarder.

Forwarder Agent REST API

The forwarder agent supports a local REST API to check health status, inputs, and export Prometheus metrics. To enable the Forwarder API, follow the instructions below:

  • Open your forwarder.conf file.
  • Add the forwarder_api_enabled = true configuration option.

When enabled, the API will listen on a Unix Domain Socket, using the file indicated with forwarder_api_socket_path unless you provide a value for forwarder_api_tcp_bind_address. For example, you can run a curl command to access the endpoint. If you need a refresher on how to use Unix sockets, review this guide.

Health Status Endpoint

To check the health of your forwarder, query the endpoint GET /api/health:


Copy 
{ 
    "healthy" : true ,
    "inputs" : { 
        "healthy" : true , 
        "running" : 2 , 
        "failed" : 0 , 
        "not running" : 0 
        }, 
     "upstream" : { 
     "healthy" : true 
     } 
} 

Input Endpoint

To obtain a list of inputs running on the forwarder, query the endpoint GET /api/inputs.


Copy 
{ 
    "inputs" : [ 
        { 
            "id" : "5fc91564d44bfd2000249e8c" , 
            "title" : "Random"
        }, 
        { 
            "id" : "5fc91550d44bfd2000249e74" , 
            "title" : "Beats" 
        } 
     ] 
 } 

Drill down to the input profile, and view the forwarder sub-menu to ensure it receives messages. If no data comes through, create a new input. Click the name of your input to go to the input's main profile, which has the details you added at the initial configuration.

If an input is in a failed state, the input endpoint returns no information on id or title.

For even deeper insights into forwarder messaging and node health, click the Details button on your node to access information about your message cache (buffers). Note the number of messages in the journal. The journal is the on-disk persistent storage location that stores forwarder messages, so, even if the rest of Graylog malfunctions, we still have all the messages.

Prometheus Metrics Exports

The forwarder alone has no interface for insight into the internal operations. To that end, you must configure a local Prometheus container; this becomes the interface for forwarder metrics. These are similar to the traditional Graylog Server metrics but are instead exported to Prometheus. The response format is the standard Prometheus export format.

To start this process:

  1. Download and start Prometheus.
  2. Install Docker on your machine.
  3. Create a Prometheus Dockerfile, e.g. touch /tmp/prometheus.yml:

Copy 
global : 
    scrape_interval : 
    15s scrape_timeout : 
    10s evaluation_interval : 15s 
alerting : 
    alertmanagers : 
        - static_configs : - 
            targets : [] 
          scheme : http 
          timeout : 10s 
          api_version : v1 
scrape_configs : 
    - job_name : prometheus 
    honor_timestamps : true 
    scrape_interval : 15s 
    scrape_timeout : 10s 
    metrics_path : /api/metrics/prometheus 
    scheme : http 
    static_configs : 
        - targets : 
            - host.docker.internal:9001 
  1. Run this Docker command to start the container:

Copy 
docker run \ 
    -p 9090 :9090 \ 
    -v /tmp/prometheus.yml:/etc/prometheus/prometheus.yml \ 
    prom/prometheus

Resiliency Models

To add more forwarders, you must incorporate tools, procedures, and policies to continue operation in the case of a major outage: widespread, long-lasting, destructive, or all three. If all the above pose a threat to your forwarder, consider both message recovery and load balancing.

Message Recovery

The forwarder’s disk journal is capable of caching data in case of a network outage. From there, they are read and sent to Graylog.

As mentioned in the Output Framework chapter, if the Internet is unavailable, the forwarder is still capable of receiving messages. So once the internet connection is restored, the workflow will resume. Messages from the journal are sent to Graylog Cloud.

Load Balancing Options

A larger deployment means more throughput, i.e. requests passing through your systems. So in a more mature, multi-forwarder scenario, we recommend that you configure a load balancer to evenly distribute data transfer and help your deployment manage bulk requests and potential latency issues while ensuring resiliency.

The load balancer distributes requests among healthy nodes in your local and/or external datacenters. Learn more about how to handle requests among multiple forwarders with testing and configuring tools, such as Apache server, Nginx, or HAProxy, in our help docs.

Versioning

The forwarder uses a MAJOR.MINOR versioning scheme, and it is released independently of the Graylog server.

It is recommended to update the Forwarder to the latest available release on a regular basis. While we try to support older versions of a Forwarder, we recommend that the MAJOR version of the forwarder (5.x) matches that of the server (5.x). Otherwise certain features (like new inputs) might not be supported end-to-end.

Essentially, older versions of the Forwarder should continue to work with newer versions of the Graylog server. The subset of functionality should continue to work. Only in special cases will older versions of the Forwarder become unsupported. (e.g. Cases where it is a really old version that has become difficult to maintain).

Newer versions of the Forwarder should be compatible with older versions of the server.