A Graylog setup needs data in it. This article explains the basic principles of getting your data into the system.

What Are Graylog Inputs?

Message inputs are responsible for accepting log messages in Graylog. Some default message types are available by default in Graylog. But you might need to install additional plugins to enable Graylog to receive particular messages.

After choosing the input type in the Graylog web interface at System / Inputs, the input is launched without a restart of Graylog. Most environments will use the defaults for the inputs, but most inputs have a granular configuration available. Some can use TLS or authentication via certificates, and others can make use of a queuing system.

Most environments will have one input of each type and ingest all messages of that type to the one input. But you need to have multiple inputs of the same kind to allow some extractors to work only on specific messages. The processing pipeline allows you to work on messages ingested from different sources and on the same input in different ways. Because every environment is different, Graylog is flexible and will enable you to adjust everything to your needs and not require you to change your settings to fit Graylog.

The section about different log sources describes how you ingest those types of logs. Some logs have individual inputs. Some of them are listed below in the individual inputs section. In addition to that, the Graylog Marketplace contains some inputs created by the community.

Log Sources

Although we can’t cover all possible options and device configurations in this section, the following describes the fundamentals of inputs and the current best practices to ingest these kind of messages into Graylog. Some can be done native others need a collector. This section includes a few examples of various available collectors.

Individual Inputs

These listed inputs are not necessarily natives available in Graylog. Most are dedicated to an individual product or framework. The default inputs like Syslog, CEF, GELF, or the RAW/Plaintext are not listed individually.

Using Apache Kafka as Transport Queue

Graylog supports Apache Kafka as a transport for various inputs such as GELF, syslog, and Raw/Plaintext inputs. The Kafka topic can be filtered by a regular expression and depending on the input, various additional settings can be configured.

Using RabbitMQ (AMQP) as a Transport Queue

Graylog supports AMQP as a transport for various inputs such as GELF, syslog, and Raw/Plaintext inputs. It can connect to any AMQP broker supporting AMQP 0-9-1 such as RabbitMQ.

Input Throttling

Throttling allows certain Graylog Inputs to slow their message intake rates (by temporarily pausing intake processing) if contention occurs in the Graylog Journal.

Graylog Inputs that Support Throttling

  • AWS Flow Logs
  • AWS Cloudtrail
  • AWS Logs
  • CEF AMQP Input
  • CEF Kafka Input
  • GELF AMQP
  • GELF Kafka
  • JSON path from API
  • Raw/Plaintext AMQP
  • Raw/Plaintext Kafka
  • Syslog AMQP
  • Syslog Kafka

Enabling Throttling

To enable throttling for one of these inputs, edit it in System > Inputs and check the Allow throttling this input checkbox.

Throttling Criteria

When enabled, the following criteria will be used to determine if throttling will occur:

  1. If there are zero uncommitted entries in the Graylog Journal, throttling will not occur. No further checks will be performed.
  2. Throttling will occur if the Journal has more than 100k uncommitted entries.
  3. Throttling will occur if the Journal is growing in size rapidly (approximately 20k entries per second or greater).
  4. Throttling will occur if the process ring buffer is full.
  5. If nothing is currently being written to the Journal; throttling will not occur. No further checks will be performed.
  6. Throttling will occur if the Journal is more than 90% full.
  7. Throttling will occur if the Journal write rate is more than twice as high as the read rate.