Get in Log Data

Fundamentally, a Graylog setup needs data in it. This article explains the basic principles of getting your data into the system.

What Are Graylog Inputs?

Message inputs are responsible for accepting log messages in Graylog. Some default message types are available by default in Graylog. But you might need to install additional plugins to enable Graylog to receive particular messages.

See the Graylog documentation on inputs for more information. Some logs have individual inputs. Some of them are listed below in the individual inputs section. In addition to that, the Graylog Marketplace contains some inputs created by the community.

Log Sources

The following describes some of the fundamentals of common log sources and the current best practices to ingest these kind of messages into Graylog. Some can be done natively, and others need a collector. The following includes a few examples of various available collectors:

Individual Inputs

Inputs are not necessarily natives available in Graylog. Most are dedicated to an individual product or framework. The default inputs like syslog, CEF, GELF, or the RAW/Plaintext are not listed individually. See Inputs for more information.

Using Apache Kafka as Transport Queue

Graylog supports Apache Kafka as a transport for various inputs such as GELF, syslog, and raw/plaintext inputs. The Kafka topic can be filtered by a regular expression, and depending on the input, various additional settings can be configured.

Using RabbitMQ (AMQP) as a Transport Queue

Graylog supports AMQP as a transport for various inputs such as GELF, syslog, and raw/plaintext inputs. It can connect to any AMQP broker supporting AMQP 0-9-1 such as RabbitMQ.

Input Throttling

Throttling allows certain Graylog inputs to slow their message intake rates (by temporarily pausing intake processing) if contention occurs in the Graylog Journal.

Graylog Inputs that Support Throttling

  • AWS Flow Logs (deprecated)
  • AWS Cloudtrail
  • AWS Logs (deprecated)
  • CEF AMQP Input
  • CEF Kafka Input
  • GELF AMQP
  • GELF Kafka
  • JSON path from API
  • Raw/Plaintext AMQP
  • Raw/Plaintext Kafka
  • Syslog AMQP
  • Syslog Kafka

Enabling Throttling

To enable throttling for one of these inputs, edit it in System > Inputs and select the Allow throttling this input checkbox.

Throttling Criteria

When enabled, the following criteria will be used to determine if throttling will occur:

  1. If there are zero uncommitted entries in the Graylog journal, throttling will not occur. No further checks will be performed.
  2. Throttling will occur if the Journal has more than 100k uncommitted entries.
  3. Throttling will occur if the Journal is growing in size rapidly (approximately 20k entries per second or greater).
  4. Throttling will occur if the process ring buffer is full.
  5. If nothing is currently being written to the Journal; throttling will not occur. No further checks will be performed.
  6. Throttling will occur if the Journal is more than 90% full.
  7. Throttling will occur if the Journal write rate is more than twice as high as the read rate.