Fundamentally, a Graylog setup needs data in it. This article explains the basic principles of getting your data into the system.
What Are Graylog Inputs?
Message inputs are responsible for accepting log messages in Graylog. Some default message types are available by default in Graylog. But you might need to install additional plugins to enable Graylog to receive particular messages.
See the Graylog documentation on inputs for more information. Some logs have individual inputs. Some of them are listed below in the individual inputs section. In addition to that, the Graylog Marketplace contains some inputs created by the community.
Although we can’t cover all possible options and device configurations, the following describes some of the fundamentals of common log sources and the current best practices to ingest these kind of messages into Graylog. Some can be done native, and others need a collector. The following includes a few examples of various available collectors:
- Ingest syslog
- Ingest journald
- Ingest Windows eventlog
- Ingest CEF
- Ingest Raw/Plaintext
- Ingest GELF
- Ingest from files
- Ingest JSON path from API
- Ingest Application Data
These listed inputs are not necessarily natives available in Graylog. Most are dedicated to an individual product or framework. The default inputs like syslog, CEF, GELF, or the RAW/Plaintext are not listed individually.
Using Apache Kafka as Transport Queue
Graylog supports Apache
Kafka as a transport for various inputs such as GELF, syslog, and Raw/Plaintext inputs. The Kafka topic can be
filtered by a regular expression and depending on the input, various additional settings can be configured.
Learn how to use rsyslog and Apache Kafka in the Sending syslog via Kafka into Graylog guide.
Using RabbitMQ (AMQP) as a Transport Queue
AMQP as a transport for various inputs such as GELF, syslog, and Raw/Plaintext
inputs. It can connect to any AMQP broker supporting AMQP
0-9-1 such as RabbitMQ.
Learn how to use rsyslog and RabbitMQ in the Sending syslog via AMQP into Graylog guide.
Throttling allows certain Graylog Inputs to slow their message intake rates (by temporarily pausing intake processing) if contention occurs in the Graylog Journal.
Graylog Inputs that Support Throttling
- AWS Flow Logs (deprecated)
- AWS Cloudtrail
- AWS Logs (deprecated)
- CEF AMQP Input
- CEF Kafka Input
- GELF AMQP
- GELF Kafka
- JSON path from API
- Raw/Plaintext AMQP
- Raw/Plaintext Kafka
- Syslog AMQP
- Syslog Kafka
To enable throttling for one of these inputs, edit it in System > Inputs and select the Allow throttling this input checkbox.
When enabled, the following criteria will be used to determine if throttling will occur:
- If there are zero uncommitted entries in the Graylog Journal, throttling will not occur. No further checks will be performed.
- Throttling will occur if the Journal has more than 100k uncommitted entries.
- Throttling will occur if the Journal is growing in size rapidly (approximately 20k entries per second or greater).
- Throttling will occur if the process ring buffer is full.
- If nothing is currently being written to the Journal; throttling will not occur. No further checks will be performed.
- Throttling will occur if the Journal is more than 90% full.
- Throttling will occur if the Journal write rate is more than twice as high as the read rate.