Upgrade to Graylog 5.1

New Functionality

Index Default Configuration

Support for configuring index defaults has been added:

1. Adds the ability to specify Index Set initialization default settings in the server configuration file for new Graylog clusters.

2. Adds the ability to subsequently maintain the current Index Set defaults from the System > Configurations page and through the Graylog API.

New Graylog Cluster Index Set Initialization Defaults

New Graylog server clusters can now initialize the settings for Index Sets with the following server configuration values. Please see the sample graylog.conf file or the server.conf documentation for more details and example values.

• elasticsearch_analyzer

• elasticsearch_shards

• elasticsearch_replicas

• disable_index_optimization

• index_optimization_max_num_segments

• rotation_strategy

• elasticsearch_max_docs_per_index

• elasticsearch_max_size_per_index

• elasticsearch_max_time_per_index

• retention_strategy

• elasticsearch_max_number_of_indices

If you are using a pre-existing version of the graylog.conf configuration file, it is recommended that you review the aforementioned settings there before upgrading, to ensure the in-database defaults are established as expected with the upgrade. Although the graylog.conf sample configuration file now ships with all index default example properties commented out, you may be using an older version of the file where certain index default values were present and not commented-out.

All previously deprecated index set configuration properties in org.graylog2.configuration.ElasticsearchConfiguration have been un-deprecated, as Graylog intends to maintain them going forward.

Once the first Graylog server instance is started to establish the cluster, the following system indexes will be created with the specified defaults.

• Default index set

• Graylog Events

• Graylog System Events

• Graylog Message Failures

• Restored Archives

In-database Cluster Index Set Defaults

The current in-database defaults for new Index Sets can now be edited from the new System > Configuration > Index Set defaults configuration area. The default values set here will be used for all new index sets created:

• Those created from the System > Index Sets page.

• New indexes created through the Graylog Illuminate system.

Once the upgrade is installed, these in-database defaults will be established, and the server configuration option values described above will no longer be used.

The in-database Index Set default configuration can also be edited VIA the Graylog API:

curl 'http://graylog-server:8080/api/system/indices/index_set_defaults' \
  -X 'PUT' \
  -H 'Content-Type: application/json' \
  -H 'X-Requested-By: my-api-request' \
  --data-raw '
  {
    "index_analyzer": "standard",
    "shards": 1,
    "replicas": 0,
    "index_optimization_max_num_segments": 1,
    "index_optimization_disabled": false,
    "field_type_refresh_interval": 300,
    "field_type_refresh_interval_unit": "SECONDS",
    "rotation_strategy_class": "org.graylog2.indexer.rotation.strategies.SizeBasedRotationStrategy",
    "rotation_strategy_config": {
      "type": "org.graylog2.indexer.rotation.strategies.SizeBasedRotationStrategyConfig",
      "max_size": 32212254720
    },
    "retention_strategy_class": "org.graylog2.indexer.retention.strategies.DeletionRetentionStrategy",
    "retention_strategy_config": {
      "type": "org.graylog2.indexer.retention.strategies.DeletionRetentionStrategyConfig",
      "max_number_of_indices": 20
    }
  }'

New Index Default Values

Unless user-specified defaults are specified, the following new defaults will be effective for all new index sets created:

• Shards: 1 (previously 4 in many cases)

• Rotation Strategy: Time Size Optimizing - 30-40 days (previously Index Time [1D] in many cases)

Import Custom Sigma Rule Repositories

Previously, the official Sigma HQ rule repository was the only repository that could be used to import Sigma rules. Now any public Git repository containing Sigma rule source files can be imported to Graylog and rules can be imported from them. Since Graylog no longer reads directly from the Sigma HQ repository, it must be imported before new rules can be added from it. To expedite this process, on the Security > Sigma Rules > Sigma Repos page, there is an Add SigmaHQ button that will import the repository. The repository is about 10MB and all Sigma rule source files will be copied into MongoDB so the clone may take a minute to complete. Once the repository has been added rules can be imported as they were in 5.0.

The rules within imported repositories must conform to the Sigma specification in order to be successfully added. Since the repositories are stored locally they will not have the latest changes automatically applied but can easily be refreshed in the Sigma Repos tab using the Refresh menu item for each repository.

Removal of Deprecated Inputs

The following inputs are no longer available:

• AWS Logs (deprecated)

• AWS Flow Logs (deprecated)

The inputs were marked as deprecated since Graylog version 3.2.0. If you still run any of those inputs, please configure the alternative Kinesis/CloudWatch input instead ahead of upgrading.

Java API Changes

The following Java Code API changes have been made.

All previously deprecated index set configuration properties in org.graylog2.configuration.ElasticsearchConfiguration have been un-deprecated, as Graylog intends to maintain them going forward.

REST API Endpoint Changes

Change to the format of Input entities in API responses

This change applies to the format of input entities in responses to the resources at or beneath /system/inputs and plugins/org.graylog.plugins.forwarder/forwarder/profiles.

Input configuration may now contain values of type EncryptedValue. Sensitive input configuration values for various inputs may be stored encrypted from now on and will therefore be represented differently in the JSON response.

For example, an input previously rendered like this in a response:

{
  "id": "63f489ee73561d699b210677",
  "attributes": {
    "not_so_secret_value": "plaintext",
    "secret_value": "plaintext",
    ...
  },
  ...
}

will be returned like this if the secret attribute contains a sensitive value:

{
  "id": "63f489ee73561d699b210677",
  "attributes": {
    "not_so_secret_value": "plaintext",
    "secret_value": {
      "is_set": true
    },
    ...
  },
  ...
}

Added Optional Default Timezone Configuration for Syslog Inputs

When creating or editing a new syslog input, it is now possible to configure a default timezone in case logs ingested are not sending dates in UTC. When left as Not configured, the system behaves as before.

The following REST API endpoints were changed:

Behavior Changes

• The JSON path value from HTTP API input will now only run on the leader node if the Global option has been selected in the input configuration. Previously, the input was started on all nodes in the cluster.

• The default connection and read timeouts for email sending have been reduced from 60 seconds to 10 seconds.

• We are now parsing the time zone information send by Fortigate syslog messages. Any workarounds to transform the date into the right timezone because the forwarded timezone information was not honored should be removed.

Configuration File Changes