DSV File from HTTP Data Adapter

A delimiter-separated values (DSV) file contains plain text data organized into columns and rows.

This adapter can be used to overcome some of the restrictions of the CSV file dapter, primarily that the CSV file needs to exist locally on all Graylog server nodes.

Hint: The DSV data adapter can read key value pairs (or check for the presence of a key) from a DSV file and is only capable of returning a single value column and cannot handle multi-values.This adapter allows Graylog to retrieve and parse data from a DSV file hosted on a web server.

Prerequisites

Before proceeding, ensure that the following prerequisites are met:

  • Ensure your DSV file is formatted accordingly:

    • The first line in the DSV file needs to be a list of field or column names.

    • The file uses utf-8 encoding.

    • The file is readable by every Graylog server node.

  • The Graylog server must have outbound network connectivity to reach the URL where the DSV file is hosted. Verify that there are no firewall rules or network restrictions blocking HTTP(S) traffic from the Graylog server to the specified URL.

  • Obtain the exact URL of the DSV file that you want Graylog to fetch. Ensure that this URL points to a valid DSV file hosted on a web server accessible to the Graylog server.

  • Understand the structure of the DSV file including the delimiter (separator), line separator, quote characters, and any special formatting considerations (e.g. header rows, comments). This understanding helps in correctly configuring parsing parameters in Graylog.

  • Ensure that the Graylog server has appropriate permissions to access and read the DSV file from the specified URL. If authentication is required (e.g. basic authentication or API tokens), ensure that credentials are correctly configured.

  • Depending on the content of your DSV file, consider any specific parsing considerations such as handling quoted fields, escaping characters, or ignoring certain characters or lines.

Configure the Data Adapter

You can create a data adapter during the lookup table creation workflow, or they can be created separately on the Data Adapters tab. The following configuration options are available for this data adapter:

Title

A short and unique title for this data adapter.

Description

Data adapter description.

Name

The name used to refer to this data adapter. This should be something unique and recognizable within your Graylog environment.

Custom Error TTL

Specifies the time to live (TTL) for custom error messages if any occur during the API interaction. Enter a value in seconds, such as 3600 for 1 hour, depending on your logging and troubleshooting needs.

File URL

Enter the URL from which the DSV file will be fetched. This URL should point to the location of the DSV file on a web server accessible to the Graylog server, e.g. https://example.org/table.csv.

Note that you must add this URL to the URL whiltelist.

Refresh interval

Define how often (in seconds) Graylog should fetch the DSV file from the specified URL to check for updates or changes.

Separator

Specify the delimiter character that separates fields within each line of the DSV file. Common separators include commas (,) and semicolons (;).

Line Separator

Define the character sequence that marks the end of each line in the DSV file. This is typically \n (newline) or \r\n (carriage return + newline) depending on the file format.

Quote character

The character to use for quoted elements.

Ignore characters

Specify any characters that should be ignored when parsing the DSV file. These could include whitespace characters or other non-data characters.

Key column

The column number that should be used for the key lookup.

Value column

The column number or index (starting from 0) that contains the corresponding value or data for each entry in the DSV file.

Allow case-insensitive lookups  

Select this checkbox if the key lookup should be case-insensitive.

Check Presence Only

Select this checkbox to verify if the key is present in the table; returns a boolean instead of the value.

Example DSV File from HTTP Pipeline Rule

This rule performs a lookup in the test-dsv lookup table using the key "a" and writes the returned value into the message field dvs_result.

Copy 
rule "DSV Example Rule"
when
    true
then
    let lookup = lookup_value(
        lookup_table: "test-dsv",
        key: to_string("a"),
        default: "_undefined_"
        );
        
    set_field("dvs_result", lookup);
end

Further Reading

Explore the following additional resources and recommended readings to expand your knowledge on related topics: