Preview Logs from a Data Lake

The following article exclusively pertains to a Graylog Enterprise feature or functionality. To learn more about obtaining an Enterprise license, please contact the Graylog Sales team.

Data Lake preview provides visibility into log data stored in your Graylog Data Lake. You can preview and examine logs before retrieving your data. Also, you can preview data without affecting license usage because log data counts against license usage only upon retrieval if it hasn’t previously been sent to your search backend.

Prerequisites

Before proceeding, ensure that the following prerequisites are met:

  • You must be a Graylog administrator or have the Data Lake User role to preview data and retrieve logs from a Data Lake.

  • You must have an existing Data Lake containing log data.

Data Lake Preview vs. Search

Data Lake preview is not the same as search and search queries in Graylog. Search operates on the indexed data in your search backend, and you are able to customize queries against that data. While the Data Lake Preview page has some similar controls to search, it has some limitation in terms of the fields you can search for and the controls you can use.

To preview data in a Data Lake, you select a specific stream that is routed to your Data Lake, then you can target desired log data through the use of filters and by adjusting the time range associated with the timestamp of the stored data.

Hint: The filters available on the Data Lake are different from those available for Graylog search. Data Lake filters are based on information available from the Data Lake storage system infrastructure rather than from the original streams. You are not able to apply filters from custom field names.

Preview Log Data

To preview data in a Data Lake:

  1. Navigate to Data Lake > Preview.

  2. Select the stream from which you want to preview log data.

  3. (Optional) Select the time range from which you want to preview log data. By default, the range is set to preview the past 30 minutes of logs.

  4. (Optional) Apply filters to limit the results. Click Filter by fields, then select the field name from the drop-down list to filter by, and enter a value to filter on. As noted above, the filters available are based on backend infrastructure. You are not able to create your own filters or queries.

    Click Add field to add additional filters. When using multiple filters, you can use AND and OR logic. You can include as many filters as needed, but they will all use the same join logic. Keep in mind that while preview can apply many filters, a data retrieval operation can use no more than three filters collectively!

    Hint: Filtering on the stream or associated_assets fields can result in extended processing times because these operations require intensive in-memory computation and data transfer. For better performance, we recommend using these filters with a narrow time range and limiting the number of additional filters applied.

  5. Click Perform Search.

If your Data Lake contains a significant amount of data, a preview request might take a considerable amount of time. Running and queued jobs can be seen in the Data Lake Jobs section on the Overview tab at Data Lake > Setup.

Hint: An individual user cannot execute more than one preview at a time. Preview job execution is limited to four hours. If a job requires more than four hours complete, it fails.

Matching logs from the Data Lake are displayed in a list view widget. Your most recent preview request is maintained on the Preview page for 24 hours.

You can refine preview results by changing any of your previous selections, then clicking Perform Search again. You can edit the list view widget by clicking the Edit icon at the upper right. You can add or remove columns and order the columns so that you can more easily see your most relevant data.

Use the expand icon at the right of any row to view a pop up window that shows all data for that item, including the columns not displayed in the table. You can apply colored highlighting to fields or values in the results with the Highlighting option at the top left of the screen.

Hint: The preview widget is limited to displaying 500 results. If you have particularly large data sets, you might need to test different filter combinations to target your relevant data.

Retrieve Log Data from Preview

After you have previewed your log data, you can start a data retrieval operation from that subset of data if necessary. In fact, there are a couple of methods you can employ.

First, you can click Retrieve logs on the top right side of the preview list to open the log retrieval dialog box. When the form opens, it includes the same filters and other selections that your preview has. You can adjust any of these setting here to further refine your selection. However, note that the retrieval operation is limited to no more than three filters. Click Retrieve when you are ready to complete.

Second, you can retrieve specific messages from the preview results. Select the check box for any log message you want to retrieve, then click Bulk actions > Retrieve Logs. Click Retrieve in the confirmation dialog box that shows how many logs you have chosen to retrieve.

In either case, the retrieval job can be tracked on the Overview tab of the Data Lake > Setup page.

Further Reading

Explore the following additional resources and recommended readings to expand your knowledge on related topics: