Splunk Tutorial 10: How does Splunk read input data?

Imaging when you upload machine data into a Spunk engine, you are basically uploading a bunch of data that are most likely unstructured and cannot be understand or stored by traditional relational database in a structured way. However, Splunk can get you the result in less than one second when you search it. Splunk must have some special way to classify the data. Have you even image how does Splunk read your machine data?

The selling point of Splunk is its unique ability to index machine data. This ability allows Splunk to quickly search for analysis, reporting and alerts.

What? Splunk index machine data? Yes! Remember, from previous blog Splunk Tutorial 03: Licensing of Splunk 7.1.1, we have mentioned that Splunk is charged per indexed data. Splunk actually read your data by indexing it with it’s own way.

Following is how splunk index your data.

  1. Indexing Pipeline
  2. Indexes
  3. Search Head

Indexing Pipeline

After Splunk received the raw data, either from forwarder or user upload, it’s indexing Pipeline will firstly reads the machine data and then divide it into a lot of different events and identifies some default fields.

Remember, an even can means one line in the raw data or as complicated as a stack trace for over a few hundred lines.

Following 4 fields are always indexed:

[vtftable cols=”{0}0-2:d9d9d9;{/}”]
Field;;;Defintion;;;Example;nn;
Source;;;Identify the source of the data;;;WinEventLog:Security;nn;
SoureType;;;Identify what kind of data is it.;;;WinEventLog:Security ;nn;
Host;;;Name of the host or machine where the data come from;;;OraclePC-PC;nn;
_time;;;When did the event happened;;;28/07/2018;nn;
[/vtftable]

Following is an example:

Splunk Data Indexing example

However, Splunk always looks for all interesting fields from raw data index it.

Indexes

Raw data is then copied to the index and will be avalable during the search process.

Search Head

The searches head distributes the search across many indexes and consolidated the result.

Leave a Comment