Working with fields

All the fields that we have used so far were either indexed fields (such as host, sourcetype, and _time) or fields that were automatically extracted from key=value pairs. Unfortunately, most logs don't follow this format, especially for the first few values in each event. New fields can be created either inline, by using commands, or through configuration.

A regular expression primer

Most of the ways to create new fields in Splunk involve regular expressions (sometimes referred to as REGEX). As mentioned in the Splunk documentation:

"Regex is a powerful part of the Splunk search interface, and understanding it is an essential component of Splunk search best practices".

There are many books and sites dedicated to regular expressions, so we will only touch upon the subject here. The following examples are really provided for completeness; the Splunk web interface may suffice for most users.

Given the log snippet ip=1.2.3.4, let's pull out the subnet (1.2.3) into a new field called subnet. The simplest pattern would be the following literal string:

ip=(?P<subnet>1.2.3).4

This is not terribly useful as it will only find the subnet of that one IP address. Let's try a slightly more complicated example:

ip=(?P<subnet>\d+\.\d+\.\d+)\.\d+

Let's step through this pattern:

• ip= simply looks for the raw string ip=.
• ( starts a capture buffer. Everything until the closing parenthesis is part of this capture buffer.
• ?P<subnet> immediately inside the parenthesis, says create a field called subnet from the results of this capture buffer.
• \d matches any single digit, from 0 to 9.
• + says one or more of the item immediately before.
• \. matches a literal period. A period without the backslash matches any character.
• \d+\.\d+ matches the next two parts of the IP address.
• ) ends our capture buffer.
• \.\d+ matches the last part of the IP address. Since it is outside the capture buffer, it will be discarded.

Now let's step through an overly complicated pattern to illustrate a few more concepts:

ip=(?P<subnet>\d+.\d*\.[01234-9]+)\.\d+

Let's step through this pattern:

• ip= simply looks for the raw string ip=.
• (?P<subnet> starts our capture buffer and defines our field name.
• \d means digit. This is one of the many backslash character combinations that represent some sets of characters.
• + says one or more of what came before, in this case \d.
• . matches a single character. This will match the period after the first set of digits, though it would match any single character.
• \d* means zero or more digits.
• \. matches a literal period. The backslash negates the special meaning of any special punctuation character. Not all punctuation marks have a special meaning, but so many do that there is no harm adding a backslash before a punctuation mark that you want to literally match.
• [ starts a character set. Anything inside the brackets will match a single character in the character set.
• 01234-9 means the characters 0, 1, 2, 3, and the range 4-9.
• ] closes the character set.
• + says one or more of what came before, in this case the character set.
• ) ends our capture buffer.
• \.\d+ is the final part of the IP address that we are throwing away. It is not actually necessary to include this, but it ensures that we only match if there were, in fact, four sets of numbers.

There are a number of different ways to accomplish the task at hand. Here are a few examples that will work:

• ip=(?P<subnet>\d+\.\d+\.\d+)\.\d+
• ip=(?P<subnet>(\d+\.){2}\d+)\.\d+
• ip=(?P<subnet>[\d\.]+)\.\d
• ip=(?P<subnet>.*?\..*?\..*?)\.
• ip=(?P<subnet>\S+)\.

For more information about regular expressions, consult the man pages for Perl Compatible Regular Expressions (PCRE), which can be found online at http://www.pcre.org/pcre.txt, or one of the many regular expression books or websites dedicated to the subject. We will build more expressions as we work through different configurations and searches, but it's definitely worthwhile to have a reference handy.

Commands that create fields

In Splunk, fields are extracted from the event data; to fully leverage the power of Splunk, you have the ability to create additional fields or to have Splunk extract additional fields that you define. This allows you to capture and track information that is important to your needs, but which is not automatically discovered and extracted by Splunk.

There are a number of commands that create new fields, but the most commonly used are eval and rex.

eval

The eval command allows you to use functions to build new fields, much as you would build a formula column in Excel, for example:

sourcetype="impl_splunk_gen" | eval req_time_seconds=date_second/1000 | stats avg(req_time_seconds)

This creates a new field called req_time_seconds on every event that has a value for date_second. Commands after this statement see the field as if it were part of the original event. The stats command then creates a table of the average value of our newly created field.

There are a huge number of functions available for use with eval. The simplest way to find the full listing is to search Google for Splunk eval functions. I would suggest bookmarking this page as you will find yourself referring to it often.

rex

The rex command lets you use regular expressions to create fields. It can work against any existing field but, by default, will use the field _raw. Let's try one of the patterns that we wrote in our short regular expression primer:

sourcetype="impl_splunk_gen" | rex "ip=(?P<subnet>\d+\.\d+\.\d+)\.\d+" | chart values(subnet) by date_minute

This would create a table like this:

With the addition of the field argument, we can work against the ip field that is already being created automatically from the name=value pair in the event.

sourcetype="impl_splunk_gen" | rex field=ip "(?P<subnet>.*)\."| chart values(subnet) by date_minute

This will create exactly the same result as the previous example.

Extracting loglevel

In some of our examples, we searched for the raw word error. You may have noticed that many of the events weren't actually errors, but simply contained the word error somewhere in the message. For example, given the following events, we probably only care about the second event:

2012-03-21T18:59:55.472-0500 INFO This is not an error
2012-03-21T18:59:42.907-0500 ERROR Something bad happened

Using an extracted field, we can easily create fields in our data without re-indexing, that allows you to search for values that occur at a specific location in your events.

Using the extract fields interface

There are several ways to define a field. Let's start by using the Extract Fields interface. To access this interface, choose Extract Fields from the workflow actions menu next to any event:

This menu launches the Extract Fields view:

In Splunk version 6.2, we have access to a wizard which helps us provide the information required for Splunk to attempt to build a regular expression that matches.

Although you may choose multiple fields, in this case, we specify Error:

In the popup, you can provide a custom Field Name (I chose CognosError) and then click the button labeled Add Extraction.

Under Preview, you can see two tabs – Events and our new field CongosError:

Under Events, we get a preview of what data was matched in context, and under CongosError we can see our new field.

Finally, under Show Regular Pattern, we see the regular expression that Splunk generated, which is as follows:

^(?P<CognosError>\w+)

You can step through the pattern and, if you are so inclined, make edits to it.

Clicking on the button labeled Edit the Regular Expression (shown in the preceding screenshot) presents a dialog to let you modify the pattern manually:

Once you make any edits to the pattern, Preview will be enabled and will launch a new search with the pattern loaded into a very useful query that shows the most common values extracted.

Save prompts you for a name for your new field. Assuming that you modified the originally generated pattern string, you can enter a new name (rather than CongosError), and then select the desired permissions for accessing this new field:

Now that we've defined our field, we can use it in a number of ways, as follows:

• We can search for the value using the fieldname; for instance, loglevel=CognosError
• When searching for values by fieldname, the fieldname is case sensitive, but the value is not case sensitive. In this case loglevel=CognosError would work just fine, but LogLevel=cognoserror would not.
• We can report on the field, whether we searched for it or not. For instance: sourcetype="impl_splunk_gen" user=mary | top loglevel
• We can search for only events that contain our field: sourcetype="impl_splunk_gen" user=mary loglevel="*"

Using rex to prototype a field

When defining fields, it is often convenient to build the pattern directly in the query and then copy the pattern into the configuration. You might have noticed that the test in the Extract Fields workflow used rex.

Let's turn the subnet pattern which we built within the section A regular expression primer into a field. First, we build the query with the rex statement:

sourcetype="impl_splunk_gen" ip="*"
| rex "ip=(?P<subnet>\d\.\d\.\d+)\.\d+"
| table ip subnet

Since we know there will be an ip field in the events which we care about, we can use ip="*" to limit the results only to events that have a value for that field.

The table command takes a list of fields and displays a table, one row per event:

As we can see, the rex statement doesn't always work. Looking at the pattern again, you may notice that the first two instances of \d are now missing their trailing +. Without the plus sign, only addresses with a single digit in both their first and second sections will match. After adding the missing plus signs to our pattern, all rows will have a subnet.

sourcetype="impl_splunk_gen" ip="*"
| rex "ip=(?P<subnet>\d+\.\d+\.\d+)\.\d+"
| table ip subnet

We can now take the pattern from the rex statement and use it to build a configuration.

Using the admin interface to build a field

Taking our pattern from the previous example, we can build the configuration to wire up this extract.

First, click on Settings in the upper menu bar. From there, select Fields. The Fields section contains everything, funnily enough, about fields.

Here you can view, edit, and set permissions on field extractions, define event workflow actions, field aliases, and even rename sourcetypes.

For now, we're interested in Field extractions.

After clicking on Add new to the right of Field extractions, or on the New button after clicking on Field extractions, we are presented with the interface for creating a new field.

Now, we step through the fields:

• Destination app lets us choose the app where this extraction will live and where it will take effect, by default. We will discuss the scope of configurations in Chapter 11, Configuring Splunk.
• Name is simply a display name for the extraction. Make it as descriptive as you like.
• Apply to lets you choose what to bind this extraction to. Your choices are sourcetype, source, and host. The usual choice is sourcetype.
• named is the name of the item we are binding our extraction to.
• Type lets you choose Inline, which means specifying the regular expression here, or Uses transform, which means we will specify a named transform that exists already in the configuration.
• Extraction/Transform is where we place either our pattern, if we chose a Type option of Inline, or the name of a Transform object.

Once you click on Save, you will return to the listing of extractions. By default, your extraction will be private to you and will only function in the application it was created in. If you have the rights to do so, you can share the extraction with other users and change the scope of where it runs. Click on Permissions in the listing to see the permissions page, which most objects in Splunk use.

The top section controls the context in which this extraction will run. Think about when the field would be useful, and limit the extractions accordingly. An excessive number of extractions can affect performance, so it is a good idea to limit the extracts to a specific app when appropriate. We will talk more about creating apps in Chapter 8, Working with Apps.

The second section controls what roles can read or write this configuration. The usual selections are the Read option for the Everyone parameter and the Write option for the admin parameter. As you build objects going forward, you will become very familiar with this dialog. Permissions and security, in general, can be complex and affect where an app will eventually be visible—the reader is advised to take the time to review whether the permissions set for apps are actually what is expected.

Indexed fields versus extracted fields

When an event is written to an index, the raw text of the event is captured along with a set of indexed fields. The default indexed fields include host, sourcetype, source, and _time. There are distinct advantages and a few serious disadvantages to using indexed fields.

First, let's look at the advantages of an indexed field (we will actually discuss configuring indexed fields in Chapter 11, Configuring Splunk):

• As an indexed field is stored in the index with the event itself, it is only calculated at index time and, in fact, can only be calculated once at index time.
• It can make finding specific instances of common terms efficient. See Indexed field case 1 – rare instances of a common term in the following section, as an example.
• You can create new words to search against those which simply don't exist in the raw text or are embedded inside a word. See from Indexed field cases 2 – splitting words to Indexed field cases 4 – slow requests in the following sections.
• You can efficiently search for words in other indexed fields. See the Indexed field case 3 – application from source section.

Now for the disadvantages of an indexed field:

• It is not retroactive. This is different from extracted fields where all events, past and present, will gain the newly defined field if the pattern matches. This is the biggest disadvantage of indexed fields and has a few implications, as follows:
  • Only newly indexed events will gain a newly defined indexed field. If the pattern is wrong in certain cases, there is no practical way to apply the field to already indexed events.
  • Likewise, if the log format changes, the indexed field may not be generated (or generated incorrectly).
• It adds to the size of your index on disk.
• It counts against your license.
• Any changes will require a restart to be applied and disrupt data flow temporarily.
• In most cases, the value of the field is already an indexed word, in which case creating an indexed field will likely have no benefit, except in the rare cases where that value is very common.

With the disadvantages out of the way, let's look at a few cases where an indexed field would improve search performance and then at one case where it would probably make no difference.

Indexed field case 1 – rare instances of a common term

Let's say your log captures process exit codes. If 1 represents a failure, you probably want to be able to search for this efficiently. Consider a log that looks something like this:

4/1/12 6:35:50.000 PM process=important_process.sh, exitcode=1

It would be easy to search for this log entry using exitcode=1. The problem is that when working with extracted fields, the search is effectively reduced to this:

1 | search exitcode="1"

Since the date contains 1, this search would find every event for the entire day and then filter the events to the few that we are looking for. In contrast, if exitcode were defined as an indexed field, the query would immediately find the events, only retrieving the appropriate events from the disk. Please note that binding an indexed field to any time (stamp) is risky. This wreaks havoc on data integrity and is not considered best practice.

Indexed field case 2 – splitting words

In some log formats, multiple pieces of information may be encoded into a single word without whitespace or punctuation to separate the useful pieces of information. For instance, consider a log message such as this:

4/2/12 6:35:50.000 PM kernel: abc5s2: 0xc014 (UNDEFINED).

Let's pretend that 5s2 (a made-up string of characters for an example) is an important piece of information that we need to be able to search for efficiently. The query *5s2 would find the events but would be a very inefficient search (in essence, a full table scan). By defining an indexed field, you can very efficiently search for this instance of the string 5s2, because essentially, we create a new word in the metadata of this event.

Defining an indexed field only makes sense if you know the format of the logs before indexing, if you believe the field will actually make the query more efficient (see previous section), and if you will be searching for the field value. If you will only be reporting on the values of this field, an extracted field will be sufficient, except in the most extreme performance cases.

Indexed field case 3 – application from source

A common requirement is to be able to search for events from a particular web application. Often, the only easy way to determine the application that created the logs is by inspecting the path to the logs, which Splunk stores in the indexed field source. For example, given the following path, the application name is app_one:

/opt/instance19/apps/app_one/logs/important.log

You could search for this instance using source="*/app_one/*", but this effectively initiates a full table scan. You could define an extracted field and then search for app="app_one", but unfortunately, this approach will be no more efficient because the word we're looking for is not contained in the field _raw. If we define this field as an indexed field, app="app_one" will be an efficient search.

Once again, if you only need this field for reporting, the extracted field is just fine.

Indexed field case 4 – slow requests

Consider a web access log with a trailing request time in microseconds:

[31/Jan/2012:18:18:07 +0000] "GET / HTTP/1.1" 200 7918 ""
"Mozilla/5.0..." 11/11033255

Let's say we want to find all requests that took longer than 10 seconds. We can easily extract the value into a field, perhaps request_ms. We could then run the search request_ms>10000000. This query will work, but it requires scanning every event in the given time frame. Whether the field is extracted or indexed, we would face the same problem as Splunk has to convert the field value to a number before it can test the value.

What if we could define a field and instead search for slow_request=1? To do this, we can take advantage of the fact that, when defining an indexed field, the value can be a static value. Having Splunk search for a static value – rather than examining the value of every event and then trying to match it – would improve the efficiency of the search. This could be accomplished with a transform, like so:

REGEX = .*/(\d{7,})$
FORMAT = slow_request::1

We will cover transforms, and the configurations involved, in Chapter 11, Configuring Splunk.

Once again, this is only worth the trouble if you need to efficiently search for these events and not simply report on the value of request_ms.

Indexed field case 5 – unneeded work

Once you learn to make indexed fields, it may be tempting to convert all your important fields into indexed fields. In most cases, it is essentially a wasted effort and ends up using extra disk space, wasting license, and adding no performance boost.

For example, consider this log message:

4/2/12 6:35:50.000 PM [vincentbumgarner] [893783] sudo bash

Assuming that the layout of this message is as follows, it might be tempting to put both userid and pid into indexed fields:

date [userid] [pid] action

Since the values are uncommon, and are unlikely to occur in unrelated locations, defining these fields as indexed fields is most likely wasteful. It is much simpler to define these fields as extracted fields and shield ourselves from the disadvantages of indexed fields.