By: Thu Nguyen
Read Time: 10 min
Searching in LogDNA is designed to be as intuitive and straightforward as possible. Just type in your search terms, and LogDNA will return your results almost instantaneously. For cases where you need to perform a more advanced search, or where you need greater control over your search results, LogDNA provides a number of features that can help you find exactly what you’re looking for.
In this article, we’ll walk through some of the expert-level features of LogDNA’s search engine and how you can use them to perform in-depth searches. We’ll start by explaining the basics of how LogDNA interprets searches, then we’ll present frequently asked questions about how searching works.
When you enter a query into the search input box, LogDNA parses it into a series of terms and operators.
A term is a string consisting of a single word, or a phrase surrounded by quotes. LogDNA scans log data to see if it contains your term(s), and if so, displays the matching log line. Terms are searched against the entire raw log line, but can be searched on a particular field if one is specified. By default, terms are also case insensitive and treated as prefixes. For example, searching lin will return log lines containing lin, line, and Linux.
Operators change how LogDNA interprets one or more search terms. Using operators gives you greater control over the search process in order to provide more relevant results. You can find a full list of operators with examples in the LogDNA search documentation page.
LogDNA also supports the boolean operators AND, OR, and NOT. Any whitespace between search terms is automatically interpreted as AND.
For example, searching warning error returns logs containing both warning and error. The only exception is when using lists, which treat whitespace as a part of the search term. For example, searching message:[file, exists] will only search for instances of exists that are preceded by a space.
Lastly, you can use parentheses to group operations that should be evaluated first. For example, the query source:node1 AND (file:/var/log/syslog OR ERROR) searches first for logs scanned from /var/log/syslog or that contain the word ERROR, then limits the results to node1. You can also use parentheses to group operations on a single field. For example, the query status:(>100 AND <=503) returns logs where the value stored in the status field is greater than 100 and less than or equal to 503.
You can add the exact match operator in different places for the same effect.
app:==lineparser.log is equivalent to
Most users would perform exact matches like this:
app:==lineparser OR app:==server
Here are other ways you can accomplish the same search in LogDNA:
1) app:(==lineparser OR ==server OR ==syslog) 2) ==app:(lineparser OR server OR syslog) 3) ==app:[lineparser,server,syslog]
You may find that one of your queries doesn’t return the results you expected, or displays a warning or error message. Here are some common pitfalls you may run into, and how to resolve them.
The most common questions we receive is not getting the results you expect. This is commonly due to search queries that are too vague, or queries that return a significant number of matches. LogDNA queries work best when they are specific. For example, searching for the or another common word would result in an excessive number of results, and not all of them will be displayed. For this reason, LogDNA doesn’t support single-character searches.
To avoid this problem, try using more specific terms or using the exact match operator (:==) . You should also search on a specific field, instead of terms matching the entire log.
Lastly, if you know exactly how your search term appears in your logs, enable case sensitivity by using the :=== operator.
After running a search, you may see results that don’t appear to match your query. LogDNA doesn’t just search visible fields, but also the raw log lines and internal metadata fields. You can click to expand a log line to see the raw data. Your query could match on these “invisible” fields, resulting in the log lines appearing in the results.
You can override this by specifying the field to search. For example, if your logs contain a level field and you want to find error-level logs, enter level:error as your query instead of just error.
If your query contains whitespace and doesn’t contain quotes, LogDNA treats the whitespace as an AND operator. For example, the query file does not exist will return any log containing all four words, regardless of their order. It could match file “myfile.txt” does not exist, but it could also match Could not open file: does it exist? Just like in Google, to search for an exact phrase, surround it in double quotes, “file does not exist“. Quotes will help escape characters as well.
The only exception is when searching in lists. With lists, whitespace is treated as part of the list item that it’s next to. For example, message:[file, exists] searches for logs where exists is preceded by a space. To avoid this, remove any whitespace between list items.
LogDNA provides visual feedback based on whether your query is properly formatted. A properly formatted field search will appear highlighted, as shown in this screenshot:
Well-formatted non-field searches won’t appear highlighted, but will still run without problems. However, if your query is improperly formatted or contains errors, the help icon on the right-hand side of the search box will turn into an alert explaining the problem in detail. In this screenshot, we’re trying to use a comparison operator on a string field, which is invalid:
A common issue is using symbols in your query, such as inequality signs. This displays the warning “Unsupported visualization query: symbols.” This warning is only relevant when using graphs or the timeline feature, but it doesn’t affect normal queries. The following example will still return results containing <hello:
If the warning persists, or if you’re not getting the expected results, you can escape the problematic characters by adding a backslash “\” before them or surround them in quotes For example, the following query searches for a specific phrase containing quotes:
message:”Host “logdna-host” not found”
This search is valid, but won’t return the expected results. Instead, you would use:
message:”Host \”logdna-host\” not found”
You can learn more in the LogDNA search documentation.
LogDNA’s search engine allows for a great deal of flexibility and control. It’s important to be aware of how LogDNA interprets each part of your query, especially for complex searches. Fortunately, LogDNA offers some of the fastest search results in the industry, so if at first you don’t succeed, you can try again right away.
First published on www.ibm.com on October 7, 2019. Written by: Norman Hsieh, VP of Business Development, LogDNA You know what they say: you can’t fix what you can’t...
First published as a case study on www.ibm.com on October 3, 2019. What is Log Analysis? IBM Cloud™ Log Analysis with LogDNA enables you to quickly find...
Single sign-on (SSO) is an authentication model designed to let users access different applications, services, and resources using a single set of credentials. Instead of...