Logging Best Practices, Part 3: Text-Based Logs and Structured Logs

4 MIN READ
MIN READ
TABLE OF CONTENTS
    4 MIN READ
    MIN READ

    Isn't all logging pretty much the same? Logs appear by default, like magic, without any further intervention by teams other than simply starting a system… right?

    While logging may seem like simple magic, there's a lot to consider. Logs don't just automatically appear for all levels of your architecture, and any logs that do automatically appear probably don't have all of the details that you need to successfully understand what a system is doing. We've talked about actionable logs, log levels, logs for all components and needs, and different methodologies for generating logs in parts 1 and 2 of this Logging Best Practices series. Now let's explore best practices specifically for text-based logging.

    There are two general methodologies for logging in modern architectures: text-based logging and structured logging. Before we can even compare the two techniques, we should define what each term means.

    What are They?

    Text-based logs are a series of text-based messages, one for each logged event, with a timestamp included. Generally, these text-based messages are strings, and they're intended to be human-readable. The log message is often constructed with one or more variables in the string to enable specific data to surface for the user to examine. That practice can make the logs appear fairly uniform. However, a machine trying to parse that message based on rules can be easily tricked by a variable containing some similar part of the message template by accident, leading to misparsed logs. For this reason, we typically say humans are the primary user for text-based logging.

    Structured logs, on the other hand, are a series of data-enriched objects, one object for each logged event. You can think of them as sets of metadata about a specific event, always including a timestamp and typically also including information on the system, platform, application, and various functions. Examples of structured log formats are JSON and XML; JSON is by far the de facto standard simply because it is so widespread. Unlike the data shared in text-based logs, the data from structured logs are tagged in their entirety and therefore easily discernible for a machine. While nearly every structured logging design includes a human-readable field to help with troubleshooting, humans are no longer the primary audience for logs with this methodology. Instead, we intend for machines to use the logs, and then humans are the audience for the resulting output.

    How Are They Used?

    In general, structured logs are preferred over text-based logs when you need to perform automated functions like alerting, monitoring, and analytics or large data-crunching jobs like searching. Structured logs are easier for a machine to parse and act upon, and that ease in parsing leads to faster search times and cleaner output. You'll also get a lot more consistency on multiple platforms. Remember: With structured logs, machines are now the primary user, not a human. You're not using the logs directly, just the machine-parsed results.

    There are best practices to consider for each methodology. Most of the best practices are good common sense, but there are things to consider based on the intended audience and overall use of each type of log. We'll explore those best practices next.

    false
    false
    Laura Santamaria

    7.24.20

    As LogDNA’s Developer Advocate, Laura Santamaria loves to learn and explain how things work. She bridges the gap between external developers and SREs and internal engineering teams. In addition, she is the curator for A Minute on the Mic. Apart from work, she co-hosts Austin DevOps and Cloud Austin, taught Python for Women Who Code Austin for many years, is an organizer for DevOpsDays Texas, and volunteers with DevOpsDays Austin. Outside of tech, Laura runs, plays with her dogs, throws discs, and watches clouds—the real kind.

    SHARE ARTICLE

    RSS FEED