By: Ming Zhou

Read Time: 3 min

The Extract and Aggregate fields feature allows users to custom parse historical logs (post-ingestion) and get an aggregated count on those newly parsed fields.

Enterprise SREs work with large systems that consist of internally built components and external products. Debugging with logs from external products can be extremely challenging. Users are working with logs that were sent from a third party system that they have no control over with no access to third party development teams to understand how that team logs. 

In these scenarios, users find that LogDNA hasn’t parsed logs in the way that they want (because it’s not in a format that LogDNA recognizes) and it’s too late to set up Custom Parsing because the logs they’re working on have already been indexed. Moreover, users won’t know what format the logs will be in until they see them for the first time, setting up custom parsing rules to anticipate this would be difficult. 

To solve this pain, LogDNA’s Extract and Aggregate fields allow users to create custom parsing rules and apply them to logs that have already been ingested. This allows for greater flexibility to switch up parsing rules in real-time to figure out which portion of the log is most useful to be parsed into fields. 

Custom and existing parsed fields will also be aggregated to give insights (metrics) to help the user further diagnose the issue. You can find this feature by expanding any log line in the log viewer. 

Create custom parsing rules and apply them to ingested logs with LogDNA’s Extract and Aggregate fields.

Pro tips of Extract and Aggregate fields

  • Get aggregated information from fields that have already been auto parsed by LogDNA without using the custom parsing portion, simply check the fields under the auto-parsed section
  • Sort results by clicking on any column name
  • Drag and drop the fields to arrange the table in the order that you prefer 
  • Toggle the count vs percentage checkbox to get a raw count or a percentage
  • Table results are cached but not saved
  • The table can be downloaded in a CSV for further analysis
  • LogDNA can only process as far back as your retention period
Create custom parsing rules and apply them to ingested logs with LogDNA’s Extract and Aggregate fields.

We are so excited to offer you greater control and the ability to manipulate your data to get the most out of your logs. As always, please email me at ming@logdna.com for feedback and suggestions on how we can improve for the future. 

Docs

 

About Ming Zhou

RELATED POSTS

LogDNA Kubernetes Enrichment Early Access

Introducing Kubernetes Enrichment Early Access

With more engineering teams adopting Kubernetes as their container orchestration platform, new challenges emerge in giving your entire team visibility into Kubernetes for monitoring, debugging,...

Kubernetes

Introducing LogDNA Web Server Template

With the ever-growing volume of application logs and analysis tools available, it can be time-consuming to set up your observability tools to keep up with...

Product Updates

LogDNA Agent Updates

Today, we are announcing a few changes around the LogDNA Agent happening today and in mid-October. Ending WinEvents Support Starting today, we begin officially recommending...

Product Updates

Get started with LogDNA