7 Free, Open Source Log Management Tools
Logs contain a wealth of data about applications, servers, and other IT resources. When used effectively, the proper log management tools will help increase stability, resolve problems, gain valuable insights, mitigate risks, and improve efficiency. It’s no wonder hosted logging platforms have become increasingly popular – it’s become nearly impossible to collect, parse, centralize, and analyze logs manually. While most feature-rich log management software are paid offerings, these open source/free log management tools are a great way to begin without the added expense. As your applications and logging needs grow, you should weigh the total cost of self-hosting or building your own logging tools versus paying the cost of feature-rich, hosted log management systems.
The best free log management tools available today
Many of these tools provide the features of hosted services, without the added expense.
1. The Elastic Stack
The Elastic Stack is one of the most widely recognized logging platforms. It contains all of the tools necessary to ship, ingest, parse, analyze, and visualize logs from countless different applications and services. As its components are all open source, the Elastic Stack has an extensive community of third-party developers, integrations, and tools. ELK consists of four separate open source projects, developed by the same team:
- Elasticsearch – a powerful and scalable search and analytics engine
- Logstash – a log ingestion tool with a large user community and variety of plugins.
- Kibana – a data visualization tool for Elasticsearch with charts and graphs
- Beats – a set of agents that collect and send data to Logstash
Although the Elastic Stack is free to use, it’s not free to run. Hosting costs can quickly become expensive, especially as your product grows at an enterprise scale. The Elastic Stack is also complex, requiring extensive customization (and often specialized personnel) before it can be deployed to production. Certain features common to other tools—such as alerting, graphing, and user authentication—are only available through the X-Pack, a paid addition. Read on to learn about the True Cost of the Elastic Stack.
- Complete stack for collecting and managing logs
- Open source base with a large development community
- Scalable from small to enterprise-level deployments
- Steep learning curve
- Expensive to host and maintain
- Requires extensive upfront and ongoing customization
- Alerting, graphing, security, and other features are NOT free - they are paid additions
2. Fluentd - A Logstash Alternative
There is a trend towards replacing Logstash for Fluentd to make the EFK slack. Fluentd is an open source log aggregation tool similar to Logstash and is a favorite amongst Kubernetes deployments because of its small footprint, better plugin library and ability to add useful meta data.
Graylog is a complete log management stack similar to the Elastic Stack. Like the Elastic Stack, it uses Elasticsearch as its indexing backend, can start ingesting log data out-of-the-box, and provides a web-based user interface with tools for searching and graphing log data. Graylog’s However, its UI also provides features such as user management, pipeline customization, and alerting at no additional charge. Where Graylog suffers is in reporting and analytics. While it does have built-in charts, it’s commonly paired with Graphite and Grafana for visualizing metrics. Graylog also has fewer integrations and plugins than the Elastic Stack, making it somewhat harder to set up.
- User-friendly interface
- Offers several advanced features for free
- Requires external log shippers
- Limited integrations with other platforms
- Limited graphing and reporting capabilities
LOGalyze combines log management and network monitoring for a complete view of your infrastructure. In addition to log data, LOGalyze also supports data collection from database queries and SNMP traps. Its web-based UI lets you browse log data, generate reports, configure alerts, and administer the stack. However, LOGalyze is much less popular than other free tools. Its interface is somewhat dated, and although it’s still under active development, updates are infrequent. LOGalyze is also much more difficult to scale, making it more viable as a local logging solution.
- Simple to install and run
- Support for monitoring network devices and databases
- Limited scaling capabilities
- Limited support for log file types
- Dated interface
- Little community activity
- Less extensible than other log management solutions
Logwatch is a lightweight tool for generating reports based on log activity. It analyzes the log files stored on a host machine and creates a report, which it can then save to file or send via email. Logwatch focuses entirely on log analysis and doesn’t actually collect, index, or store log files. However, it’s a fast and easy tool for reviewing a host’s log data.
- Easy to install and configure
- Consolidates its entire analysis into a single daily report
- Runs on each host individually
- No real-time analysis or reporting
- No centralized log management or storage
- Outdated - the last version was released in early 2016
- Requires additional software for scheduling and mailing reports
If your application is on some flavor of Linux, you might be already use syslog, rsyslog or syslog-ng for logging. Syslog implementations write log messages to plain text files and then they’re searchable using Linux tools like grep, which was revolutionary decades ago. This loose standard and lack of structure is the downfall of syslog. Finding information in large plain text files is difficult and so is splitting them up and trying to correlate so many files. Syslog also doesn’t do well with metadata like tagging what sources and servers the logs are coming from. Journald was created to add more structure and move away from plain text logs. It comes with a unix command line tool called journalctl which retrieves data more quickly, filtered by date, PID, UID, service, source and allows new possibilities of live tail of a specified type. However, Journald still relies on integration with existing syslog to route messages and is outdated for the modern software stacks that use many different systems and machines and require a centralized logging system.
- Better than rsyslog or logging individual text files
- Command line interface
- Simple queries are supported but if you have more modern stack across many nodes, machines, environments, cloud logging solutions are far superior. You can also forward journald logs.
7. LogDNA (Lowest TCO in the Industry)
While free tools can’t be beat on price, they lack many of the features organizations expect from a complete log management solution, and often end up costing more over the long run. With LogDNA, you get an enterprise-ready log management solution that scales with you, never has data caps, and provides the exact features necessary for your organization to streamline processes. LogDNA provides:
- Modern, web-based UI
- Automatic parsing for many log formats, with support for custom parsing rules
- Deploys almost instantly with no extra training required
- Auto-scaling to meet your growing logging needs
- Alerting over email, Slack, PagerDuty, and other channels
- Compliance with HIPAA, HITECH, GDPR, US-EU Privacy Shield, and SOC 2
LogDNA is available as a cloud service and as a self-hosted/on-premise solution. With our cloud service, you can start sending and managing your logs in a matter of minutes, no matter how many applications or systems you have. Start with our free plan to take advantage of unlimited logging and live tailing across all of your applications, hosts, and services. When you’re ready to add data retention and new features such as alerting, automatic parsing, custom fields, and single sign-on, simply switch to a paid plan.