The use cases and requirements of a logging platform in an organization varies between teams and job functions. The problem isn’t in collecting log data (we are a logging company after all), but in deciding how to manage these logs for each team. For example, our backend developers need detailed, short-lived logs in order to build and test new features; while our infrastructure team needs lengthy retention periods for auditing and compliance. We wanted to share an interesting way that some of our customers use LogDNA’s organization feature so that it is intuitive and seamless for different teams.
In this guide, we’ll provide tips and guidance on how to structure LogDNA for multiple deployment environments and teams. We’ll show you how to separate your development, staging, and production logs using organizations; how this can help your teams work more efficiently; and how this can help you save money.
Creating new Organizations in LogDNA
Organizations are like workspaces in the LogDNA app. They have their own separate log feed, ingestion keys, views, charts, permissions, and other resources. User accounts can be assigned to multiple organizations, and switching between organizations in the LogDNA app is as easy as clicking a button.
This makes organizations perfect for logging multiple deployment environments: developers can manage their logs in a dedicated development organization, while IT and ops can manage their logs in a dedicated production organization. If a user requires access to another organization—for instance, if a developer needs access to production logs to troubleshoot a problem—you can use role-based access controls to control their level of access.
Using organizations can also lower your costs for logging. Since each organization has a separate LogDNA plan, you can pick and choose the plan that’s best suited for the team. For example, production logs typically need longer retention periods, where development logs can be on a lower plan. Using organizations lets you choose less expensive plans for those environments that don’t need long retention or a large number of users, saving you money.
Now, let’s look at how you can structure organizations within LogDNA to match the engineering workflow in your company.
The development environment is where your developers and any stakeholders in the development process validate code changes before integrating them into the product. Logging in this environment is less about retention and more about access to real-time log data and Live Tail. If developers do need to retain logs, it’s usually for no longer than a few days.
Since these logs aren’t long-lived, you can choose a plan with lower retention. Small teams and startups might even be able to get by with a free plan, but keep in mind that this only supports live tail for a single user. All of our paid plans offer unlimited sources, unlimited custom views, alerts, archiving, indexing, and custom fields, so you can choose a lower tier plan without losing access to key features.
Lastly, the development environment is where you can start testing logging customizations such as custom parsing rules, alerts, and charts. This lets you fine-tune your monitoring strategy before deploying to production. Once the customizations have been approved, they can be easily migrated to your other environments.
Suggestions for setting up logging for developers:
- Choose a lower-cost plan with less retention.
- Limit organization members to developers and immediate stakeholders.
- Use as a test bed for views, alerts, custom parsing rules, and other customizations planned for production.
The goal of a staging environment is to mirror production as closely as possible in order to prepare the application for deployment, monitor its behavior, and catch unexpected bugs. Staging logs are kept longer than development logs to give engineers enough historical data to troubleshoot and trace problems. In addition, logs in staging are useful to several teams including development, operations, and QA. You may want to choose a plan with a moderate retention period (7–14 days) and support for all your users. If you use real-world data in your staging environment and are concerned about potential leaks, you can use role-based access controls (RBAC) to restrict user access to specific logs.
In addition to longer retention, staging environments are more likely to use monitoring tools like alerts and charts to monitor and analyze application behavior. Since staging mirrors production, this is the best opportunity to develop and test your monitoring configuration before moving it into your production organization.
Suggestions for setting up logging in staging:
- Choose a plan with moderate retention and a large number of user accounts.
- Use staging to test views, alerts, custom parsing rules, charts, and other customizations before deploying to production.
- Use RBAC to hide sensitive logs or views from users or teams.
Three of the most important aspects of logging in production are access, compliance, and retention. Since production logs can contain sensitive customer and operational data, limiting access to specific IT and ops team members can prevent accidental leaks. Complying with regulations such as HIPAA and PCI DSS are important. Required retention periods for production data are higher than in development and staging are typically 14-30 days. You may also want to archive your logs and use an external querying tool like Google BigQuery or Presto to search your archives.
If you tested your views, charts, alerts, and other customizations in staging, you can recreate them in your production organization before deploying your application. This ensures that you have everything in place to successfully monitor your application.
Suggestions for setting up logging in production:
- Choose a plan with high retention and a low number of user accounts.
- Restrict access to specific users and teams.
- Enable log archiving for longer retention and complying with regulations.
We hope this article gives you some ideas on how you create multiple organizations to structure logging for your teams. Check out our plans to see which ones best fit your use case. If you have specific requirements for users, retention, or log volume, contact us and we’ll help you find the best solution.