Cloud Foundry Application Runtime is an open source platform as a service (PaaS) for running applications and services. Frequently called simply “Cloud Foundry,” the Cloud Foundry Application Runtime (CFAR) is one of many interoperable projects within the Cloud Foundry family. For the purposes of this post, “Cloud Foundry” refers to the Application Runtime.
It allows organizations to automatically deploy, manage, and scale networked workloads on any infrastructure, including cloud IaaS providers and on-premise. Its design is similar to that of the serverless architecture, in that developers can deploy code or binaries directly to the platform. Cloud Foundry handles packaging, deploying, networking, scaling, and load balancing these workloads across the infrastructure, leaving developers to focus on building the workloads.
In this post, we’ll show you how to use our Cloud Foundry integration to effortlessly ship logs from your Cloud Foundry deployments to LogDNA. We assume that you already have a working Cloud Foundry deployment with actively running apps, as well as a LogDNA account. If you’re new to Cloud Foundry, you can learn more at the Cloud Foundry documentation site.
How Logging Works in Cloud Foundry
The Cloud Foundry logging system is called Loggregator. Loggregator collects logs and metrics from apps, aggregates them, and routes them to any clients or services that request them. Loggregator automatically collects logs from any and all apps, as long as those apps print their logs to standard output.
Loggregator has the following architecture:
- Apps generate logs which are collected by Sources.
- Sources forward logs to Metrons, which are agents running on the same node as the Sources they log.
- Metrons forward logs to Dopplers, which are servers that collect and store logs in temporary buffers.
- A Traffic Controller handles requests from external logging clients and services. The Traffic Controller exposes a Firehose, which is a WebSocket endpoint capable of streaming logs and metrics to remote services.
The Firehose is the most important aspect of this architecture, since it creates an access point for applications and services (called nozzles) to access the log stream. You can learn more about nozzles from the Cloud Foundry nozzle tutorial. In this next section, we’ll explain how LogDNA uses the Firehose to ingest logs.
Integrating LogDNA with Cloud Foundry
LogDNA provides two methods of integrating with Cloud Foundry:
- Creating a user-provided service instance and binding it to an individual app.
- Using the LogDNA nozzle to log your entire Cloud Foundry deployment.
The next steps assume that you already have the Cloud Foundry CLI installed and registered with your Cloud Foundry organization.
Creating a User-Provided Service Instance
A user-provided service instance allows you to stream logs from a Cloud Foundry app to LogDNA via
Start by logging into your LogDNA account and navigating to Integrations. Select the Cloud Foundry integration and select Provision a Syslog Port. You will receive a new LogDNA
The first command creates a new service named my-logs that points to the LogDNA endpoint. The second command binds my-logs to your Cloud Foundry app (make sure you replace <Cloud Foundry app name> with the name of your app). The final command restarts your app with the new service bound. Now, any logs that your app writes to standard output will appear in LogDNA.
Using the LogDNA Nozzle
The LogDNA nozzle streams events from your entire Cloud Foundry deployment to LogDNA. The nozzle runs as an app within Cloud Foundry, making it easy to deploy and manage. In minutes, all of your Cloud Foundry app logs will begin appearing in LogDNA.
Start by downloading the latest release of the LogDNA nozzle:
$ cd logdna-cloudfoundry
Before connecting any nozzle to the Firehose, you must first authenticate the nozzle as a User Account and Authentication (UAA) client. Access to the Firehose is controlled by the doppler.firehose scope. To learn more about nozzle authentication and permissions, read the Cloud Foundry nozzle documentation.
Once your nozzle is authorized, you can push it to Cloud Foundry. We’ll name ours firehose-to-logdna so that it’s easy to identify in the app list. The –no-start parameter specifies that we don’t want to run the app after pushing it:
$ cf push firehose-to-logdna --no-start
Next, we need to set several environment variables. These provide the app with the data needed to authenticate not just to with Firehose, but with LogDNA. They include:
|API_ENDPOINT||Your Cloud Foundry system domain.|
|DOPPLER_ENDPOINT||Your Cloud Foundry system domain.|
|INGESTION_KEY||Your LogDNA ingestion key (available here).|
|FIREHOSE_CLIENT_ID||The ID of the LogDNA UAA client.|
|FIREHOSE_CLIENT_SECRET||The secret associated with the UAA client.|
|SYSLOG_ENDPOINT||An optional endpoint for a syslog server.|
You can set each of these variables using the Cloud Foundry CLI:
$ cf set-env firehose-to-logdna API_ENDPOINT [your API_ENDPOINT]
$ cf set-env firehose-to-logdna DOPPLER_ENDPOINT [your DOPPLER_ENDPOINT]
$ cf set-env firehose-to-logdna INGESTION_KEY [your INGESTION_KEY]
$ cf set-env firehose-to-logdna FIREHOSE_CLIENT_ID [your FIREHOSE_CLIENT_ID]
$ cf set-env firehose-to-logdna FIREHOSE_CLIENT_SECRET [FIREHOSE_CLIENT_SECRET]
$ cf set-env firehose-to-logdna SYSLOG_ENDPOINT [your optional SYSLOG_ENDPOINT]
Finally, push the nozzle to your Cloud Foundry deployment:
$ cf push firehose-to-logdna
In a few moments, logs from your Cloud Foundry applications will begin appearing in LogDNA.
With LogDNA, you can quickly and easily log your entire Cloud Foundry deployment in a matter of minutes. To get started, log into your LogDNA account and navigate to the Cloud Foundry integrations page. If you wish to learn more about our Cloud Foundry Firehose nozzle, visit the project’s GitHub page. And if you have any questions, please contact us.