LogDNA is excited to be a Silver Member of Cloud Foundry this year. If you're at the North America Summit 2019 this week, drop by our booth B7 we would love to meet you.

Cloud Foundry Application Runtime is an open source platform as a service (PaaS) for running applications and services. Frequently called simply "Cloud Foundry," the Cloud Foundry Application Runtime (CFAR) is one of many interoperable projects within the family. For the purposes of this post, "Cloud Foundry" refers to the Application Runtime.

It allows organizations to automatically deploy, manage, and scale networked workloads on any infrastructure, including cloud IaaS providers and on-premise. Its design is similar to that of the serverless architecture, in that developers can deploy code or binaries directly to the platform. Cloud Foundry handles packaging, deploying, networking, scaling, and load balancing these workloads across the infrastructure, leaving developers to focus on building the workloads.

In this post, we'll show you how to use our Cloud Foundry integration to effortlessly ship logs from your Cloud Foundry deployments to LogDNA. We assume that you already have a working deployment with actively running apps, as well as a LogDNA account. If you're new, you can learn more at the documentation site.

How Logging Works in Cloud Foundry

The Cloud Foundry logging system is called Loggregator. Loggregator collects logs and metrics from apps, aggregates them, and routes them to any clients or services that request them. Loggregator automatically collects logs from any and all apps, as long as those apps print their logs to standard output.

Loggregator has the following architecture:

1. Apps generate logs which are collected by Source.

2. Sources forward logs to Metrons, which are agents running on the same node as the Sources they log.

3. Metrons forward logs to Dopplers, which are servers that collect and store logs in temporary buffers.

4. A Traffic Controller handles requests from external logging clients and services. The Traffic Controller exposes a Firehose, which is a WebSocket endpoint capable of streaming logs and metrics to remote services.

The Firehose is the most important aspect of this architecture, since it creates an access point for applications and services (called nozzles) to access the log stream. You can learn more about nozzles from the nozzle tutorial. In this next section, we'll explain how LogDNA uses the Firehose to ingest logs.

Integrating LogDNA with Cloud Foundry

LogDNA provides two methods of integration:

1. Creating a user-provided service instance and binding it to an individual app.

2. Using the LogDNA nozzle to log your entire deployment.

The next steps assume that you already have the Cloud Foundry CLI installed and registered with your Cloud Foundry organization.

1. Creating a User-Provided Service Instance

A user-provided service instance allows you to stream logs from a Cloud Foundry app to LogDNA via syslog. This is the easier method to set up, but it will only log apps that you bind the service instance to.
Start by logging into your LogDNA account and navigating to Integrations. Select the Cloud Foundry integration and select Provision a Syslog Port. You will receive a new LogDNA syslog endpoint, which you can use to connect your service:

The first command creates a new service named my-logs that points to the LogDNA endpoint. The second command binds my-logs to your app (make sure you replace <app name=""> with the name of your app). The final command restarts your app with the new service bound. Now, any logs that your app writes to standard output will appear in LogDNA.</app>

2. Using the LogDNA Nozzle (Recommended)

The LogDNA nozzle streams events from your entire deployment to LogDNA. The nozzle runs as an app within Cloud Foundry, making it easy to deploy and manage. In minutes, all of your app logs will begin appearing in LogDNA.

Start by downloading the latest release of the LogDNA nozzle:

$ git clone https://github.com/logdna/logdna-cloudfoundry.git$ cd logdna-cloudfoundry

Before connecting any nozzle to the Firehose, you must first authenticate the nozzle as a User Account and Authentication (UAA) client. Access to the Firehose is controlled by the doppler.firehose scope. To learn more about nozzle authentication and permissions, read the nozzle documentation.

Once your nozzle is authorized, you can push it. We'll name ours firehose-to-logdna so that it's easy to identify in the app list. The --no-start parameter specifies that we don't want to run the app after pushing it:

$ cf push firehose-to-logdna --no-start

Next, we need to set several environment variables. These provide the app with the data needed to authenticate not just to with Firehose, but with LogDNA. They include:

VariableDescriptionAPI_ENDPOINTYour Cloud Foundry system domain.DOPPLER_ENDPOINTYour Cloud Foundry system domain.INGESTION_KEYYour LogDNA ingestion key (available here).FIREHOSE_CLIENT_IDThe ID of the LogDNA UAA client.FIREHOSE_CLIENT_SECRETThe secret associated with the UAA client.SYSLOG_ENDPOINTAn optional endpoint for a syslog server.

You can set each of these variables using the Cloud Foundry CLI:

$ cf set-env firehose-to-logdna API_ENDPOINT [your API_ENDPOINT]
$ cf set-env firehose-to-logdna DOPPLER_ENDPOINT [your DOPPLER_ENDPOINT]
$ cf set-env firehose-to-logdna INGESTION_KEY [your INGESTION_KEY]
$ cf set-env firehose-to-logdna FIREHOSE_CLIENT_ID [your FIREHOSE_CLIENT_ID]
$ cf set-env firehose-to-logdna FIREHOSE_CLIENT_SECRET [FIREHOSE_CLIENT_SECRET]
$ cf set-env firehose-to-logdna SYSLOG_ENDPOINT [your optional SYSLOG_ENDPOINT]

Finally, push the nozzle to your deployment:

$ cf push firehose-to-logdna

In a few moments, logs from your Cloud Foundry applications will begin appearing in LogDNA.

Conclusion

With LogDNA, you can quickly and easily log your entire Cloud Foundry deployment in a matter of minutes. To get started, log into your LogDNA account and navigate to the integrations page. If you wish to learn more about our Cloud Foundry Firehose nozzle, visit the project's GitHub page. And if you have any questions, please contact us.