Heroku Integration / Heroku Button

Many Heroku customers have asked how they could use Vector to ship their Heroku logs. This is especially true for current Timber Hosted customers as many of them are on Heroku. In order to unblock them, we need to provide a simple solution.

Background

This is not a new problem. We've attempted to solve this problem a couple of times in the past but the goals and context were different:

Attempt 1

Our first attempt at solving this prioritized ease of setup, and attempted to move towards a hosted Vector service. The goal was to provide a blanket solution that would reliably transition all Heroku users at various engineering levels. Heroku customers are generally not interested in hosting infrastructure, configuring networks, and setting up HTTP load balancers. They just want to configure a Heroku log drain and be on their way. Unfortunately, we didn't have a way to achieve this for them, but we did have plans to offer a hosted Vector service at some point in the future. So we decided to take the easiest first step towards a hosted Vector service by deploying Vector in an ECS container for each customer that needed it. From the customer's perspective, this would feel like a hosted Vector service. They'd configure a Heroku log drain and be on their way. As we increased the number of customers on this platform we would then take the next steps towards automation, scaling, and formalizing this as a true Vector hosted service. This was never implemented because of attempt 2...

Attempt 2

The Alloy team needed something quick and easy to unblock customers, and they attempted to create a Lambda function that customers could deploy in their AWS accounts as a serverless solution. Many of the customers were attempting to use AWS CloudWatch, so having them deploy an AWS Lambda function did not feel intrusive. Because this was done quickly, many of the users waiting on attempt 1 decided to use this. This worked ok for getting logs from Heroku to CloudWatch, but it fell short when the customer wanted to do anything with their data, like parse it. It doesn't make sense to invest any more time or money into this function when this is Vector's domain. And that brings me to the solution proposed in this issue...

Proposal

I propose that we add Heroku to the list of platforms in our installation section. The approach would leverage Heroku's button feature, where a user could click the Heroku button and deploy Vector as a dyno in their own Heroku account.

Requirements

• Create a Heroku button that deploys Vector into their Heroku account.
• It should be preconfigured with the heroku_logplex source.
• It should be preconfigured with parsers for the various Heroku logs (this will need fleshing out).
• It should be preconfigured with the aws_cloudwatch_logs sink, or somehow easily configurable for this sink.
• It should be required that users provide AWS credentials if they are using AWS sinks. This is probably best achieved with environment variables since this is Heroku's means of configuring services.
• The HTTP URL should be readily available so that users can configure log drains for their other Heroku services to point to the Vector service.

Open Questions

1. We need to audit and list all of the various log formats that Heroku includes so that we can include parsers. Most of them are in logfmt, you can see a list of error logs here.
2. Should we also address other common formats, such as Rails. Or should we require users to simplify configure their app to log JSON. I vote for the latter.
3. How can we make it easy for users to configure Vector upon deployment? #1966 proposes a simple pipline syntax that is perfect for scenarios like this. This syntax could be provided as an environment variable.

Start with the installation section and inspect the existing heroku_logplex source and aws_cloudwatch_logs sink entry points. Review Heroku's button and logging requirements, then consider the pipeline syntax proposed in #1966. Done means a deployable Heroku button, documented configuration and credentials, supported log parsing, and a usable HTTP log-drain URL.