carvel-dev / carvel-dev/carvel

Migrate `ytt` playground rendering from Dmitriy's AWS account to Netlify's Functions

Open
#75 7 comments 0 reactions 0 assignees View on GitHub
enhancement priority/important-soon
Dominant language
HTML
Stars
408
Forks
148
PR merge metrics
No merged PRs in 30d

Description

**Describe the problem/challenge you have**
Want to be able to deploy a new version of the ytt playground backend without having to bug Dmitriy about it.

**Describe the solution you'd like**
Since we're hosting our docs at Netlify and Netlify also hosts AWS Lambdas under their "Functions" feature, seems like that would be a great alternative.

**Anything else you would like to add:**
Functions configured here: https://app.netlify.com/sites/carvel/functions (and there are links to docs from there)

---

### Implementation Notes and context

See [this public document](https://hackmd.io/iUmIBKrOTM6TjHjm3L9fzQ?both) for context.

---

## Recommended next things to try:
- **Modify the current function to use a `Handler` and a APIGatewayProxyResponse** This is based purely on a hunch that the netlify docs say that a function [MUST](https://docs.netlify.com/functions/build-with-go/#synchronous-function-format) contain a handler function, but [ours](https://github.com/vmware-tanzu/carvel-ytt/blob/develop/cmd/ytt-lambda-website/main.go#L20) does not exactly. That could be a reason the function is not recognized. May also require work to combine our current 2 binary build into a single binary.
- **Don't use Netlify**. We have a working AWS lambda function code that doesn't work on Netlify. Can we get an AWS or Google account and just use that?
- **Rewrite the ytt playground function in WASM or javascript**. Netlify should be able to recognize a javascript function easier. It is not clear if even as a javascript function if the current 2 binary approach will work.

## The incomplete work in progress:
- Most recent branch uses vendir to dynamically include and build carvel-ytt lambda and copy the 2 binaries to the default `netlify/functions` folder. Doesn't work, netlify does not recognize the binaries and deploy them. https://github.com/vmware-tanzu/carvel/pull/428

- The work for combining the ytt lambda into a single binary is on this branch (unclear if it is working, but it compiles): https://github.com/vmware-tanzu/carvel-ytt/tree/website-no-ytt-binary

## Important details:
- **Golang functions don't get deployed locally**. You must use Netlify deploy previews that are created in a PR to recognize and deploy golang functions that are compatable with netlify. The netlify-cli (`netlify dev` command) will not recognize golang functions ever (using netlify-cli 10.1.0).
- **The command to build locally is `netlify build --context development`**. This will run the `[build]` step in the `netlify.toml` file. Note: the PR deploy preview actually runs the `[context.deploy-preview]` step.
- **A working example of a golang function getting deployed to netlify is at..** [This commit](https://github.com/vmware-tanzu/carvel/commit/d357dd8a080abb3328a2560fd27fc638c159d395) shows that with a simple function. Note: you will have to recreate a PR with this commit to see the deploy preview since the history in netlify doesn't stick around. Visit it at `/.netlify/functions/playground`.
- **Golang precompiled binaries don't work**. It doesn't appear that placing golang function binaries directly in the `neltify/functions` folder works. It appears that golang functions must be built by netlify.
- This is based on the data that a deploy preview [PR](https://github.com/vmware-tanzu/carvel/commit/d357dd8a080abb3328a2560fd27fc638c159d395) where netlify built the binaries worked, but another [PR](https://github.com/vmware-tanzu/carvel/pull/428/commits/0dfa3bb3c90bb48c664e4086ee355263392f3017) that commited the same binary directly did not recognize any functions.
- **Zip file golang functions don't work**. Placing the function binaies in a zip file and in the netlify/functions folder misrecognizes them as a Node function. It does see the function, and deploys it, however when you visit it it yeilds an error.
- **Netlify docs can't be trusted**. Much of the documentation for go lambda functions for netlify is slightly incorrect or just wrong.
- **Once a solution is working think about this**. There are some copied SCSS files in both the Carvel-ytt repo and the Carvel repo for styling the playground. It would be cool if the solution could get rid of the duplication (not required).

---
Netlify Functions:
- has three deployment modes:
- _auto-detect_ — Netlify scans the "Functions Folder" for source files and builds automatically
- _custom-build_ — configure a command (in `netlify.toml`) to build the functions from source
- _pre-built binary_ — commit the pre-built binary (i.e. zip file) to the site repo.
- by default the "Functions Folder" is `./netlify/functions`
- functions map from `.///{main.go,.go,main.zip}` to `https://carvel.dev/.netlify/functions/`

DNS config:
- today, the playground templating is sent to `get-ytt.io/template`
- when configured through Netlify, this will become `carvel.dev/.netlify/functions/template`

### **Possible logistic approaches**
- Approach 1: Build the ytt playground lambda from source (meaning that we're not committing binaries to a git repo)
- Approach 2: Download lambda function during build (publish lambda as release artifact, and then fetch and deploy it during the website build)

#### **The work that was done for Approach 1: Build the ytt playground lambda from source**
- [x] Add `vmware-tanzu/carvel-ytt` to `vmware-tanzu/carvel` via vendir
- [x] Add build script to `netlify.toml` that builds `ytt` in addition to the site build command (i.e. `hugo --gc --minify`)
- [x] This script builds the lambda and copies it to `netlify/functions`
- [x] (doesn't work) Verify locally that it is recognized by running `netlify build --context development`
- [ ] (doesn't work as is) Deploy the function
- [ ] Ensure that our Netlify build image is using Go 1.17
- [ ] Change the endpoint the playground currently uses (get-ytt.io/template) to the function endpoint probably (https://carvel.dev/netlify/functions/template)
- [ ] Use the CSS from git submodule ytt repo (playground) (see [comment, below](https://github.com/vmware-tanzu/carvel/issues/75#issuecomment-808490883)) instead of copying it.
- [ ] ...

Signifigant progress was made to do the above on [PR ](https://github.com/vmware-tanzu/carvel/pull/428). HOWEVER, Netlify does not recognize the lambda binaries for a unknown reason, so this work was set down.

Contributor guide

Open the contributing guide

Research direction

Start with cmd/ytt-lambda-website/main.go, netlify.toml, PR 428, and the working Netlify Go function commit referenced in the issue; validate changes through a Netlify deploy preview rather than netlify dev. Done means the playground function is deployed, the endpoint is changed from get-ytt.io/template to the carvel.dev function endpoint, and the build uses a supported Go version.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
backend, cloud, devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.