GoogleContainerTools / GoogleContainerTools/skaffold
helm templates with lookup function not working with skaffold v2
- Dominant language
- Go
- Stars
- 15.9k
- Forks
- 1.7k
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 10
Description
### Context
In our Helm templates, we heavily rely on the `lookup` function to apply data to our application templates.
One use case is that we use the lookup function to fetch the ClusterIP from our internal namespaced service to add hostAliases to our deployments and statefulsets to route http data internally instead of externally.
### Expected behavior
In skaffold v1, when skaffold would directly execute `helm upgrade --install`, this lookup method would work as expected. With skaffold v2, we expect our use case to be still available. So when executing `skaffold build` and `skaffold deploy`, skaffold should not return an error about the lookup function or data is unavailable.
Maybe we are missing something, and is the option still available. However, I could not find any information in the documentation to use the old behaviour.
### Actual behavior
Because of the `helm template` action in the manifest step, the `lookup` function is unavailable (restriction of `helm template`). Then executing our deployment strategy using `skaffold build` and `skaffold deploy` we now get the error that the `template` could not be executed properly and fails on the `lookup` in the templates.
The error we get is the following
```sh
Error: template: example/templates/application.yaml:31:45: executing "example/templates/application.yaml" at <$internalLoadBalancerService.spec.clusterIP>: nil pointer evaluating interface {}.clusterIP
Use --debug flag to render out invalid YAML
)
```
### Information
- Skaffold version: v2.13.1
- Operating system: Mixed. macOS for development, Linux for CI/CD process
- Installed via: Google Cloud SDK (GitHub releases)
- Contents of skaffold.yaml:
```yaml
# skaffold.yaml
apiVersion: skaffold/v4beta11
kind: Config
metadata:
name: example
build:
local:
push: true
useDockerCLI: false
useBuildkit: true
concurrency: 0
tagPolicy:
inputDigest: { }
artifacts:
- image: registry.example.com/example.com
manifests:
helm:
releases:
- name: example
chartPath: .helm
wait: true
valuesFiles:
- .helm/values.yaml
setValueTemplates:
example.image: registry.example.com/example.com
flags:
install:
- --timeout=120m
- --render-subchart-notes
upgrade:
- --timeout=5m
- --render-subchart-notes
deploy:
helm:
releases:
- name: example
chartPath: .helm
wait: true
valuesFiles:
- .helm/values.yaml
setValueTemplates:
example.image: registry.example.com/example.com
flags:
install:
- --timeout=120m
- --render-subchart-notes
upgrade:
- --timeout=5m
- --render-subchart-notes
profiles:
- name: dev
activation:
- command: dev
patches:
- op: add
path: /manifests/helm/releases/0/valuesFiles
value:
- .helm/values.dev.yaml
- op: add
path: /deploy/helm/releases/0/valuesFiles
value:
- .helm/values.dev.yaml
- op: replace
path: /build/artifacts/0/image
value: registry.example.com/dev/example.com
- op: replace
path: /manifests/helm/releases/0/setValueTemplates/example.image
value: registry.example.com/dev/example.com
- op: replace
path: /deploy/helm/releases/0/setValueTemplates/example.image
value: registry.example.com/dev/example.com
```
### Steps to reproduce the behavior
#### At first, create a simple helm chart using the files below
```yaml
# values.yaml
global:
internalLoadBalancerService: private-http-proxy
```
```yaml
# Chart.yaml
apiVersion: v2
name: example-chart
version: 1.0.0
```
```yaml
# templates/deployment.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
meta.helm.sh/release-name: example-chart
labels:
app.kubernetes.io/instance: example-chart
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: example-app
helm.sh/chart: example-1.0.0
name: example-app
spec:
replicas: 0
selector:
matchLabels:
app.kubernetes.io/instance: example-chart
app.kubernetes.io/name: example-app
template:
metadata:
labels:
app.kubernetes.io/instance: example-chart
app.kubernetes.io/name: example-app
spec:
containers:
- env:
- name: CONFIG_CHECKSUM
value: 1fed86344f801e7f1a1e5490202fade968f1fd54
envFrom:
- configMapRef:
name: example-application
- secretRef:
name: example-application
image: registry.example.com/example.com:e57801aa5fb3576ec30407301963bad85be187cca670dbbfadcf4982962e33e4
imagePullPolicy: IfNotPresent
name: example-app
ports:
- containerPort: 54663
name: http
protocol: TCP
{{- if .Values.global.internalLoadBalancerService }}
{{- $internalLoadBalancerService := (lookup "v1" "Service" .Release.Namespace .Values.global.internalLoadBalancerService) }}
hostAliases:
- ip: {{ $internalLoadBalancerService.spec.clusterIP }}
hostnames:
{{- $hosts := list "example.com" }}
{{- range $host := $hosts }}
- {{ $host | quote }}
{{- end}}
{{- end }}
```
#### Apply the service below using `kubectl apply -f service.yaml -n [NAMESPACE]`
```yaml
# service.yaml
---
apiVersion: v1
kind: Service
metadata:
name: private-http-proxy
spec:
clusterIP: 172.18.0.254
clusterIPs:
- 172.18.0.254
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- name: http
port: 80
protocol: TCP
targetPort: 980
- name: https
port: 443
protocol: TCP
targetPort: 9443
type: ClusterIP
```
#### Use helm to deploy the chart created earlier
`helm upgrade --install example .helm --debug`
The correct ClusterIP should be set in the applications `hostAliases` settings.
#### Use skaffold to apply the helm chart, using the `skaffold.build-artifacts.json` below and the skaffold.yaml provided above.
`skaffold.build-artifacts.json`
```json
{"builds": [{"imageName": "registry.example.com/example.com","tag": "registry.example.com/example.com:e57801aa5fb3576ec30407301963bad85be187cca670dbbfadcf4982962e33e4"}]}
```
`skaffold deploy --build-artifacts skaffold.build-artifacts.json -n [NAMESPACE]`
See that you get the error
```
std out err: %!(EXTRA *errors.errorString=Error: template: example-chart/templates/deployment.yaml:44:45: executing "example-chart/templates/deployment.yaml" at <$internalLoadBalancerService.spec.clusterIP>: nil pointer evaluating interface {}.clusterIP
Use --debug flag to render out invalid YAML
)
```
When using skaffold v1.39.18, and the `skaffold-v1.yaml` provided below, the `skaffold deploy` will succeed and the correct ClusterIP should be set in the applications `hostAliases` settings.
```yaml
# skaffold-v1.yaml
apiVersion: skaffold/v2beta29
kind: Config
metadata:
name: example
build:
local:
push: true
useDockerCLI: false
useBuildkit: true
concurrency: 0
tagPolicy:
inputDigest: { }
artifacts:
- image: registry.example.com/example.com
deploy:
helm:
releases:
- name: example
chartPath: .helm
wait: true
valuesFiles:
- .helm/values.yaml
artifactOverrides:
example.image: registry.example.com/example.com
flags:
install:
- --timeout=120m
- --render-subchart-notes
upgrade:
- --timeout=5m
- --render-subchart-notes
profiles:
- name: dev
activation:
- command: dev
patches:
- op: add
path: /manifests/helm/releases/0/valuesFiles
value:
- .helm/values.dev.yaml
- op: add
path: /deploy/helm/releases/0/valuesFiles
value:
- .helm/values.dev.yaml
- op: replace
path: /build/artifacts/0/image
value: registry.example.com/dev/example.com
- op: replace
path: /manifests/helm/releases/0/artifactOverrides/example.image
value: registry.example.com/dev/example.com
- op: replace
path: /deploy/helm/releases/0/artifactOverrides/example.image
value: registry.example.com/dev/example.com
```
`skaffold deploy --build-artifacts skaffold.build-artifacts.json -f skaffold-v1.yaml -n [NAMESPACE]`
Contributor guide
Assessment
This issue has not been assessed yet.