Azure / Azure/AgentBaker

Explicit configuration interface

Open
#2,815 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
159
Forks
277
Avg merge
1d 16h
Merged PRs (30d)
178

Description

Moving some private discussion I've had with @alexeldeib into a public ticket to increase bus factor.

**Is your feature request related to a problem?/Why is this needed**
Our team uses a custom AKS image which has a few dependencies which are currently provided by AgentBaker's customnodedata:

We currently depend on the following files to be provided via [nodecustomdata](https://github.com/Azure/AgentBaker/blob/master/parts/linux/cloud-init/nodecustomdata.yml):
/etc/default/kubelet
/var/lib/kubelet/bootstrap-kubeconfig 
/etc/kubernetes/certs/ca.crt

Within those files we depend on:
/etc/default/kubelet:
- node labels
- kubelet command line parameters

/var/libe/kubelet/bootstrap-kubeconfg:
- cluster-server:
```
name: localcluster
cluster:
certificate-authority: /etc/kubernetes/certs/ca.crt
server: https://dev-azure-westus-xxx-cc782afe.hcp.westus.azmk8s
```
- token
```
- name: kubelet-bootstrap
user:
token: "sbiizf.avcjfgfj5h3oni"
```

- ca.crt - we depend on the whole file for the bootstrap kube-config.

We explicitly prevent the CustomScriptExtension from running by touching the `/opt/azure/containers/provision.complete` file which CSE checks prior to running. We don't want CSE to run because it does node level configuration which conflicts with our own.

**Describe the solution you'd like in detail**
Ideally, the interface in AgentBaker we come up with:
- Allows us to test that these files are present prior to CSE running 
- Allows us to confirm that CSE does not run if the following file is already set: /opt/azure/containers/provision.complete 

There are a few options which stand out to me:

1. Write a test just confirming that those files are populated in nodedata
2. Write a new file interface (bootstrap.cfg) which either:
a. Has those files included in it
b. Has the fields we need only included
3. Some mechanism to fetch these dynamically from within the node, so we can completely remove the dependency on data provided by AgentBaker (though we’d shift the dependency to requiring that the fields are accessible somehow)

After discussions with @alexeldeib - there's a preference to *not* have the data provided via customdata.

**Describe alternatives you've considered**

**Additional context**
We've had a number of incidents due to us not having a clear contract around node configuration, so hoping to work with y'all to get one defined. Thanks in advance!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.