Proposal: buildctl daemon command

BuildKit uses daemon model for many benefits like parallel job processing, easier extendability, precise privileged execution control, remote control etc. The downside of it is that it makes the setup harder for new users and people who want to run it occasionally.

The proposal is to hide the presence of daemon unless some more advanced use cases are needed.

Automatic bootstrapping

If the user runs a buildctl command that requests API access and there is no response.

• If connecting address has been changed from the default, a standard dialing error is shown.

• If the connecting address is default buildctl tries to start the daemon automatically and shows a log message about that.

buildctl first finds if there is a buildd binary(or other dependencies for macOS) and gives up if it can't locate it.

Then it tries to check if it has enough privileges to start the daemon. In Linux, this means root with needed capabilities enabled. On macOS, the daemon would be launched in a vm without requiring extra privileges.

• If there are enough permissions daemon will start on background and user command executes. A log message will be shown about it. Otherwise, the user shouldn't even notice it except for a small delay. The daemon continues running after the user command has finished. If the daemon doesn't come up(or no workers can be started), then an error will be shown.

• If there are NOT enough permissions, buildctl will show a helpful message explaining that daemon needs to be started and the command that user needs to run to do it.

The command would look something like:

sudo buildctl daemon start --root=/home/myuser/buildkitstate --allow-user=myuser --entitlements=security.confined

This command will:

• Check if it has enough permission to run daemon. Even with sudo, this check may fail if the caps are bound.

• Check that user calling sudo has write permission to root dir. This is to avoid buildkit taking disk space from partitions the user doesn't have access to.

• next, it uses the sudo privileges to set a helper binary buildkitd-launcher that comes with buildkit to suid and makes sure root owns it.

• It writes a configuration file that lists the allowed user values and entitlements they are allowed to use. The default entitlements that don't have a security list can be added automatically. Makes sure the file is owned by root and only modifiable by root.

• Executes buildkitd-launcher that starts the daemon. Only unix socket requests are allowed against this daemon. Every connection checks the dialing user is in the configuration file and make sure the build requests don't use any entitlements that have not been granted.

• Next time after daemon needs to be started; the user doesn't need to type the start command again as it can directly execute the suid binary now.

An unprivileged user using the buildkit daemon this way should be considered as secure as containers. There shouldn't be any way for the user to issue a command against buildkit daemon that has access to anything outside of the state directory they already have access to.

Extra commands

When the daemon is running user can use some helper command to control it.

buildctl daemon state - shows info about current config(maybe logs as well)

buildctl daemon stop - stop the server. restarted automatically by buildctl when needed. There could also be a --timeout for the daemon start command for automatically shutting down server on inactivity.

buildctl daemon reset - start a fresh daemon with empty state dir

Start with the buildctl command entry point and map the proposed daemon start, state, stop, and reset flows. Review the automatic bootstrapping, privilege checks, Unix-socket access, configuration ownership, and macOS virtualization requirements described here. Done would require an agreed design and implementation covering these cross-platform and security-sensitive behaviors.