Dstack-TEE / Dstack-TEE/dstack

Feature: declarative host-service lifecycle for system-container workloads

Open
#1,033 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
544
Forks
96
Avg merge
23h 40m
Merged PRs (30d)
126

Description

## Summary

We run Incus/LXC system containers inside a DStack CVM. For our storage, cgroup, and reboot semantics, the Incus daemon runs as a PID 1-owned host systemd service rather than as an ordinary application container.

The integration works, but the application currently owns an imperative host-service installer and its complete reboot, readiness, update, and rollback protocol. We would like DStack to provide a constrained, declarative lifecycle contract for this class of host service.

## Existing DStack mechanisms

DStack already provides host execution mechanisms; this request does not claim otherwise.

At current `next` commit `312dc605ebaaf93372534ffb135ea011533d28e9`:

- [`app-compose.service`](https://github.com/Dstack-TEE/dstack/blob/312dc605ebaaf93372534ffb135ea011533d28e9/os/common/rootfs/app-compose.service#L1-L17) is a host systemd oneshot that runs `app-compose.sh` and remains active after the script exits.
- [`app-compose.sh`](https://github.com/Dstack-TEE/dstack/blob/312dc605ebaaf93372534ffb135ea011533d28e9/os/common/rootfs/app-compose.sh#L19-L125) supports a pre-launch script, Docker or nerdctl Compose, and a bash runner.
- The vsock [`HostApi`](https://github.com/Dstack-TEE/dstack/blob/312dc605ebaaf93372534ffb135ea011533d28e9/dstack/host-api/proto/host_api.proto#L30-L34) exposes `Info`, `Notify`, and `GetSealingKey`.

These are useful imperative building blocks, but the current public contract does not describe a typed host-service declaration with persistent/boot-scoped resources, service ownership, readiness, or rollback semantics.

## Why an ordinary Compose service is insufficient for this workload

For our required deployment model, the Incus daemon must:

- own a delegated cgroup subtree for nested system containers;
- use host-native ZFS state under `/dstack/persistent`;
- remain a PID 1-owned service independently of the finite installer; and
- receive explicit persistent and boot-scoped binds into a content-addressed userspace root.

Our current integration therefore uses a finite privileged Compose job with host PID/root access. One centralized `nsenter --target 1` boundary verifies a pinned payload, materializes a boot-scoped root, installs runtime systemd units, binds persistent state and boot-scoped resources, starts the daemon, and checks readiness. A mismatch or partial generation fails closed.

This is workable, but every consumer of this pattern must implement and secure its own installer, convergence, reboot recovery, update, and rollback logic.

## Requested capability

Please consider a constrained host-service declaration that can express:

- a content-addressed and authenticated payload;
- explicit persistent state and boot-scoped runtime mounts;
- bounded systemd unit properties, capabilities, cgroup delegation, and ordering;
- readiness and health conditions; and
- atomic update, reboot recovery, and rollback behavior.

The declaration should be authenticated and capability-scoped. We are not requesting unrestricted host command execution, and the implementation need not extend `HostApi`; a parallel mechanism would be equally suitable.

## Namespace-safe host resources

One concrete edge case is resolver projection. On the released image, `/etc/resolv.conf` was an absolute symlink chain ending under `/run/systemd/resolve`. Reading it through a container's `/host` root bind resolves the absolute target in the container mount namespace and misses the host file. Current mkosi source similarly creates [`/etc/resolv.conf -> /run/systemd/resolve/stub-resolv.conf`](https://github.com/Dstack-TEE/dstack/blob/312dc605ebaaf93372534ffb135ea011533d28e9/os/mkosi/mkosi.skeleton/usr/lib/tmpfiles.d/dstack-image.conf#L10-L12).

Our installer handles this inside the PID 1 mount namespace. A declarative contract should instead document or provide a namespace-safe projection for host resources consumed by the declared service. We are not requiring `resolv.conf` to be a regular file.

## Suggested acceptance criteria

- A declared service is restored to ready state after an unchanged CVM reboot.
- Persistent state survives reboot and application update, while boot-scoped resources are reconstructed.
- A failed update leaves or restores the last ready generation.
- The application does not need a host-root bind or `nsenter` to install and reconcile the declared service.

## Scope

This is a feature request, not a report that DStack Compose or `HostApi` is broken. Incus networking/kernel compatibility is tracked separately.

Contributor guide

Open the contributing guide

Research direction

Read os/common/rootfs/app-compose.service, os/common/rootfs/app-compose.sh, dstack/host-api/proto/host_api.proto, and os/mkosi/mkosi.skeleton/usr/lib/tmpfiles.d/dstack-image.conf to understand the existing host mechanisms and namespace behavior. Define and review the constrained declarative contract against the listed acceptance criteria: reboot restoration, persistent versus boot-scoped resources, atomic rollback, and no application-managed host-root installation.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, docker-compose, linux, rust
Domain
infrastructure, operating-systems, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.