Containerized Builds with Tilt

I'm trying to set up a monorepo with multiple microservices for local development using Tilt, and I’ve hit a few roadblocks with live reloading and syncing that I’m hoping you can help with.

Project setup

• Monorepo: Multiple microservices are located in first-level subdirectories, each written in different languages (e.g., Rust, Go, Node.js).
• Tiltfile: There's a single Tiltfile at the root of the monorepo that describes how these microservices work together in the development environment.
• Local Kubernetes Cluster: All microservices, along with their dependencies (e.g., Postgres, Kafka, Redis), run on a local Kubernetes cluster.
• Containerized Builds: Importantly, each microservice is built inside a container, so developers don’t need to install language-specific build tooling on their host machines. Each service comes with its own Dockerfile.dev and Deployment.yaml.

Example Dockerfile for a Rust Microservice:

FROM rust:1.81

RUN apt-get update && apt-get install -y lldb
WORKDIR /app
COPY ./Cargo.toml ./Cargo.lock ./
RUN mkdir -p src && echo "fn main() {}" > src/main.rs && cargo build
COPY ./src ./src

ENV RUST_LOG=debug
CMD ["cargo", "run"]

The Challenge

While building inside containers is fantastic for keeping the developer's machine clean, I’m running into issues with Tilt’s live reload and syncing features, which seem more geared towards builds happening on the host machine. Here are some of the problems I’ve encountered:

1. Building Inside the Container with cargo watch: I run cargo watch inside the container to automatically detect source code changes. I use sync() in live_update() to transfer source files into the container. However:
   • Problem 1: If I only sync source files, build artifacts (like Rust’s target/ directory) are generated inside the container, and rust-analyzer on the host machine doesn’t work because it requires access to target/ to provide its code analysis features.
2. Syncing the target/ Directory: To resolve this, I tried sync()ing the target/ directory back to the host, but:
   • Problem 2: Syncing the target/ directory causes any change in target/ to trigger a live reload, which leads to a continuous loop of rebuilding defeating the purpose of live reloading efficiently.
3. Mounting target/ with Kubernetes: I could theoretically mount the target/ directory using a Kubernetes volume (hostPath), but:
   • Problem 3: Kubernetes requires an absolute path on the host for mounting, which isn't ideal because I want to keep the setup agnostic of each engineer's specific directory structure on their dev machine.

The Ideal Setup

• Build inside the container: All build artifacts, such as target/, should be generated inside the container.
• Sync source code and target/: Source files should be synced into the container, and target/ should be synced back to the host for tools like rust-analyzer to function correctly.
• Avoid reload triggers on target/ changes: I need to sync() target/ back to the host without triggering live reloads whenever the target/ directory is updated.

Question

How can I approach this setup using Tilt to achieve:

• Containerized builds without needing local tooling on the host.
• Working rust-analyzer on the host with access to the target/ directory.
• Syncing of the target/ directory from container to host, without triggering live reload on target/ changes.
• Avoiding Kubernetes hostPath absolute path requirements to make this setup agnostic of engineers’ local directory structures.

Am I thinking about this the wrong way altogether?

Looking forward to your thoughts!

Start with the root Tiltfile and compare how each service's Dockerfile.dev and Deployment.yaml define source syncing, builds, and live updates. Determine whether Tilt can support containerized builds while syncing target/ without reload loops or host-specific Kubernetes paths; the issue names no tests or implementation entry point.