no space left on device errors from kubernetes

failed to watch file "/var/lib/docker/containers/15ed6162d3ba16ba4e165b00a71d2b2a897cb32f1bd16d81f50121e43b5958ee/15ed6162d3ba16ba4e165b00a71d2b2a897cb32f1bd16d81f50121e43b5958ee-json.log": no space left on device

This error typically comes from Kubernetes itself, trying to set up a fsnotify watch for the log file, but is failing:
https://github.com/kubernetes/kubernetes/blob/323f34858de18b862d43c40b2cced65ad8e24052/pkg/kubelet/kuberuntime/logs/logs.go#L339

The "no space left on device" error message is a well-known bug in the underlying filesystem libraries. What's usually happening is that it's hitting an inotify limit, which the syscall reports as "out of resources", which the filesystem library assumes means "out of space".

Usually the fix is to configure the VM that Kubernetes is running on to have a higher inotify limit
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf

i wonder if Tilt could warn when this is happening and auto-fix it for you

Start with the Kubernetes kubelet logs.go entry point linked in the issue and trace how Tilt observes Kubernetes workloads. Define how an inotify-limit condition should be detected, what warning or automatic remediation is safe, and how completion would be verified without treating ordinary disk exhaustion the same way.