tilt-dev / tilt-dev/tilt

Lots of resources and large configmaps cause tilt to hang

Open
#6,597 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Go
Stars
10.1k
Forks
413
Avg merge
1d 10h
Merged PRs (30d)
6

Description

Expected Behavior

Tilt should not hang and become unresponsive

Current Behavior

Tilt hangs and becomes unresponsive to any actions. The tiltfile loads, but show status stuck in updating.

Debugging shows the main goroutine being stuck on:

goroutine 1 [sync.RWMutex.Lock]:
sync.runtime_SemacquireRWMutex(0x1401a090d78?, 0x4?, 0x14000c52501?)
        /usr/local/go/src/runtime/sema.go:105 +0x28
sync.(*RWMutex).Lock(0x1401a091468?)
        /usr/local/go/src/sync/rwmutex.go:155 +0xfc
github.com/tilt-dev/tilt/internal/store.(*Store).Loop(0x1400072be60, {0x104ac6380, 0x14000c4fb90})
        /root/project/internal/store/store.go:161 +0x1d8
github.com/tilt-dev/tilt/internal/engine.Upper.Init({0x10466fd40?}, {0x104ac6380, 0x14000c4fb90}, {{0x14000b55b30, 0x42}, {0x106d684e8, 0x0, 0x0}, {{0x103ef7d38, 0x6}, ...}, ...})
        /root/project/internal/engine/upper.go:102 +0x58
github.com/tilt-dev/tilt/internal/engine.Upper.Start({0x2?}, {0x104ac6380, 0x14000c4fb90}, {0x106d684e8, 0x0, 0x0}, {{0x103ef7d38, 0x6}, {0x0, 0x0}, ...}, ...)
        /root/project/internal/engine/upper.go:88 +0x178

and a large percentage of time spent on apicmp.DeepEqual while holding a RLock on the store mtx.

Steps to Reproduce
  1. Spin up large tilt environment with 500 resources, using lots of file watches
  2. Create large configmaps and load them into tilt
  3. trigger a watched file change to retrigger the bug

I have also included a tar.gz file with a setup that seems to reliably reproduce the issue. I used a kind cluster for that setup, as shown in the tilt doctor output below.

Context
tilt doctor Output
$ tilt doctor
Tilt: v0.35.0, built 2025-06-13
System: darwin-arm64
---
Docker
- Host: unix:///Users/greg.lightfoot/.docker/run/docker.sock
- Server Version: 28.3.2
- API Version: 1.51
- Builder: 2
- Compose Version: v2.38.2-desktop.1
---
Kubernetes
- Env: kind
- Context: kind-tilt-repro
- Cluster Name: kind-tilt-repro
- Namespace: default
- Container Runtime: containerd
- Version: v1.27.3
- Cluster Local Registry: none
---
Thanks for seeing the Tilt Doctor!
Please send the info above when filing bug reports. 💗

The info below helps us understand how you're using Tilt so we can improve,
but is not required to ask for help.
---
Analytics Settings
--> (These results reflect your personal opt in/out status and may be overridden by an `analytics_settings` call in your Tiltfile)
- User Mode: opt-out
- Machine: 323e9d1ad6016ab4822de03a28c50b84
- Repo: 3y4NopketDVRWDViKN9sPA==
...
About Your Use Case

We use tilt extensively for developer environments, and some of those are quite large. We have seen an increasing number of problems with tilt becoming stuck, requiring the developers to have to recreate their environment.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the hang with the large-resource and large-configmap setup described in the issue, then inspect internal/store/store.go and internal/engine/upper.go around Store.Loop and Upper.Init. Profile the time spent in apicmp.DeepEqual while the store read lock is held. Done means Tilt remains responsive after a watched file change in a similarly large environment.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devtools, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.