No way to disable Context builders

Description

In order to maintain a builder instance for every Docker Context, buildx seems to continually try to connect to remote contexts. This isn’t a behaviour I want and doesn’t seem to be something I can turn off. When creating a context, my intention was to use it as a better-supported alternative to exporting DOCKER_HOST when temporarily connecting to another machine, not to set up anything long-running.

As-is, there are three issues this causes

• Docker is continually making SSH connections to external systems, including attempting to connect to production systems, without user input
• In doing this it tries to pull private keys from an agent over and over again. For agents that require a user to authenticate on key access this is a nuisance
• docker builder ls takes forever to return because it’s trying to connect to every context I have set up, and half of them require VPNs etc to be set up before they’re accessible

Fundamentally I think the first of these is the biggest issue. In a feature that can’t be disabled and isn’t well-documented, Docker is trying to make background connections to remote systems without a user asking it to.

If context-builders could be disabled, or better yet were opt-in per context, this would be a lot easier. As it is I either need to firewall off the systems I don’t want this connecting to so it doesn’t get as far as attempting an SSH connection, or not using the Context functionality which is otherwise really useful.

Start by tracing how buildx creates and connects context builders, especially the behavior observed through docker builder ls and remote Docker contexts. Define whether disabling builders globally or opting out per context is the intended behavior, then verify that background SSH attempts and builder listing no longer occur for disabled contexts.