tailscale / tailscale/tailscale

FR: Improve error messaging when node registration fails due to ACL IP pool exhaustion

Open
#19,056 0 comments 0 reactions 0 assignees View on GitHub
fr needs-triage
Dominant language
Go
Stars
36.5k
Forks
3.2k
Avg merge
2d 3h
Merged PRs (30d)
123

Description

### What are you trying to do?

When running Tailscale in Kubernetes pods with tag-based IP pools configured via ACL nodeAttrs, I need to be able to diagnose why a pod fails to register. Currently, if the IP pool assigned to a tag is exhausted (e.g., more devices with tag:k8s than the /28 pool can hold), the control server returns a bare HTTP 500. The client surfaces this as:
register request: http 500:
There is no indication that the root cause is IP pool exhaustion. The node silently retries forever (since control/controlclient/errors.go classifies 500 as retryable), and there's no way to surface the actual failure reason through standard Kubernetes tooling like kubectl describe pod.
The only way to diagnose this today is to manually reason about your ACL ipPool CIDR ranges vs. the number of devices using that tag, which is not obvious at all.

### How should we solve this?

1. Control server: When IP pool allocation fails during registration, populate RegisterResponse.Error (which already exists in tailcfg/tailcfg.go) with a descriptive message instead of returning HTTP 500. Something like:
IP pool exhausted for tag:k8s (pool: 100.64.1.0/28, 14/14 addresses allocated). Expand the ipPool range in your ACL nodeAttrs.
The client already handles this path at control/controlclient/direct.go via vizerror.New(resp.Error), so this would surface immediately.
2. Client error classification: IP pool exhaustion is a permanent failure, not a transient server error. Using RegisterResponse.Error instead of HTTP 500 would bypass the retry loop in control/controlclient/errors.go and surface the error immediately rather than retrying forever.
3. Kubernetes operator: When the Tailscale client in a pod receives a registration error, emit a Kubernetes Event on the pod so kubectl describe pod shows something like:
**Warning RegistrationFailed Registration failed: IP pool exhausted for tag:k8s ...**


4. NAT connector (cmd/natc/natc.go): When IPForDomain() fails due to pool exhaustion, return a DNS SERVFAIL response instead of silently dropping the request (current behavior is log-and-return with no response, causing client timeouts).

### What is the impact of not solving this?

Anyone using IP pools with tags — particularly in Kubernetes where ephemeral pods churn through pool addresses — hits a wall of cryptic 500 errors with no actionable information. Debugging requires guessing that the 500 means "your pool is too small," which is not documented anywhere and not inferable from the error message. The silent infinite retry loop also wastes resources and masks the real problem, making it look like a connectivity or auth issue rather than a configuration issue.

### Anything else?

Relevant code paths in the open-source repo:
- control/controlclient/direct.go:~718 — client-side registration error handling (wraps 500 as generic string)
- control/controlclient/direct.go:~738 — already handles RegisterResponse.Error via vizerror.New() (this path works, the server just doesn't use it for this case)
- control/controlclient/errors.go:~44 — classifies 500 as retryable, causing infinite retry
- tailcfg/tailcfg.go:1319 — RegisterResponse.Error field already exists for this purpose
- cmd/natc/natc.go:~389 — silent drop on IP pool exhaustion in DNS handler
- cmd/natc/ippool/ippool.go — ErrNoIPsAvailable error exists but isn't propagated descriptively
The RegisterResponse.Error → vizerror.New() path is the lowest-friction fix since both sides already support it — the server just needs to use it instead of returning a bare 500.

**DISCLAMER: this has been researched in tandem with multiple LLM Agents. To the best of my knowledge and experience, this looks to be a sound argument and reasoning**

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.