hashicorp / hashicorp/consul

ca: UpdateConfiguration fails if initialization is not complete

Open
#11,351 1 comment 1 reaction 0 assignees View on GitHub
theme/certificates theme/connect type/bug
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

This issue was reported by @lkysow from this [test failure](https://app.circleci.com/pipelines/github/hashicorp/consul-k8s/3002/workflows/02c23af9-df43-489d-b100-364ce3e9c090/jobs/21771).

The problem seems to be that while the CAManager "state machine" allows `UpdateConfiguration` to be called, it ends up failing [here](https://github.com/hashicorp/consul/blob/v1.10.3/agent/consul/leader_connect_ca.go#L841-L865) because there is no existing provider.

Before https://github.com/hashicorp/consul/pull/9570 this would not have been an issue because nothing would work until `InitializeCA` was complete.

It seems like we have two options to solve this:
1. Do not allow `UpdateConfiguration` until the provider is Initialized. This seems like the easiest option and would help reduce complexity in the `CAMananger`.
2. Allow `UpdateConfiguration` to save the config, but skip the other steps and let `InitializeCA` handle it when it runs.

I think there is a really good argument to be made for option 1. The failures in the test output are a great example. The test continually ran `CASetConfig` (which calls `UpdateConfiguration`) [in a loop](https://github.com/hashicorp/consul-k8s/blob/main/control-plane/subcommand/get-consul-client-ca/command_test.go#L255-L264). It would sleep for 25ms between iterations, but because `UpdateConfiguration` would run for 500ms before failing (see the log output), every `InitializeCA` attempt would still fail, because the "state" was `reconfig` every time. It spent 20x more time with the state in "reconfig" than in "uninitialized" (the state required by InitializeCA).

Related issue #10634, for making more general improvements to this state management.

Contributor guide

Open the contributing guide

Research direction

Start with the CAManager state handling around UpdateConfiguration and InitializeCA, then inspect agent/consul/leader_connect_ca.go at the referenced failure and control-plane/subcommand/get-consul-client-ca/command_test.go around the retry loop. Compare the two proposed state-management options and confirm the test no longer leaves initialization unable to progress when configuration updates arrive early.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.