Altinity / Altinity/clickhouse-operator

Operator does not remove stale users configuration when applying an updated CHI manifest

Open
#1,518 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
2.6k
Forks
574
Avg merge
8d 6h
Merged PRs (30d)
6

Description

Consider a CHI manifest:

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "clickhouse-installation-1"
spec:
  configuration:
    users:
      datadog_agent/password: datadog_agent
      datadog_agent/networks/ip: "::/0"
      datadog_agent/profile: default
      datadog_agent/quota: default
  # ... etc.

Note that the datadog_agent's networks configuration has a single entry (ip)

When the operator finishes reconciling this CHI, you can find the following entry in the clickhouse container's /etc/clickhouse-server/users.d/chop-generated-users.xml:

        <datadog_agent>
            <networks>
                <host_regexp>(chi-clickhouse-installation-1-[^.]+\d+-\d+|clickhouse\-clickhouse-installation-1)\.transaction-tracing-data\.svc\.cluster\.local$</host_regexp>
                <ip>::1</ip>
                <ip>127.0.0.1</ip>
                <ip>::/0</ip>
            </networks>
            <password_sha256_hex>ae1dbda236a2e10c903ae908e6338bdf3c60aab5530b07414f10ba1eedb6c051</password_sha256_hex>
            <profile>default</profile>
            <quota>default</quota>
        </datadog_agent>

Note that the datadog_agent's networks configuration has the ip config we specified in the manifest (::/0).

It also has a host_regexp config and two ip configs, which seem to be inherited/copied from the default user. AFAICT this behaviour isn't documented and so seeing this was a little confusing; however, this is not what my issue is about.

Now, kubectl apply the following (updated) CHI manifest

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "clickhouse-installation-1"
spec:
  configuration:
    users:
      datadog_agent/password: datadog_agent
      datadog_agent/networks/host_regexp: \.datadog\.datadog\.svc\.cluster\.local$
      datadog_agent/profile: default
      datadog_agent/quota: default
  # ... etc.

Note the only change is that the networks configuration no longer has an ip; it now (only) has a host_regexp.

When the operator has finished reconciling the changes produced by this manifest, the chop-generated-users.xml

        <datadog_agent>
            <networks>
                <host_regexp>\.datadog\.datadog\.svc\.cluster\.local$</host_regexp>
                <ip>::1</ip>
                <ip>127.0.0.1</ip>
                <ip>::/0</ip>
            </networks>
            <password_sha256_hex>ae1dbda236a2e10c903ae908e6338bdf3c60aab5530b07414f10ba1eedb6c051</password_sha256_hex>
            <profile>default</profile>
            <quota>default</quota>
        </datadog_agent>

Note that the host_regexp entry has been added, but the original ip entry ::/0 still remains.

Is this intentional? It is not something I would have expected, and afaict it is not documented anywhere in the operator's spec (https://github.com/Altinity/clickhouse-operator/blob/master/tests/requirements/requirements.md).

If we fetch CHI resource from kubernetes, it will show that both entries now exist (and so it seems like the issues lies somewhere in the reconciliation loop, and not the "config file generation" process):

# truncated output from `kubectl get chi -o yaml`
          users:
            datadog_agent/networks/host_regexp: \.datadog\.datadog\.svc\.cluster\.local$
            datadog_agent/networks/ip:
            - ::1
            - 127.0.0.1
            - ::/0
            datadog_agent/password_sha256_hex: ae1dbda236a2e10c903ae908e6338bdf3c60aab5530b07414f10ba1eedb6c051
            datadog_agent/profile: default
            datadog_agent/quota: default

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the update with kubectl apply and inspect the CHI resource plus chop-generated-users.xml after reconciliation. Trace the reconciliation loop that merges configuration.users entries; done means removing the old datadog_agent/networks/ip value when the updated manifest specifies only host_regexp, while retaining the new configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
databases, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.