ClickHouseOperatorConfiguration watch.namespaces.include is overwritten by env WATCH_NAMESPACE in OLM installation

Description

When ClickHouse Operator is installed via OLM, the generated CSV contains the following environment variable:

- name: WATCH_NAMESPACE
  valueFrom:
    fieldRef:
      fieldPath: metadata.namespace

As a result, the operator pod starts with:

WATCH_NAMESPACE=openshift-operators

I created a ClickHouseOperatorConfiguration resource to enable watch-all mode:

apiVersion: clickhouse.altinity.com/v1
kind: ClickHouseOperatorConfiguration
metadata:
  name: watch-all-namespaces
  namespace: openshift-operators
spec:
  watch:
    namespaces:
      include:
        - ".*"

According to the default operator configuration template, [".*"] should force watch-all mode:

https://github.com/Altinity/clickhouse-operator/blob/master/deploy/builder/templates-config/config.yaml

Observed behavior

The operator successfully discovers and loads the ClickHouseOperatorConfiguration resource:

I0615 12:13:42.047018       1 config_manager.go:150] getAllCRBasedConfigs():Looking for ClickHouseOperatorConfigurations in namespace 'openshift-operators'.
I0615 12:13:43.086423       1 config_manager.go:183] getAllCRBasedConfigs():Append ClickHouseOperatorConfigurations 'openshift-operators/watch-all-namespaces'.

The CR-based configuration contains the expected value:

watch:
  namespaces:
    include:
      - .*

The merged configuration also contains the expected value:

Unified CHOP config - with secret data fetched (but not post-processed yet)

runtime:
    configFilePath: /etc/clickhouse-operator/config.yaml
    configFolderPath: /etc/clickhouse-operator
    configCRNamespace: openshift-operators
    configCRName: watch-all-namespaces
    configCRSources:
        - namespace: openshift-operators
          name: watch-all-namespaces
    namespace: ""
watch:
    namespaces:
        include:
            - .*

However, after post-processing, the final configuration changes to:

Final CHOP config

runtime:
    configFilePath: /etc/clickhouse-operator/config.yaml
    configFolderPath: /etc/clickhouse-operator
    configCRNamespace: openshift-operators
    configCRName: watch-all-namespaces
    configCRSources:
        - namespace: openshift-operators
          name: watch-all-namespaces
    namespace: openshift-operators

watch:
    namespaces:
        include:
            - openshift-operators

As a result, ClickHouseInstallation resources created in other namespaces are not reconciled.

Investigation

The operator calls Postprocess() between the "Unified CHOP config" and "Final CHOP config" stages:

https://github.com/Altinity/clickhouse-operator/blob/release-0.27.1/pkg/chop/config_manager.go#L129

log.V(1).Info("Unified CHOP config - with secret data fetched (but not post-processed yet):")
log.V(1).Info("\n" + cm.config.String(true))

cm.Postprocess()

log.V(1).Info("Final CHOP config:")
log.V(1).Info("\n" + cm.config.String(true))

Postprocess() invokes applyEnvVarParams():

https://github.com/Altinity/clickhouse-operator/blob/release-0.27.1/pkg/apis/clickhouse.altinity.com/v1/type_configuration_chop.go#L1486

func (c *OperatorConfig) applyEnvVarParams() {
	if ns := os.Getenv(deployment.WATCH_NAMESPACE); len(ns) > 0 {
		// We have WATCH_NAMESPACE explicitly specified
		c.Watch.Namespaces.Include = types.NewStrings([]string{ns})
	}

	if nss := os.Getenv(deployment.WATCH_NAMESPACES); len(nss) > 0 {
		// We have WATCH_NAMESPACES explicitly specified
		if namespaces := c.splitNamespaces(nss); len(namespaces) > 0 {
			c.Watch.Namespaces.Include = types.NewStrings(namespaces)
		}
	}

	if nss := os.Getenv(deployment.WATCH_NAMESPACES_EXCLUDE); len(nss) > 0 {
		// We have WATCH_NAMESPACES_EXCLUDE explicitly specified
		if namespaces := c.splitNamespaces(nss); len(namespaces) > 0 {
			c.Watch.Namespaces.Exclude = types.NewStrings(namespaces)
		}
	}
}

This appears to overwrite the value previously loaded from ClickHouseOperatorConfiguration.

Expected behavior

If spec.watch.namespaces.include is explicitly configured in ClickHouseOperatorConfiguration, it should remain unchanged during post-processing.

Alternatively, if WATCH_NAMESPACE is expected to have higher priority in OLM deployments, this behavior should be documented explicitly.

Question

Is this expected behavior for OLM installations, or is this a configuration precedence issue?

Environment

• RedHat OpenShift cluster version 4.20.19
• Operator installed from OLM Community Operators
• Altinity® Kubernetes Operator for ClickHouse version 0.27.1

Start with pkg/chop/config_manager.go to trace Postprocess(), then inspect applyEnvVarParams() in pkg/apis/clickhouse.altinity.com/v1/type_configuration_chop.go. Determine the intended precedence between WATCH_NAMESPACE and spec.watch.namespaces.include, and verify that the final configuration preserves the explicitly configured CR value or documents the environment variable's priority.