Special characters in Passwords may produce invalid users.xml

After setting up our clickhouse like this

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "clickhouse"
spec:
  configuration:
    users:
      user1/password:
        valueFrom:
          secretKeyRef:
            name: clickhouse-user-creds
            key: user1
      user2/password:
        valueFrom:
          secretKeyRef:
            name: clickhouse-user-creds
            key: user2
      user3/password:
        valueFrom:
          secretKeyRef:
            name: clickhouse-user-creds
            key: user3

apiVersion: v1
kind: Secret
metadata:
    name: clickhouse-user-creds
type: Opaque
stringData:
    user1: l%XubpKqz2y!QsKlsynEEE6#Thknj&fG
    user2: Hb6*N8EcrUMVwCYJieZDgxU3#*5#Yq!j
    user3: x$!XU9hK#RTxZjeJ8GKoTGD!hxpzG*jq

I got the following error when the pod tried to start up

2024.11.26 12:26:58.536489 [ 63 ] {} <Error> Application: Poco::Exception. Code: 1000, e.code() = 0, Exception: Failed to preprocess config '/etc/clickhouse-server/users.xml': SAXParseException: Invalid token in line 1 column 42, Stack trace (when copying this message, always include the lines below):
0. DB::ConfigProcessor::processConfig(bool*, zkutil::ZooKeeperNodeCache*, std::shared_ptr<Poco::Event> const&) @ 0x000000001330d461
1. DB::ConfigProcessor::loadConfig(bool) @ 0x000000001330dbb4
2. DB::ConfigReloader::reloadIfNewer(bool, bool, bool, bool) @ 0x000000001331526b
3. DB::ConfigReloader::ConfigReloader(std::basic_string_view<char, std::char_traits<char>>, std::vector<String, std::allocator<String>> const&, String const&, zkutil::ZooKeeperNodeCache&&, std::shared_ptr<Poco::Event> const&, std::function<void (Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool)>&&) @ 0x00000000133143ad
4. DB::UsersConfigAccessStorage::load(String const&, String const&, String const&, std::function<std::shared_ptr<zkutil::ZooKeeper> ()> const&) @ 0x000000001073e750
5. DB::AccessControl::addUsersConfigStorage(String const&, String const&, String const&, String const&, std::function<std::shared_ptr<zkutil::ZooKeeper> ()> const&, bool) @ 0x000000001062cf7a
6. DB::AccessControl::addStoragesFromUserDirectoriesConfig(Poco::Util::AbstractConfiguration const&, String const&, String const&, String const&, String const&, std::function<std::shared_ptr<zkutil::ZooKeeper> ()> const&) @ 0x00000000106305fc
7. DB::AccessControl::setUpFromMainConfig(Poco::Util::AbstractConfiguration const&, String const&, std::function<std::shared_ptr<zkutil::ZooKeeper> ()> const&) @ 0x000000001062b550
8. DB::Server::main(std::vector<String, std::allocator<String>> const&) @ 0x000000000d79792a
9. Poco::Util::Application::run() @ 0x00000000159c1c26
10. DB::Server::run() @ 0x000000000d786a90
11. Poco::Util::ServerApplication::run(int, char**) @ 0x00000000159caa67
12. mainEntryClickHouseServer(int, char**) @ 0x000000000d783ce6
13. main @ 0x00000000079f53ce
14. ? @ 0x00007f52c5b33083
15. _start @ 0x00000000060dc02e
 (version 24.7.3.42 (official build))
2024.11.26 12:26:58.536512 [ 63 ] {} <Information> Application: shutting down

Rotating the passwords solved the issue. I believe the % for user1 was the problem. But I have only limited access to the cluster and couldn't dump /etc/clickhouse-server/users.xml to look at the result.

I believe – generally – XML values are not escaped?

https://github.com/Altinity/clickhouse-operator/blob/d7ecc2e968a6b7c1bbc0457f5512f790b45a414f/pkg/xml/xml.go#L256-L258

I'll try to create a PR.

Start in pkg/xml/xml.go at lines 256-258 and inspect how password values are written into users.xml. Reproduce the issue with the special characters shown in the report, then verify that generated XML remains valid and preserves the password values.