Certificate Configuration in Consul Service Mesh

We are experiencing an issue with setting up certificates in Consul Service Mesh. We followed the documentation and enabled Consul Service Mesh using the built-in CA, but we continue to receive the following error:

consul[3026554]: 2024-07-02T14:26:53.702Z [ERROR] agent.proxycfg: Failed to handle update from watch: kind=connect-proxy proxy=apill-sidecar-proxy service_id=apill-sidecar-proxy id=peering-trust-bundles error="error filling agent cache: failed to list all discovery chains referring to \"apill\": failed to fetch discovery chain for \"apill\": no cluster ca config setup"
consul[3026554]: agent.proxycfg: Failed to handle update from watch: kind=connect-proxy proxy=apill-sidecar-proxy service_id=apill-sidecar-proxy id=peering-trust-bundles error="error filling agent cache: failed to list all discovery chains referring to \"apill\": failed to fetch discovery chain for \"apill\": no cluster ca config setup"

We have taken the following steps to resolve the issue:

Enabled Service Mesh by adding the following to the configuration file /etc/consul.d/config.hcl:
connect {
enabled = true
}

Created and configured a root certificate and private key using openssl:

openssl genpkey -algorithm RSA -out root.key -pkeyopt rsa_keygen_bits:2048
openssl req -x509 -new -nodes -key root.key -sha256 -days 3650 -out root.crt -subj "/C=US/ST=State/L=City/O=Organization/OU=OrgUnit/CN=example.com"
Created the configuration file ca_config.json: jq --null-input --rawfile key root.key --rawfile cert root.crt '
{
"Provider": "consul",
"Config": {
"LeafCertTTL": "72h",
"PrivateKey": $key | sub("\\n$"; ""),
"RootCert": $cert | sub("\\n$"; ""),
"IntermediateCertTTL": "8760h"
}
}' > ca_config.json

Applied the new CA configuration using the command:

curl --request PUT --data @ca_config.json localhost:8500/v1/connect/ca/configuration
Restarted Consul on all server and agent nodes Despite these steps, the error persists. We have checked the current CA configuration using:

curl localhost:8500/v1/connect/ca/configuration
{
"Provider": "consul",
"Config": {
"IntermediateCertTTL": "8760h",
"LeafCertTTL": "72h",
"PrivateKey": "-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCWvKyqKeiU9x7x\n64obHYsVJvmO0Qbx4iZEW6lhbHTKvnen6wXLYOrVDy6s1NvKmMpU51CbIapMSRVg\nDhGCc53iQCxsYi6xLzA8dXHOdVeDM1rqWnnTJHy/qZItK8ZxEdjz4boSsMg/yRN3\nK+Yq/UB5cldDGGlHucAgIoyVqCyVSgNKzINsNVpSiFr4711jSsr1Ttscw2FvMWKm\n9ZDcMNHNZEgfPTi3FktNLI6Po4X3MmVh7fSjgfaNFdLqHXbLDTyXlnZ1iXAbySDf\nN5YQNngMD/nanaQyiZCwHjQsD9OfRNCa31sn270yITqsE7ecQnbZ/FUOHiRJ6WyB\nHZ2MeyCTAgMBAAECggEAIOzHXkfXKKqt1LPZL8ENPyqLOjnIJY97QlUvMzuIazyd\nUE1Sr7DUuoMdc9zuwVsMXu1Duw7obs9L3vZTtsXgR6oyyd3MSyF+jMYufKU+30aZ\nZjUrBQakRQ0aA8nbB5mZZPP6O9jndJ9MJCyvktZMffqOuvYbEPKu15YUVSAt1NHG\n+mDC19sFIBMB6a5eZrfw6V2G0vlSIbz9Y02fQeBDfQyUgLP1adtCDKa4VH/0nEqe\nM1aJdku3Yxyfy83TRW5e+THLbNLYEMyU/KRdYmEU4WbE0GohV99HWdieJptnU47P\nAlE0Hs18831zF6lnXvlCo4P8Vl4l9o48El0SYd+tmQKBgQDKG192R+49xGa8w93d\nOViq6uj1/OotMJ867YoWD8UFOVFmRKSDfw+fwy7BOdT/N/Lz3IOr7MM7QSUdMHUX\nXIauJ/CfLJdH53c5eMYYoOa3ViAvo//JFh8vwj2nQsgHQvBswQyznZvQJLmvTrOb\nL6789eswBb3/psSy5/Fcx9Kg/QKBgQC+7pXtvtP6nGZcneU85y5heGP++Yos+5Pn\nzdMV8gnPTxizzJp+f8emn2t2ixyjiOoGGMK9FW8jeFy+qMPEyjhKqoGLaarpbB9M\nQhkcLAIZQqYpC6BGRJaZopIq9c6IVUf8m6yplCDZYY5pkDDl/5Vurpo9sAFbAgdn\nKc+M4XMEzwKBgBDiVohIHBQW60pLxqYrm0e0FkAun9WkoTjIDWgoaxoznqQ/icre\ne4yqPqav0SZ9IxPSUUG0pt61Y4dbu/x2o127QnPZws2j/abUeyAlmQCp9khkmALR\nkUtpG8N0ZceNhQ5g1lHPEexClSybE5AcXK8PUFWTtTq9UONtUxGgM+11AoGBALmE\n27tRg2lsFG4Dod1miAN/VjJGx8LjhBhlPqiBMJT1tvvlX2xwvk6hcmqRyvb/rtzT\nrEzLidy+M+ex1d7d2op6WDtdJgt+2n5fb0SXYkanIwqY2FfOh0Kq2znQ86g2rSa/\nIKr5Wl2+VAEwcMLab5lHIGm2HesO8qCD2L57xSWzAoGAUYscLHqSH44m0UYuAz+i\n+wSWZtxDvhnKA5Hg/tjLr2BHlSym5qW5cgGUAMKuarNOjzZcGzepDvGdns+o/kw0\ng59+DO5BWtf/659alfO+/JbS94YQi940pR0+hPii55FDXHrUe/DI4DL6E4lH+a3Z\ns6u+GQzZJVkBpyA9u1SkJok=\n-----END PRIVATE KEY-----",
"RootCert": "-----BEGIN CERTIFICATE-----\nMIIDtzCCAp+gAwIBAgIUDzhpQUQo7XQAFhHUoUpPW0N7DpEwDQYJKoZIhvcNAQEL\nBQAwazELMAkGA1UEBhMCVVMxDjAMBgNVBAgMBVN0YXRlMQ0wCwYDVQQHDARDaXR5\nMRUwEwYDVQQKDAxPcmdhbml6YXRpb24xEDAOBgNVBAsMB09yZ1VuaXQxFDASBgNV\nBAMMC2V4YW1wbGUuY29tMB4XDTI0MDcwMjE0MjUxOFoXDTM0MDYzMDE0MjUxOFow\nazELMAkGA1UEBhMCVVMxDjAMBgNVBAgMBVN0YXRlMQ0wCwYDVQQHDARDaXR5MRUw\nEwYDVQQKDAxPcmdhbml6YXRpb24xEDAOBgNVBAsMB09yZ1VuaXQxFDASBgNVBAMM\nC2V4YW1wbGUuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAlrys\nqinolPce8euKGx2LFSb5jtEG8eImRFupYWx0yr53p+sFy2Dq1Q8urNTbypjKVOdQ\nmyGqTEkVYA4RgnOd4kAsbGIusS8wPHVxznVXgzNa6lp50yR8v6mSLSvGcRHY8+G6\nErDIP8kTdyvmKv1AeXJXQxhpR7nAICKMlagslUoDSsyDbDVaUoha+O9dY0rK9U7b\nHMNhbzFipvWQ3DDRzWRIHz04txZLTSyOj6OF9zJlYe30o4H2jRXS6h12yw08l5Z2\ndYlwG8kg3zeWEDZ4DA/52p2kMomQsB40LA/Tn0TQmt9bJ9u9MiE6rBO3nEJ22fxV\nDh4kSelsgR2djHsgkwIDAQABo1MwUTAdBgNVHQ4EFgQUqD7heG+C4f5aMFm8F2qq\n61UqlBAwHwYDVR0jBBgwFoAUqD7heG+C4f5aMFm8F2qq61UqlBAwDwYDVR0TAQH/\nBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEABsoL8a92YL+FdW7LVyY8r/Az6qr4\nz0krW+PHn4S3lbgpZfFYP5Q+s4Wx1YIAZzb/D2VezNx5oYG7qrEZhdV9XAHZ0d+g\nW36WBd4m1U3JGGM2XvLFtciEdLoTL7rKrHA9O82CceoCFliPezyfvOopRFZNExTY\nC/r08NlB6tJf4aX8fqRvv4iAU9y7CcqDhvlxIMPuxpbhQy3FMiDyJELI8ZHAw1uV\nELyf0wW/Yqk7LpGLpSlrZAXYAclnAmeSjTUI3qK0L0Nr42XSY4pLmSqMsffgR+Hl\nV3KZWn+KZ+s0QE86tA+Am4A0vbrKmoWBpRgqr7cW8tLLIt1pbeUfM/06FA==\n-----END CERTIFICATE-----"
},
"State": null,
"ForceWithoutCrossSigning": false,
"CreateIndex": 5,
"ModifyIndex": 1886
} The configuration appears to be correct.

The issue names /etc/consul.d/config.hcl, ca_config.json, and the /v1/connect/ca/configuration endpoint; start by comparing the connect and CA settings with the Service Mesh documentation. Reproduce the reported “no cluster ca config setup” error and inspect the returned CA configuration. Done means the cause is isolated and the configuration works or the issue is narrowed with a minimal reproduction.

Written by the indexing model from the issue text.