Dynamic namespace handling for ServiceMonitor serverName in operator metrics scraping

Feature Request

Describe the problem you need a feature to resolve.

We have the /metrics endpoint for the operator, and we create a ServiceMonitor to scrape those metrics. However, the ServiceMonitor requires a CA and a serverName. The serverName field depends on the namespace where the operator is installed.

If the user installs the operator in a namespace different from the default one, the serverName field is incorrect. This leads to the certificate being invalid, and as a result, the ServiceMonitor cannot scrape the metrics.

For example:

 tlsConfig:
   ca: {}
   caFile: /etc/prometheus/configmaps/serving-certs-ca-bundle/service-ca.crt
   cert: {}
   serverName: my-operator-controller-manager-metrics-service.my-namespace.svc

The current solution is to create the ServiceMonitor from the operator at runtime, but this solution is not ideal because it introduces OpenShift-specific logic in upstream operators.

Describe the solution you'd like.

We want the operator-sdk to dynamically handle the serverName configuration for the ServiceMonitor based on the namespace where the operator is installed. This would ensure that the correct serverName is used, regardless of the installation namespace, making the certificate valid and allowing the ServiceMonitor to scrape the metrics properly.

Or any alternative solution that can help with this use case.

No source file or test is identified in the issue. Start by tracing the operator's /metrics ServiceMonitor configuration and how the installed namespace reaches the serverName field. Done means installations in non-default namespaces use a matching serverName so the certificate validates and metrics scraping succeeds.