Consider moving Prometheus scrape configs to PodMonitor relabelings

At present Prometheus scrape configs are defined in prometheus-template.yaml. This is fine for setting up a dedicated Prometheus server specially for ClickHouse. However if Prometheus is already up and running it it is more convenient to configure Prometheus label replacements in a PodMonitor Kubernetes object as below.

---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: myclickhouse
  namespace: "clickhouse"
  labels:
    release: kube-prometheus-stack
spec:
  podMetricsEndpoints:
    - interval: 30s
      path: /metrics
      port: "metrics"
      scheme: http
    - relabelings:
      - action: drop
        sourceLabels:
          - __meta_kubernetes_pod_annotation_prometheus_io_scrape
      - action: replace
        sourceLabels:
          - __meta_kubernetes_pod_ip
          - __meta_kubernetes_pod_annotation_prometheus_io_port
        regex: "(.+);(.+)"
        replacement: "$1:$2"
        targetLabel: __address__
      - action: replace
        sourceLabels:
          - __meta_kubernetes_pod_annotation_prometheus_io_path
        regex: "()"
        replacement: "/metrics"
        targetLabel: __meta_kubernetes_pod_annotation_prometheus_io_path
      - action: replace
        sourceLabels:
          - __meta_kubernetes_pod_annotation_prometheus_io_path
        targetLabel: __metrics_path__
      - action: replace
        sourceLabels:
          - __meta_kubernetes_namespace
        targetLabel: namespace
      - action: replace
        sourceLabels:
          - __meta_kubernetes_pod_name
        targetLabel: pod_name
      - action: replace
        sourceLabels:
          - __meta_kubernetes_pod_container_name
        targetLabel: container_name
      - action: labelmap
        regex: "__meta_kubernetes_pod_label_(.+)"
      - action: labeldrop
        regex: "container_id|image_id|controller_revision_hash|pod_template_generation|pod_template_hash"
  selector:
    matchLabels:
      clickhouse.altinity.com/app: chop

Start with deploy/prometheus/prometheus-template.yaml and compare its current scrape configuration with the proposed PodMonitor relabelings. Determine how the dedicated Prometheus setup should map to a PodMonitor for an existing Prometheus installation, and verify that the resulting configuration preserves the needed ClickHouse metrics and labels.