oauth2 multiple auth requests

What did you do?
I implemented oauth2 authentication. I see 2 requests to the token server, one with basic-auth credentials (generated from client_id and client_secret) which gets a http 200 back from my server, but right after this successful request there is another request with credentials inside the payload. This request leads to a http 4xx, because this is not implemented at my oauth server.
So why does the alertmanager make two authentication calls to the oauth2 server?

What did you expect to see?
Alertmanager only calls the oauth2 server once, if the call was successful.

What did you see instead? Under which circumstances?
There is another call after the first successful one, this call leads to an error and because of this, the alertmanager is not firing alerts.

Environment
Alertmanager Version: 0.23
Docker-container (prom/alertmanager:latest) @ CentOS7 Dockerhost

• System information:

Linux 3.10.0-1160.42.2.el7.x86_64 x86_64

• Alertmanager version:

  alertmanager, version 0.23.0 (branch: HEAD, revision: 61046b17771a57cfd4c4a51be370ab930a4d7d54)
  build user: root@e21a959be8d2
  build date: 20210825-10:48:55
  go version: go1.16.7
  platform: linux/amd64

• Prometheus version:

  prometheus, version 2.21.0 (branch: HEAD, revision: e83ef207b6c2398919b69cd87d2693cfc2fb4127)
  build user: root@a4d9bea8479e
  build date: 20200911-11:35:02
  go version: go1.15.2

• Alertmanager configuration file:

global:
  resolve_timeout: 5m
  http_config:
    tls_config:
      ca_file: <paht_to_file>

route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 2m
  repeat_interval: 2h
  receiver: 'email'
  routes:
    - receiver: 'zis'
      group_by: ['alertname']
      group_wait: 10s
      group_interval: 2m
      repeat_interval: 2h

receivers:
- name: 'zis'
  webhook_configs:
  - url: https://example.com/api_esb_prometheus/events
    http_config:
      oauth2:
        client_id: MY_USERNAME
        client_secret: MY_SUPER_SECRET
        token_url: https://example.com/api_esb_prometheus/auth/token
      tls_config:
        ca_file: <path_to_file>
       

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'

• Prometheus configuration file:

global:
  scrape_interval: 30s

rule_files:
  - /etc/prometheus/rules_prometheus.yml

alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - <TARGET_SERVER_AND_PORT>

scrape_configs:
  - job_name: 'kafka'
    scrape_interval: 30s
    metrics_path: '/metrics'
    static_configs:

      - targets: ['<SOME_SERVER_AND_PORT>']
        labels:
          instance: 'dev'
          host: 'myhost'

• Logs:

level=debug ts=2021-10-11T09:13:37.230Z caller=dispatch.go:516 component=dispatcher aggrGroup="{}/{}:{alertname=\"InstanceDown\"}" msg=flushing alerts=[InstanceDown[f96c67a][active]]
level=warn ts=2021-10-11T09:13:37.249Z caller=notify.go:724 component=dispatcher receiver=zis integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://example.com/api_esb_prometheus/events\": oauth2: cannot fetch token: 422 Unprocessable Entity\nResponse: {\"errorMessage\":\"Check your parameters!\",\"errors\":[{\"msg\":\"is required\",\"param\":\"authorization\",\"location\":\"headers\"}]}"

Start at the webhook_configs http_config oauth2 entry point and trace the token requests made for the configured token_url. Compare the basic-auth request with the subsequent payload-auth request, then run the relevant OAuth2 or webhook notification tests. Done means a successful token exchange uses the supported authentication form and the webhook notification succeeds.