Requests to the Pushover API will fail if the the contents of the user key file or token file end with a newline character

What did you do?
I wrote my Pushover user key and app token to files using vim (which adds a newline to the end of each file by default) and set user_key_file and token_file in alertmanager.yml to point to those files.

What did you expect to see?
Alertmanager should have successfully sent alerts to Pushover, with a notification appearing on my phone.

What did you see instead? Under which circumstances?
When Alertmanager tried sending a notification to Pushover, Pushover returned an HTTP 400 error from its API and said that my application token was invalid.

Workaround
Either write the secrets to each file without any newlines using echo -n secret > secret_file.txt or after writing the files with newlines strip the newline character from each file using truncate --size=-1 secret_file.txt.

Environment

• System information:

Linux 6.5.6-4k-00001-g6e5115d7f640 ppc64le

• Alertmanager version:

alertmanager, version 0.26.0 (branch: HEAD, revision: d7b4f0c7322e7151d6e3b1e31cbc15361e295d8d)
  build user:       root@9623f4d745ac
  build date:       20230824-11:10:34
  go version:       go1.20.7
  platform:         linux/ppc64le
  tags:             netgo

• Prometheus version:

prometheus, version 2.48.1 (branch: HEAD, revision: 63894216648f0d6be310c9d16fb48293c45c9310)
  build user:       root@bd93f1a7309d
  build date:       20231208-23:33:20
  go version:       go1.21.5
  platform:         linux/ppc64le
  tags:             netgo,builtinassets,stringlabels

• Alertmanager configuration file:

route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h
  receiver: 'pushover'
receivers:
  - name: 'pushover'
    pushover_configs:
      - user_key_file: 'pushover_user_key.txt'
        token_file: 'pushover_token.txt'
        priority: '0'
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']

• Logs:

ts=2023-12-18T04:35:54.049Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="pushover/pushover[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 400: {\"token\":\"invalid\",\"errors\":[\"application token is invalid, see https://pushover.net/api\"],\"status\":0,\"request\":\"9eaadfe6-562e-4eae-9ed9-979374db4c11\"}"

Start by tracing Alertmanager's Pushover notification path and how the user_key_file and token_file values from alertmanager.yml are read before the Pushover API request. Reproduce the failure with files ending in newline characters, then verify that requests using those files are accepted by the API and that existing secret-file behavior remains intact.