APM Server refuses to start on invalid config option
- Dominant language
- Go
- Stars
- 1.3k
- Forks
- 543
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 109
Description
**APM Server version** (`apm-server version`): upgrade from <=`8.3.0` to >=`8.3.1`
**Description of the problem including expected versus actual behavior**:
In previous versions, it was possible to configure values of invalid type for some package policy values. For example, `rum_allow_origins: "['*']"` was allowed, although it gets interpreted as an array and the expected type is a string. The UI does not allow this anymore in more recent versions. However, it can happen that the invalid values were configured in previous versions and are still defined in the policy when upgrading to newer versions.
In versions up until and including `8.3.0` the APM Server logged an error for config options of invalid type, but gracefully ignored them and started up. From version `8.3.1` onwards, the APM Server refuses to start when receiving a configuration of an invalid type. This leads to deployments entering a boot loop for invalid apm package policy configurations.
Investigating the changes that went into the APM Server between `8.3.0` and `8.3.1`, https://github.com/elastic/apm-server/pull/8501/files pulled in dependency updates, which most certainly contained an update on how returned config errors are handled.
**Steps to reproduce**:
* Create an `8.3.0` deployment on ESS and change the `rum_allow_origins` to `"['*']"`

* Observe that the APM Server logs an error but is still reachable
* Upgrade to `8.3.1` and observe that the APM Server is never recovering from the invalid config option.
**Provide logs (if relevant)**:
APM Server logs:
**8.3.0**

**8.3.1**
Contributor guide
Assessment
This issue has not been assessed yet.