elastic / elastic/apm-server

APM Server refuses to start on invalid config option

Open
#9,913 0 comments 0 reactions 0 assignees View on GitHub
bug v8.3.1
Dominant language
Go
Stars
1.3k
Forks
543
Avg merge
1d 18h
Merged PRs (30d)
109

Description

**APM Server version** (`apm-server version`): upgrade from <=`8.3.0` to >=`8.3.1`

**Description of the problem including expected versus actual behavior**:

In previous versions, it was possible to configure values of invalid type for some package policy values. For example, `rum_allow_origins: "['*']"` was allowed, although it gets interpreted as an array and the expected type is a string. The UI does not allow this anymore in more recent versions. However, it can happen that the invalid values were configured in previous versions and are still defined in the policy when upgrading to newer versions.
In versions up until and including `8.3.0` the APM Server logged an error for config options of invalid type, but gracefully ignored them and started up. From version `8.3.1` onwards, the APM Server refuses to start when receiving a configuration of an invalid type. This leads to deployments entering a boot loop for invalid apm package policy configurations.

Investigating the changes that went into the APM Server between `8.3.0` and `8.3.1`, https://github.com/elastic/apm-server/pull/8501/files pulled in dependency updates, which most certainly contained an update on how returned config errors are handled.

**Steps to reproduce**:
* Create an `8.3.0` deployment on ESS and change the `rum_allow_origins` to `"['*']"`
Screenshot 2022-12-28 at 16 05 23

* Observe that the APM Server logs an error but is still reachable
* Upgrade to `8.3.1` and observe that the APM Server is never recovering from the invalid config option.

**Provide logs (if relevant)**:

APM Server logs:
**8.3.0**

Screenshot 2022-12-28 at 16 03 12

**8.3.1**
Screenshot 2022-12-28 at 16 32 15

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.