Sideload for maintenance

I'm trying to implement a maintenance solution and followed this guide.
https://docs.influxdata.com/kapacitor/v1.5/guides/scheduled-downtime/

When I change the maintenance value in the [hostname].yml file on disk it does not work until I restart kapacitor. After a restart it picks up the new value and all alerts are suppressed. So it looks like sideload only picks up changes in files during startup of kapacitor and not every time it gets called.

Is this by design, a bug or have i done something wrong in my implementation?

Read the scheduled-downtime guide and inspect the sideload behavior around [hostname].yml during Kapacitor startup. Reproduce an edit without restarting, then compare it with a restart; done means determining whether reload-on-use is expected and documenting or fixing the observed behavior.

Written by the indexing model from the issue text.