Service starts with a (unkillable) left-over process in control group

systemd version the issue has been seen with

237

I'm sorry about breaking "two most recently released versions" rule but I have a strong suspicion that it's still not fixed yet (e.g. https://bugzilla.redhat.com/show_bug.cgi?id=1530556). I've also read the changelog.

Used distribution

Ubuntu 18.04.2 LTS

Expected behaviour you didn't see
With relevant settings any left-over process in control group should not be ignored while starting unit because it can cause wild undefined behavior (including silent data corruption). Systemd should either at least just refuse to start such service in such circumstances (because a service clearly is in an inconsistent broken state this way) or try to kill the leftover and refuse to start if kill fails.

Unexpected behaviour you saw
Systemd just started a new process with "Found left-over process ... in control group while starting unit. Ignoring." in log and the left-over process continued to run along with the new one.

Steps to reproduce the problem
I don't know how to reproduce it - the hanged process became a zombie in D state, likely while communicating with a disk (baremetal nvme ssd).
The service is aerospike, it entered some weird state and I decided to restart it:

Aug 07 10:48:13 asd[3126]: Aug 07 2019 10:48:13 GMT: WARNING (service): (service.c:240) (repeated:20533828) out of file descriptors
Aug 07 10:48:13 asd[3126]: Aug 07 2019 10:48:13 GMT: INFO (as): (signal.c:194) SIGTERM received, starting normal shutdown
Aug 07 10:48:13 asd[3126]: Aug 07 2019 10:48:13 GMT: INFO (storage): (storage.c:774) initiating storage shutdown ...
Aug 07 10:48:13 asd[3126]: Aug 07 2019 10:48:13 GMT: INFO (storage): (storage.c:775) flushing data to storage ...
Aug 07 10:48:13 systemd[1]: Stopping Aerospike Server...
Aug 07 10:48:28 systemd[1]: aerospike.service: State 'stop-sigterm' timed out. Killing.
Aug 07 10:48:28 systemd[1]: aerospike.service: Killing process 3126 (asd) with signal SIGKILL.
Aug 07 10:48:43 systemd[1]: aerospike.service: Processes still around after SIGKILL. Ignoring.
Aug 07 10:48:59 systemd[1]: aerospike.service: State 'stop-post' timed out. Terminating.
Aug 07 10:49:14 systemd[1]: aerospike.service: State 'stop-final-sigterm' timed out. Killing.
Aug 07 10:49:14 systemd[1]: aerospike.service: Killing process 3126 (asd) with signal SIGKILL.
Aug 07 10:49:14 systemd[1]: aerospike.service: Killing process 3139 (asd) with signal SIGKILL.
Aug 07 10:49:29 systemd[1]: aerospike.service: Processes still around after final SIGKILL. Entering failed mode.
Aug 07 10:49:29 systemd[1]: aerospike.service: Failed with result 'timeout'.
Aug 07 10:49:29 systemd[1]: Stopped Aerospike Server.
Aug 07 10:49:29 systemd[1]: aerospike.service: Found left-over process 3139 (asd) in control group while starting unit. Ignoring.
Aug 07 10:49:29 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Aug 07 10:49:29 systemd[1]: Starting Aerospike Server...
Aug 07 10:49:29 systemd[1]: aerospike.service: Found left-over process 3139 (asd) in control group while starting unit. Ignoring.
Aug 07 10:49:29 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Aug 07 10:49:29 systemd[1]: Started Aerospike Server.

3126 is the process (in Z state thus far) and 3139 is its thread (in D state thus far).
Service settings (from systemctl show aerospike --all):

Type=simple
RemainAfterExit=no
KillMode=control-group
KillSignal=15
SendSIGKILL=yes

Start with the systemd service-start path and the KillMode=control-group handling shown in the report; inspect how the “Found left-over process” condition is logged after stop timeouts. Use the aerospike.service settings and systemctl output as reproduction context, noting that the report has no reliable reproducer. Done means the behavior for an unkillable D-state process is decided and covered by a regression test.