[ML] macOS: Controller process sometimes terminated with SIGKILL

Some developers working in the Elasticsearch repository have reported intermittent problems with the machine learning controller process crashing when running a locally built Elasticsearch. The usual symptoms are Elasticsearch will fail to start and the log will contain this message:

```
[ERROR][o.e.b.Elasticsearch ] [runTask-0] fatal exception while booting Elasticsearch org.elasticsearch.ElasticsearchException: Failure running machine learning native code. This could be due to running on an unsupported OS or distribution, missing OS libraries, or a problem with the temp directory. To bypass this problem by running Elasticsearch without machine learning functionality set [xpack.ml.enabled: false].
```

A crash report can be found in the macOS Console app.

```
Path: /Users/USER/*/controller.app/Contents/MacOS/controller
Identifier: co.elastic.ml-cpp.controller
Version: 8.7.0
Code Type: ARM-64 (Native)

Exception Type: EXC_BAD_ACCESS (SIGKILL (Code Signature Invalid))
Exception Subtype: UNKNOWN_0x32 at 0x000000010249c000
Exception Codes: 0x0000000000000032, 0x000000010249c000
VM Region Info: 0x10249c000 is in 0x10249c000-0x1024b0000; bytes after start: 0 bytes before end: 81919
REGION TYPE START - END [ VSIZE] PRT/MAX SHRMOD REGION DETAIL
UNUSED SPACE AT START
---> mapped file 10249c000-1024b0000 [ 80K] r-x/r-x SM=COW ...t_id=787f689f
mapped file 1024b0000-1024b4000 [ 16K] rw-/rw- SM=COW ...t_id=787f689f
Exception Note: EXC_CORPSE_NOTIFY
Termination Reason: CODESIGNING 2
```

The error has been observed on Apple silicon only (so far).

### Reproducing
It has not been possible to reproduce reliably but once the problem occurs a crash report can be generated by running `controller --help` in the Elasticsearch repository.

```
/distribution/archives/darwin-aarch64-tar/build/install/elasticsearch-8.7.0-SNAPSHOT/modules/x-pack-ml/platform/darwin-aarch64/controller.app/Contents/MacOS/controller --help
```

### Running the app from a different location works ?!
Copy the app to a folder in the home directory and running the copy does not result in a crash:

```
cd ~/Desktop
cp -r /distribution/archives/darwin-aarch64-tar/build/install/elasticsearch-8.7.0-SNAPSHOT/modules/x-pack-ml/platform/darwin-aarch64/controller.app .
controller.app/Contents/MacOS/controller --help
```

### Possible Causes
#### macOS Quarantine
No.

The downloaded `controller.app` does not have the the *quarantine* attribute set.
`find . -xattrname com.apple.quarantine` returns nothing.

#### Security Policy
No.
After disabling security with `sudo spctl --global-disable` the controller app still crashes.

```
cd
sudo spctl --global-disable
sudo spctl --asses -vv ./distribution/archives/darwin-aarch64-tar/build/install/elasticsearch-8.7.0-SNAPSHOT/modules/x-pack-ml/platform/darwin-aarch64/controller.app/Contents/MacOS/controller

./distribution/archives/darwin-aarch64-tar/build/install/elasticsearch-8.7.0-SNAPSHOT/modules/x-pack-ml/platform/darwin-aarch64/controller.app/Contents/MacOS/controller: accepted
override=security disabled

echo $?
0
```

When security is enabled the `spctl --assess` function returns the same message as `codesign --verify`

```
cd
sudo spctl --asses -vv ./distribution/archives/darwin-aarch64-tar/build/install/elasticsearch-8.7.0-SNAPSHOT/modules/x-pack-ml/platform/darwin-aarch64/controller.app/Contents/MacOS/controller

./distribution/archives/darwin-aarch64-tar/build/install/elasticsearch-8.7.0-SNAPSHOT/modules/x-pack-ml/platform/darwin-aarch64/controller.app/Contents/MacOS/controller: code has no resources but signature indicates they must be present

echo $?
1
```

#### Code Signing
Maybe.

The crash report indicates code signing is involved
```
Exception Type: EXC_BAD_ACCESS (SIGKILL (Code Signature Invalid))
```
and
```
Termination Reason: CODESIGNING 2
```

Verifying the signing returns an error message
```
cd
codesign -d --verify --verbose=4 ./distribution/archives/darwin-aarch64-tar/build/install/elasticsearch-8.7.0-SNAPSHOT/modules/x-pack-ml/platform/darwin-aarch64/controller.app

./distribution/archives/darwin-aarch64-tar/build/install/elasticsearch-8.7.0-SNAPSHOT/modules/x-pack-ml/platform/darwin-aarch64/controller.app/Contents/MacOS/controller: code has no resources but signature indicates they must be present
```

It is not clear if that is a terminal error however.

### Workarounds
_In the commands below replace `elasticsearch-8.7.0-SNAPSHOT` with your version._

- Deleting the bundled app from the local build and rebuilding is the most reliable fix:
```
cd
rm -rf distribution/archives/darwin-aarch64-tar/build/install/elasticsearch-8.7.0-SNAPSHOT/modules/x-pack-ml/platform/darwin-aarch64/controller.app
./gradlew run
```

- Resigning the app with an ad-hoc signature works for some
```
codesign --force --deep --sign - /distribution/archives/darwin-aarch64-tar/build/install/elasticsearch-8.7.0-SNAPSHOT/modules/x-pack-ml/platform/darwin-aarch64/controller.app
```
- If all else fails Restart the Machine