elastic / elastic/ml-cpp

[ML] macOS: Controller process sometimes terminated with SIGKILL

Aperta
#2,429 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub
>build
Lingua principale
C++
Stelle
157
Fork
67
Merge medio
17h 52m
PR unite (30g)
20

Descrizione

Some developers working in the Elasticsearch repository have reported intermittent problems with the machine learning controller process crashing when running a locally built Elasticsearch. The usual symptoms are Elasticsearch will fail to start and the log will contain this message:

```
[ERROR][o.e.b.Elasticsearch ] [runTask-0] fatal exception while booting Elasticsearch org.elasticsearch.ElasticsearchException: Failure running machine learning native code. This could be due to running on an unsupported OS or distribution, missing OS libraries, or a problem with the temp directory. To bypass this problem by running Elasticsearch without machine learning functionality set [xpack.ml.enabled: false].
```

A crash report can be found in the macOS Console app.

```
Path: /Users/USER/*/controller.app/Contents/MacOS/controller
Identifier: co.elastic.ml-cpp.controller
Version: 8.7.0
Code Type: ARM-64 (Native)

Exception Type: EXC_BAD_ACCESS (SIGKILL (Code Signature Invalid))
Exception Subtype: UNKNOWN_0x32 at 0x000000010249c000
Exception Codes: 0x0000000000000032, 0x000000010249c000
VM Region Info: 0x10249c000 is in 0x10249c000-0x1024b0000; bytes after start: 0 bytes before end: 81919
REGION TYPE START - END [ VSIZE] PRT/MAX SHRMOD REGION DETAIL
UNUSED SPACE AT START
---> mapped file 10249c000-1024b0000 [ 80K] r-x/r-x SM=COW ...t_id=787f689f
mapped file 1024b0000-1024b4000 [ 16K] rw-/rw- SM=COW ...t_id=787f689f
Exception Note: EXC_CORPSE_NOTIFY
Termination Reason: CODESIGNING 2
```

The error has been observed on Apple silicon only (so far).

### Reproducing
It has not been possible to reproduce reliably but once the problem occurs a crash report can be generated by running `controller --help` in the Elasticsearch repository.

```
/distribution/archives/darwin-aarch64-tar/build/install/elasticsearch-8.7.0-SNAPSHOT/modules/x-pack-ml/platform/darwin-aarch64/controller.app/Contents/MacOS/controller --help
```

### Running the app from a different location works ?!
Copy the app to a folder in the home directory and running the copy does not result in a crash:

```
cd ~/Desktop
cp -r /distribution/archives/darwin-aarch64-tar/build/install/elasticsearch-8.7.0-SNAPSHOT/modules/x-pack-ml/platform/darwin-aarch64/controller.app .
controller.app/Contents/MacOS/controller --help
```

### Possible Causes
#### macOS Quarantine
No.

The downloaded `controller.app` does not have the the *quarantine* attribute set.
`find . -xattrname com.apple.quarantine` returns nothing.

#### Security Policy
No.
After disabling security with `sudo spctl --global-disable` the controller app still crashes.

```
cd
sudo spctl --global-disable
sudo spctl --asses -vv ./distribution/archives/darwin-aarch64-tar/build/install/elasticsearch-8.7.0-SNAPSHOT/modules/x-pack-ml/platform/darwin-aarch64/controller.app/Contents/MacOS/controller

./distribution/archives/darwin-aarch64-tar/build/install/elasticsearch-8.7.0-SNAPSHOT/modules/x-pack-ml/platform/darwin-aarch64/controller.app/Contents/MacOS/controller: accepted
override=security disabled

echo $?
0
```

When security is enabled the `spctl --assess` function returns the same message as `codesign --verify`

```
cd
sudo spctl --asses -vv ./distribution/archives/darwin-aarch64-tar/build/install/elasticsearch-8.7.0-SNAPSHOT/modules/x-pack-ml/platform/darwin-aarch64/controller.app/Contents/MacOS/controller

./distribution/archives/darwin-aarch64-tar/build/install/elasticsearch-8.7.0-SNAPSHOT/modules/x-pack-ml/platform/darwin-aarch64/controller.app/Contents/MacOS/controller: code has no resources but signature indicates they must be present

echo $?
1
```

#### Code Signing
Maybe.

The crash report indicates code signing is involved
```
Exception Type: EXC_BAD_ACCESS (SIGKILL (Code Signature Invalid))
```
and
```
Termination Reason: CODESIGNING 2
```

Verifying the signing returns an error message
```
cd
codesign -d --verify --verbose=4 ./distribution/archives/darwin-aarch64-tar/build/install/elasticsearch-8.7.0-SNAPSHOT/modules/x-pack-ml/platform/darwin-aarch64/controller.app

./distribution/archives/darwin-aarch64-tar/build/install/elasticsearch-8.7.0-SNAPSHOT/modules/x-pack-ml/platform/darwin-aarch64/controller.app/Contents/MacOS/controller: code has no resources but signature indicates they must be present
```

It is not clear if that is a terminal error however.

### Workarounds
_In the commands below replace `elasticsearch-8.7.0-SNAPSHOT` with your version._

- Deleting the bundled app from the local build and rebuilding is the most reliable fix:
```
cd
rm -rf distribution/archives/darwin-aarch64-tar/build/install/elasticsearch-8.7.0-SNAPSHOT/modules/x-pack-ml/platform/darwin-aarch64/controller.app
./gradlew run
```

- Resigning the app with an ad-hoc signature works for some
```
codesign --force --deep --sign - /distribution/archives/darwin-aarch64-tar/build/install/elasticsearch-8.7.0-SNAPSHOT/modules/x-pack-ml/platform/darwin-aarch64/controller.app
```
- If all else fails Restart the Machine

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia con il controller incluso in distribution/archives/darwin-aarch64-tar/build/install/elasticsearch-8.7.0-SNAPSHOT/modules/x-pack-ml/platform/darwin-aarch64/controller.app ed esegui controller --help su Apple silicon. Confronta l’app che presenta l’errore con un’app copiata, esamina i risultati di codesign e spctl e prova le soluzioni alternative documentate per la ricompilazione e la firma ad hoc. Il lavoro è completato quando viene identificata la causa del SIGKILL e viene stabilita una soluzione affidabile o una risoluzione documentata.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
cpp, macos
Ambito
build-system, machine-learning, operating-systems
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Da chiarire
Idoneità per principianti
35/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.