elastic / elastic/ml-cpp

[ML] --skipModelValidation is compiled out of distributed builds, so graph-validation cannot be disabled in production

Aperta
#3,098 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
C++
Stelle
157
Fork
67
Merge medio
17h 52m
PR unite (30g)
20

Descrizione

## Summary

The `--skipModelValidation` command-line flag on `pytorch_inference` is guarded by the `ML_ALLOW_SKIP_MODEL_VALIDATION` CMake option, which defaults to `OFF` and is **not enabled by any build in this repository** — including the distributed/release build. As a result, shipped `pytorch_inference` binaries do not recognize `--skipModelValidation`, and passing it makes the process **fail to start**.

The Elasticsearch cluster setting `xpack.ml.trained_models.graph_validation_enabled: false` works by appending `--skipModelValidation` to the `pytorch_inference` command. Because the flag is unrecognized in production binaries, using that setting causes model deployment to fail rather than skipping validation. The documented operator escape hatch is therefore effectively non-functional outside dev/test builds.

## Root cause

The option defaults to `OFF`:

```
# cmake/variables.cmake:231
option(ML_ALLOW_SKIP_MODEL_VALIDATION "Allow --skipModelValidation on pytorch_inference (dev/test builds only)" OFF)
```

When it is `OFF`, the flag is not even registered with the argument parser (`bin/pytorch_inference/CCmdLineParser.cc`, guarded by `#ifdef ML_ALLOW_SKIP_MODEL_VALIDATION`), and the skip branch in `bin/pytorch_inference/Main.cc` is compiled out.

A repository-wide search shows the option is only ever *defined* (`cmake/variables.cmake`) and *consumed* (`bin/pytorch_inference/CMakeLists.txt`, `CCmdLineParser.cc`, `Main.cc`, and `test/test_pytorch_inference_evil_models.py`). Nothing — not `build.gradle`, the Buildkite pipelines, the Docker entrypoint, nor any toolchain file — ever sets it to `ON`. So distributed builds ship with the flag absent.

## Impact

- `boost::program_options` throws on the unrecognized option; `CCmdLineParser::parse` catches the exception and returns failure, so `pytorch_inference` exits with `EXIT_FAILURE`.
- Any deployment attempted while graph validation is disabled via the cluster setting fails to start the native process in a production build.
- The escape hatch that is documented as available for all deployment types only actually works in builds manually configured with `-DML_ALLOW_SKIP_MODEL_VALIDATION=ON`.

## Reproduction

1. Build/obtain a standard distributed `pytorch_inference` (option `OFF`, as shipped).
2. Invoke it with `--skipModelValidation` (equivalently: set `xpack.ml.trained_models.graph_validation_enabled: false` in Elasticsearch and deploy a model).
3. Observe the process fail to start with a command-line parse error instead of skipping validation.

## Expected behaviour

When an operator disables graph validation via the supported setting, the production `pytorch_inference` binary should accept `--skipModelValidation` and skip validation (with the existing WARN log), rather than failing to start.

## Suggested direction (for discussion)

Make the `--skipModelValidation` flag recognized in distributed builds so the operator setting is honored in production, while keeping the actual skip a deliberate, logged action. Decoupling "the flag is recognized" (should be always) from "validation is skipped" (runtime-controlled) would avoid the compile-out behaviour. Given this relaxes a security check, the exact approach is worth reviewing with the Security team.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia con cmake/variables.cmake e bin/pytorch_inference/CMakeLists.txt, quindi segui la gestione condizionata dei flag in CCmdLineParser.cc e Main.cc. Controlla test/test_pytorch_inference_evil_models.py e la configurazione della build distribuita. Il lavoro è completato quando un binario distribuito standard accetta --skipModelValidation, salta la validazione con l’avviso esistente e le implicazioni di sicurezza sono state esaminate.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
cmake, cpp
Ambito
backend, build-system, cli, security
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Tranquilla
Chiarezza
Abbastanza chiara
Idoneità per principianti
45/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.