[ML] --skipModelValidation is compiled out of distributed builds, so graph-validation cannot be disabled in production
- Vorherrschende Sprache
- C++
- Sterne
- 157
- Forks
- 67
- Ø Merge
- 17 Std. 52 Min.
- Gemergte PRs (30 T.)
- 20
Beschreibung
## Summary
The `--skipModelValidation` command-line flag on `pytorch_inference` is guarded by the `ML_ALLOW_SKIP_MODEL_VALIDATION` CMake option, which defaults to `OFF` and is **not enabled by any build in this repository** — including the distributed/release build. As a result, shipped `pytorch_inference` binaries do not recognize `--skipModelValidation`, and passing it makes the process **fail to start**.
The Elasticsearch cluster setting `xpack.ml.trained_models.graph_validation_enabled: false` works by appending `--skipModelValidation` to the `pytorch_inference` command. Because the flag is unrecognized in production binaries, using that setting causes model deployment to fail rather than skipping validation. The documented operator escape hatch is therefore effectively non-functional outside dev/test builds.
## Root cause
The option defaults to `OFF`:
```
# cmake/variables.cmake:231
option(ML_ALLOW_SKIP_MODEL_VALIDATION "Allow --skipModelValidation on pytorch_inference (dev/test builds only)" OFF)
```
When it is `OFF`, the flag is not even registered with the argument parser (`bin/pytorch_inference/CCmdLineParser.cc`, guarded by `#ifdef ML_ALLOW_SKIP_MODEL_VALIDATION`), and the skip branch in `bin/pytorch_inference/Main.cc` is compiled out.
A repository-wide search shows the option is only ever *defined* (`cmake/variables.cmake`) and *consumed* (`bin/pytorch_inference/CMakeLists.txt`, `CCmdLineParser.cc`, `Main.cc`, and `test/test_pytorch_inference_evil_models.py`). Nothing — not `build.gradle`, the Buildkite pipelines, the Docker entrypoint, nor any toolchain file — ever sets it to `ON`. So distributed builds ship with the flag absent.
## Impact
- `boost::program_options` throws on the unrecognized option; `CCmdLineParser::parse` catches the exception and returns failure, so `pytorch_inference` exits with `EXIT_FAILURE`.
- Any deployment attempted while graph validation is disabled via the cluster setting fails to start the native process in a production build.
- The escape hatch that is documented as available for all deployment types only actually works in builds manually configured with `-DML_ALLOW_SKIP_MODEL_VALIDATION=ON`.
## Reproduction
1. Build/obtain a standard distributed `pytorch_inference` (option `OFF`, as shipped).
2. Invoke it with `--skipModelValidation` (equivalently: set `xpack.ml.trained_models.graph_validation_enabled: false` in Elasticsearch and deploy a model).
3. Observe the process fail to start with a command-line parse error instead of skipping validation.
## Expected behaviour
When an operator disables graph validation via the supported setting, the production `pytorch_inference` binary should accept `--skipModelValidation` and skip validation (with the existing WARN log), rather than failing to start.
## Suggested direction (for discussion)
Make the `--skipModelValidation` flag recognized in distributed builds so the operator setting is honored in production, while keeping the actual skip a deliberate, logged action. Decoupling "the flag is recognized" (should be always) from "validation is skipped" (runtime-controlled) would avoid the compile-out behaviour. Given this relaxes a security check, the exact approach is worth reviewing with the Security team.
Beitragsleitfaden
Rechercherichtung
Beginne mit cmake/variables.cmake und bin/pytorch_inference/CMakeLists.txt und verfolge dann die bedingte Flag-Verarbeitung in CCmdLineParser.cc und Main.cc. Prüfe test/test_pytorch_inference_evil_models.py und die Konfiguration des verteilten Builds. Erledigt bedeutet, dass eine standardmäßige verteilte Binärdatei --skipModelValidation akzeptiert, die Validierung mit der vorhandenen Warnung überspringt und die Sicherheitsauswirkungen geprüft wurden.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- cmake, cpp
- Bereich
- backend, build-system, cli, security
- Issue-Typ
- Bug
- Schwierigkeit
- 4/5
- Geschätzter Aufwand
- 3-5 Tage
- Aktivitätsstatus
- Ruhig
- Klarheit
- Größtenteils klar
- Anfängerfreundlichkeit
- 45/100