elastic / elastic/ml-cpp

[ML] --skipModelValidation is compiled out of distributed builds, so graph-validation cannot be disabled in production

Offen
#3,098 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
C++
Sterne
157
Forks
67
Ø Merge
17 Std. 52 Min.
Gemergte PRs (30 T.)
20

Beschreibung

## Summary

The `--skipModelValidation` command-line flag on `pytorch_inference` is guarded by the `ML_ALLOW_SKIP_MODEL_VALIDATION` CMake option, which defaults to `OFF` and is **not enabled by any build in this repository** — including the distributed/release build. As a result, shipped `pytorch_inference` binaries do not recognize `--skipModelValidation`, and passing it makes the process **fail to start**.

The Elasticsearch cluster setting `xpack.ml.trained_models.graph_validation_enabled: false` works by appending `--skipModelValidation` to the `pytorch_inference` command. Because the flag is unrecognized in production binaries, using that setting causes model deployment to fail rather than skipping validation. The documented operator escape hatch is therefore effectively non-functional outside dev/test builds.

## Root cause

The option defaults to `OFF`:

```
# cmake/variables.cmake:231
option(ML_ALLOW_SKIP_MODEL_VALIDATION "Allow --skipModelValidation on pytorch_inference (dev/test builds only)" OFF)
```

When it is `OFF`, the flag is not even registered with the argument parser (`bin/pytorch_inference/CCmdLineParser.cc`, guarded by `#ifdef ML_ALLOW_SKIP_MODEL_VALIDATION`), and the skip branch in `bin/pytorch_inference/Main.cc` is compiled out.

A repository-wide search shows the option is only ever *defined* (`cmake/variables.cmake`) and *consumed* (`bin/pytorch_inference/CMakeLists.txt`, `CCmdLineParser.cc`, `Main.cc`, and `test/test_pytorch_inference_evil_models.py`). Nothing — not `build.gradle`, the Buildkite pipelines, the Docker entrypoint, nor any toolchain file — ever sets it to `ON`. So distributed builds ship with the flag absent.

## Impact

- `boost::program_options` throws on the unrecognized option; `CCmdLineParser::parse` catches the exception and returns failure, so `pytorch_inference` exits with `EXIT_FAILURE`.
- Any deployment attempted while graph validation is disabled via the cluster setting fails to start the native process in a production build.
- The escape hatch that is documented as available for all deployment types only actually works in builds manually configured with `-DML_ALLOW_SKIP_MODEL_VALIDATION=ON`.

## Reproduction

1. Build/obtain a standard distributed `pytorch_inference` (option `OFF`, as shipped).
2. Invoke it with `--skipModelValidation` (equivalently: set `xpack.ml.trained_models.graph_validation_enabled: false` in Elasticsearch and deploy a model).
3. Observe the process fail to start with a command-line parse error instead of skipping validation.

## Expected behaviour

When an operator disables graph validation via the supported setting, the production `pytorch_inference` binary should accept `--skipModelValidation` and skip validation (with the existing WARN log), rather than failing to start.

## Suggested direction (for discussion)

Make the `--skipModelValidation` flag recognized in distributed builds so the operator setting is honored in production, while keeping the actual skip a deliberate, logged action. Decoupling "the flag is recognized" (should be always) from "validation is skipped" (runtime-controlled) would avoid the compile-out behaviour. Given this relaxes a security check, the exact approach is worth reviewing with the Security team.

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Beginne mit cmake/variables.cmake und bin/pytorch_inference/CMakeLists.txt und verfolge dann die bedingte Flag-Verarbeitung in CCmdLineParser.cc und Main.cc. Prüfe test/test_pytorch_inference_evil_models.py und die Konfiguration des verteilten Builds. Erledigt bedeutet, dass eine standardmäßige verteilte Binärdatei --skipModelValidation akzeptiert, die Validierung mit der vorhandenen Warnung überspringt und die Sicherheitsauswirkungen geprüft wurden.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
cmake, cpp
Bereich
backend, build-system, cli, security
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Ruhig
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
45/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.