[ML] --skipModelValidation is compiled out of distributed builds, so graph-validation cannot be disabled in production
- Dominant language
- C++
- Stars
- 157
- Forks
- 67
- Avg merge
- 12h 48m
- Merged PRs (30d)
- 16
Description
## Summary
The `--skipModelValidation` command-line flag on `pytorch_inference` is guarded by the `ML_ALLOW_SKIP_MODEL_VALIDATION` CMake option, which defaults to `OFF` and is **not enabled by any build in this repository** — including the distributed/release build. As a result, shipped `pytorch_inference` binaries do not recognize `--skipModelValidation`, and passing it makes the process **fail to start**.
The Elasticsearch cluster setting `xpack.ml.trained_models.graph_validation_enabled: false` works by appending `--skipModelValidation` to the `pytorch_inference` command. Because the flag is unrecognized in production binaries, using that setting causes model deployment to fail rather than skipping validation. The documented operator escape hatch is therefore effectively non-functional outside dev/test builds.
## Root cause
The option defaults to `OFF`:
```
# cmake/variables.cmake:231
option(ML_ALLOW_SKIP_MODEL_VALIDATION "Allow --skipModelValidation on pytorch_inference (dev/test builds only)" OFF)
```
When it is `OFF`, the flag is not even registered with the argument parser (`bin/pytorch_inference/CCmdLineParser.cc`, guarded by `#ifdef ML_ALLOW_SKIP_MODEL_VALIDATION`), and the skip branch in `bin/pytorch_inference/Main.cc` is compiled out.
A repository-wide search shows the option is only ever *defined* (`cmake/variables.cmake`) and *consumed* (`bin/pytorch_inference/CMakeLists.txt`, `CCmdLineParser.cc`, `Main.cc`, and `test/test_pytorch_inference_evil_models.py`). Nothing — not `build.gradle`, the Buildkite pipelines, the Docker entrypoint, nor any toolchain file — ever sets it to `ON`. So distributed builds ship with the flag absent.
## Impact
- `boost::program_options` throws on the unrecognized option; `CCmdLineParser::parse` catches the exception and returns failure, so `pytorch_inference` exits with `EXIT_FAILURE`.
- Any deployment attempted while graph validation is disabled via the cluster setting fails to start the native process in a production build.
- The escape hatch that is documented as available for all deployment types only actually works in builds manually configured with `-DML_ALLOW_SKIP_MODEL_VALIDATION=ON`.
## Reproduction
1. Build/obtain a standard distributed `pytorch_inference` (option `OFF`, as shipped).
2. Invoke it with `--skipModelValidation` (equivalently: set `xpack.ml.trained_models.graph_validation_enabled: false` in Elasticsearch and deploy a model).
3. Observe the process fail to start with a command-line parse error instead of skipping validation.
## Expected behaviour
When an operator disables graph validation via the supported setting, the production `pytorch_inference` binary should accept `--skipModelValidation` and skip validation (with the existing WARN log), rather than failing to start.
## Suggested direction (for discussion)
Make the `--skipModelValidation` flag recognized in distributed builds so the operator setting is honored in production, while keeping the actual skip a deliberate, logged action. Decoupling "the flag is recognized" (should be always) from "validation is skipped" (runtime-controlled) would avoid the compile-out behaviour. Given this relaxes a security check, the exact approach is worth reviewing with the Security team.
Contributor guide
Assessment
This issue has not been assessed yet.