elastic / elastic/ml-cpp

[ML] --skipModelValidation is compiled out of distributed builds, so graph-validation cannot be disabled in production

Open
#3,098 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
157
Forks
67
Avg merge
12h 48m
Merged PRs (30d)
16

Description

## Summary

The `--skipModelValidation` command-line flag on `pytorch_inference` is guarded by the `ML_ALLOW_SKIP_MODEL_VALIDATION` CMake option, which defaults to `OFF` and is **not enabled by any build in this repository** — including the distributed/release build. As a result, shipped `pytorch_inference` binaries do not recognize `--skipModelValidation`, and passing it makes the process **fail to start**.

The Elasticsearch cluster setting `xpack.ml.trained_models.graph_validation_enabled: false` works by appending `--skipModelValidation` to the `pytorch_inference` command. Because the flag is unrecognized in production binaries, using that setting causes model deployment to fail rather than skipping validation. The documented operator escape hatch is therefore effectively non-functional outside dev/test builds.

## Root cause

The option defaults to `OFF`:

```
# cmake/variables.cmake:231
option(ML_ALLOW_SKIP_MODEL_VALIDATION "Allow --skipModelValidation on pytorch_inference (dev/test builds only)" OFF)
```

When it is `OFF`, the flag is not even registered with the argument parser (`bin/pytorch_inference/CCmdLineParser.cc`, guarded by `#ifdef ML_ALLOW_SKIP_MODEL_VALIDATION`), and the skip branch in `bin/pytorch_inference/Main.cc` is compiled out.

A repository-wide search shows the option is only ever *defined* (`cmake/variables.cmake`) and *consumed* (`bin/pytorch_inference/CMakeLists.txt`, `CCmdLineParser.cc`, `Main.cc`, and `test/test_pytorch_inference_evil_models.py`). Nothing — not `build.gradle`, the Buildkite pipelines, the Docker entrypoint, nor any toolchain file — ever sets it to `ON`. So distributed builds ship with the flag absent.

## Impact

- `boost::program_options` throws on the unrecognized option; `CCmdLineParser::parse` catches the exception and returns failure, so `pytorch_inference` exits with `EXIT_FAILURE`.
- Any deployment attempted while graph validation is disabled via the cluster setting fails to start the native process in a production build.
- The escape hatch that is documented as available for all deployment types only actually works in builds manually configured with `-DML_ALLOW_SKIP_MODEL_VALIDATION=ON`.

## Reproduction

1. Build/obtain a standard distributed `pytorch_inference` (option `OFF`, as shipped).
2. Invoke it with `--skipModelValidation` (equivalently: set `xpack.ml.trained_models.graph_validation_enabled: false` in Elasticsearch and deploy a model).
3. Observe the process fail to start with a command-line parse error instead of skipping validation.

## Expected behaviour

When an operator disables graph validation via the supported setting, the production `pytorch_inference` binary should accept `--skipModelValidation` and skip validation (with the existing WARN log), rather than failing to start.

## Suggested direction (for discussion)

Make the `--skipModelValidation` flag recognized in distributed builds so the operator setting is honored in production, while keeping the actual skip a deliberate, logged action. Decoupling "the flag is recognized" (should be always) from "validation is skipped" (runtime-controlled) would avoid the compile-out behaviour. Given this relaxes a security check, the exact approach is worth reviewing with the Security team.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.