[ML] --skipModelValidation is compiled out of distributed builds, so graph-validation cannot be disabled in production

[valeriy42]

Summary

The --skipModelValidation command-line flag on pytorch_inference is guarded by the ML_ALLOW_SKIP_MODEL_VALIDATION CMake option, which defaults to OFF and is not enabled by any build in this repository — including the distributed/release build. As a result, shipped pytorch_inference binaries do not recognize --skipModelValidation, and passing it makes the process fail to start.

The Elasticsearch cluster setting xpack.ml.trained_models.graph_validation_enabled: false works by appending --skipModelValidation to the pytorch_inference command. Because the flag is unrecognized in production binaries, using that setting causes model deployment to fail rather than skipping validation. The documented operator escape hatch is therefore effectively non-functional outside dev/test builds.

Root cause

The option defaults to OFF:

# cmake/variables.cmake:231
option(ML_ALLOW_SKIP_MODEL_VALIDATION "Allow --skipModelValidation on pytorch_inference (dev/test builds only)" OFF)

When it is OFF, the flag is not even registered with the argument parser (bin/pytorch_inference/CCmdLineParser.cc, guarded by #ifdef ML_ALLOW_SKIP_MODEL_VALIDATION), and the skip branch in bin/pytorch_inference/Main.cc is compiled out.

A repository-wide search shows the option is only ever defined (cmake/variables.cmake) and consumed (bin/pytorch_inference/CMakeLists.txt, CCmdLineParser.cc, Main.cc, and test/test_pytorch_inference_evil_models.py). Nothing — not build.gradle, the Buildkite pipelines, the Docker entrypoint, nor any toolchain file — ever sets it to ON. So distributed builds ship with the flag absent.

Impact

• boost::program_options throws on the unrecognized option; CCmdLineParser::parse catches the exception and returns failure, so pytorch_inference exits with EXIT_FAILURE.
• Any deployment attempted while graph validation is disabled via the cluster setting fails to start the native process in a production build.
• The escape hatch that is documented as available for all deployment types only actually works in builds manually configured with -DML_ALLOW_SKIP_MODEL_VALIDATION=ON.

Reproduction

1. Build/obtain a standard distributed pytorch_inference (option OFF, as shipped).
2. Invoke it with --skipModelValidation (equivalently: set xpack.ml.trained_models.graph_validation_enabled: false in Elasticsearch and deploy a model).
3. Observe the process fail to start with a command-line parse error instead of skipping validation.

Expected behaviour

When an operator disables graph validation via the supported setting, the production pytorch_inference binary should accept --skipModelValidation and skip validation (with the existing WARN log), rather than failing to start.

Suggested direction (for discussion)

Make the --skipModelValidation flag recognized in distributed builds so the operator setting is honored in production, while keeping the actual skip a deliberate, logged action. Decoupling "the flag is recognized" (should be always) from "validation is skipped" (runtime-controlled) would avoid the compile-out behaviour. Given this relaxes a security check, the exact approach is worth reviewing with the Security team.