[ML] Fix PyTorch Docker builds: copy libtorch from installed package

[edsavage]

Summary

The nightly ml-cpp-pytorch-builds image build fails at the post-install header copy:

cp: cannot stat 'torch/include/*': No such file or directory

(build 765)

Root cause

The build itself succeeds — the failure is the artifact copy afterwards. Recent PyTorch turns python setup.py install into a redirect to pip install . --no-build-isolation (see pytorch/pytorch#152276):

WARNING: Redirecting 'python setup.py install' to 'pip install . -v --no-build-isolation'

A wheel install places the built torch package (headers + libs) under Python site-packages and no longer leaves torch/include populated in the source checkout. The Dockerfiles then copy from the stale source-tree path torch/include / torch/lib and fail.

This bit the nightly first because it tracks viable/strict. The pinned v2.7.1 images (linux_image, linux_aarch64_native_image) still build in-tree today, so they pass — but they will break the same way when PyTorch is next bumped.

Fix

Resolve the installed torch package directory via sysconfig and copy the headers/libraries from there. This works whether the build is in-tree (older tags) or a wheel install (newer):

TORCH_DIR=$(python3.12 -c "import sysconfig, os; print(os.path.join(sysconfig.get_paths()['platlib'], 'torch'))")
cp -r "${TORCH_DIR}"/include/* ...
cp    "${TORCH_DIR}"/lib/libtorch_cpu.so ...
cp    "${TORCH_DIR}"/lib/libc10.so ...

Applied to all three Linux images (nightly + both pinned-release builds) for consistency and future-proofing.

Test plan

• Verified the failure in build 765 is the source-tree torch/include copy after the setup.py install -> pip install . redirect
• Confirmed BuildKit strips the inline # comment lines from the continued RUN (chained && stays intact) via a local docker build
• Nightly ml-cpp-pytorch-builds produces a working image (headers + libtorch present under /usr/local/gcc133) — validated non-destructively against a real installed torch (see comment below)
• Pinned v2.7.1 linux / aarch64 images still build and are unchanged in content — validated non-destructively on both arches (see comment below)

Made with Cursor

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[Copilot]

Pull request overview

This PR fixes PyTorch Docker image builds by copying LibTorch headers and shared libraries from the installed torch Python package location (site-packages), rather than from the PyTorch source tree, which is no longer populated reliably after setup.py install redirects to pip install ..

Changes:

• Resolve the installed torch package directory via Python sysconfig and store it in TORCH_DIR.
• Copy headers from ${TORCH_DIR}/include and shared libs from ${TORCH_DIR}/lib into /usr/local/gcc133 for all Linux PyTorch build images.
• Apply the same approach consistently across the nightly PyTorch image and both pinned v2.7.1 Linux images.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
dev-tools/docker/pytorch_linux_image/Dockerfile	Copy headers/libs from installed torch package directory instead of the source tree to prevent post-install copy failures.
dev-tools/docker/linux_image/Dockerfile	Same installed-package copy approach for the pinned Linux build image.
dev-tools/docker/linux_aarch64_native_image/Dockerfile	Same installed-package copy approach for the pinned aarch64 native build image.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[edsavage]

Test plan item 1 — non-destructive validation

The nightly ml-cpp-pytorch-builds pipeline pushes to the shared tag docker.elastic.co/ml-dev/ml-linux-dependency-build:pytorch_latest (via build_pytorch_linux_build_image.sh, x86_64-only, no build-only flag), so a real pre-merge run would overwrite production infra. Instead I validated the exact changed logic against a real installed torch package, which is layout-equivalent to what setup.py install → pip install . produces in site-packages.

Ran the fixed Dockerfile's TORCH_DIR resolution + cp commands verbatim in python:3.12-slim (linux/amd64, matching the nightly's x86_64 / python3.12):

python: Python 3.12.13  arch: x86_64
torch version: 2.13.0+cpu
TORCH_DIR (sysconfig platlib/torch): /usr/local/lib/python3.12/site-packages/torch
torch.__file__ dir (sanity):         /usr/local/lib/python3.12/site-packages/torch
include/ dir exists (9431 files)
-rwxr-xr-x libc10.so           (1,525,128 bytes)
-rwxr-xr-x libtorch_cpu.so   (434,184,800 bytes)
# after the exact `cp -rf "${TORCH_DIR}"/include/* ...` and lib copies:
header present: torch/script.h
header present: ATen/ATen.h
header present: c10/core/Device.h
/usr/local/gcc133/lib/libc10.so          present
/usr/local/gcc133/lib/libtorch_cpu.so    present
VALIDATION OK

This confirms the fix's core assumption holds against a modern (wheel-installed) torch: sysconfig.get_paths()['platlib']/torch resolves to the installed package, and include/, lib/libtorch_cpu.so, lib/libc10.so are all present and copy cleanly into /usr/local/gcc133.

End-to-end confirmation of the full image (source build of viable/strict) will come from the first post-merge nightly, which I'll watch.

[edsavage]

Test plan item 2 — non-destructive validation (pinned v2.7.1, both arches)

Both pinned Dockerfiles (linux_image, linux_aarch64_native_image) build torch v2.7.1 with python3.12 and use the identical TORCH_DIR logic. Validated the exact resolution + cp commands against a real torch==2.7.1 install on both arches:

linux_image — linux/amd64:

python: 3.12.13  arch: x86_64   torch: 2.7.1+cpu
TORCH_DIR: /usr/local/lib/python3.12/site-packages/torch  (== torch.__file__ dir)
include/ (8765 files); libc10.so (1,363,872 B); libtorch_cpu.so (418,117,016 B)
-> copied: torch/script.h, ATen/ATen.h, c10/core/Device.h + both libs under /usr/local/gcc133 — VALIDATION OK

linux_aarch64_native_image — linux/arm64:

python: 3.12.13  arch: aarch64  torch: 2.7.1+cpu
TORCH_DIR: /usr/local/lib/python3.12/site-packages/torch  (== torch.__file__ dir)
include/ (8765 files); libc10.so (1,382,416 B); libtorch_cpu.so (249,487,721 B)
-> copied: torch/script.h, ATen/ATen.h, c10/core/Device.h + both libs under /usr/local/gcc133 — VALIDATION OK

On "unchanged in content": the pinned images build v2.7.1 from source and setup.py install installs that same built package into site-packages — the exact directory the fix now copies from. The header set (include/, 8765 files) and the two libs (libtorch_cpu.so, libc10.so) match what the previous in-tree copy pulled from torch/include / torch/lib, so the copied content is equivalent. The change only relocates the source path of the copy, not the artifacts.