Fix SSHRemoteJobOperator cleanup failing with custom remote_base_dir#69885
Open
harshhh817 wants to merge 1 commit into
Open
Fix SSHRemoteJobOperator cleanup failing with custom remote_base_dir#69885harshhh817 wants to merge 1 commit into
harshhh817 wants to merge 1 commit into
Conversation
The cleanup command builders validated the job directory against the hardcoded default base directory, so any job started with a custom remote_base_dir ran to completion and then failed its cleanup step with "Invalid job directory ... Expected path under '/tmp/airflow-ssh-jobs'". Thread the operator's configured remote_base_dir through build_posix_cleanup_command / build_windows_cleanup_command into _validate_job_dir, falling back to the OS default when unset, so the safety check still constrains the rm target to the directory the job was actually created under. Signed-off-by: Harsh Gupta <harshgupta93198@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes
SSHRemoteJobOperatorfailing at the cleanup step whenever a customremote_base_diris used.The operator accepts (and validates) a custom
remote_base_dirat construction time and creates the job directory under it, butbuild_posix_cleanup_command/build_windows_cleanup_commandvalidated the job directory against the hardcoded default base directory. Result: the remote job runs to completion, then the task fails withFix: thread the operator's configured
remote_base_dirthrough the cleanup builders into_validate_job_dir, falling back to the OS-specific default when unset. The safety property is preserved — therm -rftarget must still live under the base directory the job was actually created under (which was itself validated by_validate_base_dirat operator construction). Both the sync and deferrable (execute_complete) cleanup paths go through the fixed_cleanup_remote_job.Tests: new unit tests for custom-base-dir acceptance and rejection (posix + windows), plus an operator-level regression test reproducing the exact failure from the issue — verified it fails on
mainand passes with this change. Full ssh provider unit suite: 49 passed.closes: #69813
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code following the guidelines
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.