Skip to content

fix: Don't process the DPU CR which is being deleted (backport #4125)#4159

Open
abvarshney-nv wants to merge 1 commit into
NVIDIA:release/v2.0from
abvarshney-nv:backport/v2.0/dpu-cr-deletion
Open

fix: Don't process the DPU CR which is being deleted (backport #4125)#4159
abvarshney-nv wants to merge 1 commit into
NVIDIA:release/v2.0from
abvarshney-nv:backport/v2.0/dpu-cr-deletion

Conversation

@abvarshney-nv

@abvarshney-nv abvarshney-nv commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Backport of #4125 to release/v2.0.

Problem

When reprovisioning a DPU, reprovision_dpu deletes the DPU CR, but the DPF operator's finalizer keeps the old CR around briefly with a deletionTimestamp and a stale status.phase (often Ready, or Error). On the immediate requeue into WaitingForReady, NICO read that stale Ready/Error and jumped straight to the next state — so reprovisioning was never actually observed.

Fix

  • get_dpu_phase now maps a set deletionTimestamp to DpuPhase::Deleting.
  • handle_dpf_waiting_for_ready short-circuits to a wait on DpuPhase::Deleting (and on NotFound, the brief gap before the fresh CR is created) before any side-effecting calls.

NICO now waits for the operator to delete the old CR and create the new one, then proceeds only when the fresh CR reports Ready.

Related issues

#4125

Type of Change

  • Add - New feature or capability
  • Change - Changes in existing functionality
  • Fix - Bug fixes
  • Remove - Removed features or deprecated functionality
  • Internal - Internal changes (refactoring, tests, docs, etc.)

Breaking Changes

  • This PR contains breaking changes

Testing

  • Unit tests added/updated
  • Integration tests added/updated
  • Manual testing performed
  • No testing required (docs, internal refactor, etc.)

Additional Notes

When reprovisioning a DPU, reprovision_dpu deletes the DPU CR, but the
DPF operator's finalizer keeps the old CR around briefly with a
deletionTimestamp and a stale status.phase (specially Ready or Error).
On the immediate requeue into WaitingForReady, NICO read that stale
Ready/Error and jumped straight to next state — so reprovisioning was
never actually observed.

get_dpu_phase now maps a set `deletionTimestamp` to
`DpuPhase::Deleting`, and `handle_dpf_waiting_for_ready` short-circuits
to a wait on Deleting (and on NotFound, the brief gap before the fresh
CR is created) before any side-effecting calls. NICO now waits for the
operator to delete the old CR and create the new one, then proceeds only
when that fresh CR reports Ready.

- [ ] **Add** - New feature or capability
- [ ] **Change** - Changes in existing functionality
- [x] **Fix** - Bug fixes
- [ ] **Remove** - Removed features or deprecated functionality
- [ ] **Internal** - Internal changes (refactoring, tests, docs, etc.)

- [ ] **This PR contains breaking changes**

- [x] Unit tests added/updated
- [ ] Integration tests added/updated
- [ ] Manual testing performed
- [ ] No testing required (docs, internal refactor, etc.)

(cherry picked from commit a991005)
@abvarshney-nv
abvarshney-nv requested a review from a team July 25, 2026 04:50
@coderabbitai

coderabbitai Bot commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 3f43da66-7794-4fdf-bcd0-685ad992bffb

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown

🔐 TruffleHog Secret Scan

No secrets or credentials found!

Your code has been scanned for 700+ types of secrets and credentials. All clear! 🎉

🔗 View scan details

🕐 Last updated: 2026-07-25 04:53:23 UTC | Commit: 03ef700

@github-actions

github-actions Bot commented Jul 25, 2026

Copy link
Copy Markdown

🔍 Container Scan Summary

Service Total Critical High Medium Low Other
boot-artifacts-aarch64 3 0 0 3 0 0
boot-artifacts-x86_64 3 0 0 3 0 0
forge-admin-cli-x86_64 260 16 29 79 8 128
machine-validation-runner 548 41 117 191 18 181
machine_validation 548 41 117 191 18 181
machine_validation-aarch64 548 41 117 191 18 181
nvmetal-carbide 548 41 117 191 18 181
TOTAL 2458 180 497 849 80 852

Per-CVE detail lives in the per-service grype-* artifacts (JSON + SARIF). Severity counts only — no CVE IDs published here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant