Fix - NotAuthenticated errors after service disruptions

[kolluria]

Summary

#3787 introduced retry mechanism to handle container restarts during WCP password rotation. But, the cleanup routine that this PR introduced violates an undocumented assumption that Client of the virtual center should not be nullified.
#3864 reverted this broken password rotation change.

Root Cause

The cleanupVCClient() function was nullifying vc.Client during error recovery. This broke the intended reconnection flow:

1. After service disruption, cleanupVCClient() nullified vc.Client
2. Next connect() call created a new client with a valid session
3. Session validity check passed → early return at line 393
4. Dependent client recreation code (lines 420-451) never reached
5. Dependent clients (VsanClient, CnsClient, etc.) retained stale session references
6. Operations using dependent clients failed with NotAuthenticated

This PR cherry-picks the password rotation back to main with the correct cleanup routine on top. Fixes #3787.

Testing Done

Test Summary

Validated fix across service disruption and password rotation scenarios:

Test	Scenario	Expected Behavior	Result
Bug Reproduction	vpxd restart with buggy code	NotAuthenticated on VsanClient	PASS
Fix Validation	vpxd restart with fix	All operations succeed(new clients created)	PASS
Password Rotation 1	WCP down + secret change + restart	3-min backoff → success	PASS
Password Rotation 2	Secret change + vpxd restart(service disruption)	3-min backoff → success	PASS

Test Scenarios

Bug Reproduction (Buggy Code)

Setup: Deployed buggy code, enabled DEBUG logging, restarted vpxd

Timeline:

08:02:10 - New vCenter session created successfully
08:03:40 - Auth manager refresh triggered
08:03:40 - HasUserPrivilegeOnEntities: SUCCESS (main SDK works)
08:03:40 - VsanClusterGetConfig: FAILED NotAuthenticated (VsanClient fails)

Result: Successfully reproduced exact customer scenario

Fix Validation (Fixed Code)

Setup: Deployed fix, enabled DEBUG logging, restarted vpxd

Timeline:

08:49:06 - New vCenter session created successfully  
08:50:39 - Auth manager refresh triggered
08:50:39 - HasUserPrivilegeOnEntities: SUCCESS (main SDK works)
08:50:39 - vsan cluster config: SUCCESS (no NotAuthenticated errors)

Result: No NotAuthenticated errors after service disruption

Password Rotation Test 1: WCP Service Down

Scenario: Simulate password rotation with WCP service maintenance

Steps:

1. Stopped WCP service
2. Updated secret with wrong password
3. Restarted CSI deployment
4. Started WCP service (auto-updated secret)
5. Observed backoff and retry

Timeline:

10:45:35 - InvalidLogin detected, 3-minute backoff started
10:47:58 - WCP auto-updated secret with correct password
10:48:35 - Retry after backoff → SUCCESS

Result: Automatic recovery after backoff

Password Rotation Test 2: vpxd Restart

Scenario: Simulate password rotation with service restart (no pod restart)

Steps:

1. Updated secret with wrong password (pod still running)
2. Restarted vpxd (forced re-authentication)
3. Observed InvalidLogin and backoff
4. Restored correct password
5. Observed successful connection after backoff

Timeline:

10:52:04 - InvalidLogin detected, 3-minute backoff started
10:53:34 - Secret restored with correct password
10:55:07 - Retry after backoff → SUCCESS

Result: No pod restart needed, automatic recovery

Precheckin runs

WCP

Ran 16 of 2324 Specs
SUCCESS! -- 16 Passed | 0 Failed | 0 Pending | 2308 Skipped | 0 Flaked

VKS

Ran 5 of 2324 Specs
SUCCESS! -- 5 Passed | 0 Failed | 0 Pending | 2319 Skipped | 0 Flaked

Vanilla

Special Notes for Reviewer

• The existing vc.connect code had an implicit assumption that vc.Client should persist across reconnection attempts, but this was never documented
• This fix makes the implicit assumption explicit through code comments
• This pattern is critical for proper session management but was not obvious from the code

Release Note

Fix NotAuthenticated errors on dependent client operations after vCenter service disruptions by correcting error recovery logic

[k8s-ci-robot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kolluria

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [kolluria]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[kolluria]

/cc @divyenpatel @deepakkinni @xing-yang
/assign @kolluria

[deepakkinni]

FAILED --- Jenkins Build #918

[deepakkinni]

SUCCESS --- Jenkins Build #821

[deepakkinni]

FAILED --- Jenkins Build #919

[deepakkinni]

FAILED --- Jenkins Build #920

[deepakkinni]

Triggering CSI-WCP Pre-checkin Pipeline for this PR... Job takes approximately an hour to complete
Jenkins Build #948

[deepakkinni]

Triggering CSI-WCP Pre-checkin Pipeline for this PR... Job takes approximately an hour to complete
Jenkins Build #947

[deepakkinni]

Triggering CSI-WCP Pre-checkin Pipeline for this PR... Job takes approximately an hour to complete
Jenkins Build #950

[deepakkinni]

FAILED --- Jenkins Build #950

[deepakkinni]

Triggering CSI-WCP Pre-checkin Pipeline for this PR... Job takes approximately an hour to complete
Jenkins Build #951

[deepakkinni]

SUCCESS --- Jenkins Build #951

[deepakkinni]

FAILED --- Jenkins Build #838

[deepakkinni]

Triggering CSI-TKG Pre-checkin Pipeline for this PR... Job takes approximately an hour to complete
Jenkins Build #839