ATLAS-5317: Resilient bulk purge with pre-validation, enriched respon…

[sheetalshah1007]

…ses, worker batches, and audit APIs

Attachments

1. Design specification:ATLAS-5317-Resilient-Bulk-Purge-In-Atlas-DESIGN.pdf`
2. Implementation notes: ATLAS-5317-Resilient-Bulk-Purge-In-Atlas-IMPLEMENTATION_NOTES.md

---

Summary

This PR implements resilient bulk purge according to the Resilient Bulk Purge in Atlas Using Transaction Batching design spec. It replaces the old, single-transaction REST bulk purge with a pre-validated, worker-batch processing model that is now shared by both REST and scheduled (cron) purges.

---

What changes were proposed in this pull request?

Key changes

Pre-validation before writes — REST purge GUIDs are verified (UUID format, graph existence, DELETED state, and registered type) before any mutations occur. Invalid or ineligible GUIDs are returned in failedEntities with an errorCode and errorMessage.

Request size limit — New config atlas.purge.api.max.request.size (default: 1000) limits GUIDs per REST purge request.

Shared worker-batch path — Both REST and cron use PurgeService.executePurgeWithWorkers → PurgeBatchOrchestrator / WorkItemManager, configured via atlas.purge.worker.batch.size (default: 100) and atlas.purge.workers.count (default: 2).

Isolated batch transactions — Each worker batch runs in its own graph transaction. Lock conflicts retry at batch level (500 ms, 1000 ms, 1500 ms; up to 3 attempts). Other batch failures roll back only that batch; processing continues.

Partial success — Earlier committed batches are preserved when a later batch fails permanently.

Enriched response — EntityMutationResponse adds failedEntities and PurgeSummary (purgedDependenciesCount, failedDependenciesCount, executionFailed, runId).

Two-tier purge audit with runId correlation — Each purge run (REST or cron) writes:

• One summary __AtlasAuditEntry (requested GUIDs in params, PurgeSummary JSON in result)
• Batch rows per worker batch, correlated by indexed runId
• Per-GUID failures to purgefailure.log ([PURGE_FAILURE] runId=...)

Default POST /admin/audits for PURGE / AUTO_PURGE returns summary rows only. A runId eq filter returns all rows for a run.

Admin audit read APIs — New endpoints for drill-down without loading full batch payloads in the default list:

• GET /admin/audit/{summaryGuid}/purgedEntities — paginated purged GUIDs for the run
• GET /admin/audit/{summaryGuid}/batches — batch audit entry GUIDs for the run
• GET /admin/audit/{batchGuid}/details — per-batch entity drill-down (unchanged)

Cron alignment — Scheduled purge keeps its worker-batch model; REST is aligned to the same path with shared orchestration, enriched accounting, batch retry, and shutdown reconciliation. AdminResource no longer writes one monolithic AUTO_PURGE audit row per run.

Parallel-batch hardening

• DeleteHandlerV1 tolerates vertices/edges already removed by other workers (ATLAS-4766)
• accumulateDeletionCandidates() returns Set<String> (GUIDs) for worker-safe enqueueing

Schema updates

• AUTO_PURGE added to atlas_operation in 0010-base_model.json (fresh installs)
• Indexed optional runId on __AtlasAuditEntry (typedef patch 009-base_model_add_audit_runid.json)

Compatibility and behavior notes

• accumulateDeletionCandidates() now returns Set<String> instead of Set<AtlasVertex>. Custom AtlasEntityStore implementations must update the method signature (Apache Atlas ships only AtlasEntityStoreV2).
• Missing or ineligible GUIDs appear in failedEntities on REST (no longer silently ignored).
• Completed purge responses return HTTP 200 with outcome in the body — read summary and failedEntities (design HTTP 207/400/500 for some outcomes is not used).
• HTTP 400 still applies for empty input, request too large, auth failure, or invalid GUID length.
• REST and cron may run concurrently; back-to-back cron runs are serialized via cronPurgeOperationLock.

See attached implementation notes for full flow, design deviations, configuration, and review focus areas.

---

How was this patch tested?

Unit and integration tests added or extended:

Area	Tests
End-to-end purge	PurgeTest — REST/cron purge, audit write/read, REST+cron overlap
Worker orchestration	PurgeBatchOrchestratorTest, PurgeBatchExecutorTest — workers, retry, reconciliation, batch audit
Graph layer	AtlasEntityStoreV2Test — pre-validation, batch primitives, concurrency
REST boundary	AdminResourceTest — size/auth guards, audit list filtering, /purgedEntities, /batches
Client / parallel batches	AtlasClientV2Test, DeleteHandlerV1Test

Replaced AdminPurgeTest and PurgeServiceTest. Manual QE suggested for large partial-failure runs, REST+cron overlap, and audit drill-down by runId.

---

Performance testing

Load: 1,000 hive_table entities (2 columns each) — 5,000 entities total (tables + columns).

Config	Avg purge_ms	Time	vs pre-5317
Pre-5317 (single transaction)	273,318	4m 33s	baseline
Post-5317, workers.count=2	~58,541	~59s	~79% faster (~4.7×)
Post-5317, workers.count=1	~109,471	~1m 49s	~60% faster (~2.5×)

Regarding atlas.purge.workers.count

Default = 2 — Cron purge historically used 2 workers; the new design keeps that. One batch can commit while another runs — solid speed-up without overwhelming the database.

Setting	Behavior	When to use
2 (default)	Two parallel batches	Regular large purges; best balance of speed and load on this test
1	Sequential (one batch at a time)	Prefer legacy-like behavior; still ~60% faster than pre-5317
> 2 (e.g. 5)	More parallel batches	Only after cluster testing — may not scale linearly

Notes:

• Most of the speedup comes from smaller batches, not from extra workers alone.
• Higher worker counts increase graph lock contention and backend load; REST and cron can both purge concurrently.
• Timings are environment-specific; tune workers.count for your cluster.