---
id: WF-PFM-TNL-014
title: Recover a stalled deletion
trigger: Replace stopped deletion machinery under staff authority.
product_area: platform
module: tenant-lifecycle
revision: 1
status: draft
design_intent: confirmed
origin: services-platform:docs/tenants/tenant-lifecycle-plan.md#§5.8 Resume a stuck schedule, Retry a failed purge
  (approved 2026-10-09)
implementation_evidence: observed
inference: []
source_sha:
  services-platform: ad3f6a6582843008ed0f5a17b131878d15829af1
implements:
- CAP-PFM-TNL-016
- CAP-PFM-TNL-017
moves:
- STATE-PFM-TNL-004
crosses:
- SVC-PFM-TNL-tenants
idempotency: Open watcher checks and conditional purge repointing restrict recovery. Separate job creation can still
  leave losing records under concurrency.
concurrency: Recovery creates a job before repointing the request in a later commit. Conditional refusal can leave
  a losing job, while task ownership checks restrict stale execution.
hops:
- n: 1
  actor_or_service: Platform API (resumeTenantDeletionSchedule)
  action: Authorize schedule recovery without accepting a reason or reviewed version.
  anchor: services-platform:modules/tenants/aws/api/tenant-deletion-requests.routes.ts:resumeTenantDeletionSchedule
  seam: sync
  on_failure: Refuse missing staff authority.
  left_inconsistent: No recovery job is intentionally started.
  observer: Job progress and service logs.
  provenance: observed
  source: services-platform:modules/tenants/aws/api/tenant-deletion-requests.routes.ts
- n: 2
  actor_or_service: Operator recovery
  action: Require a scheduled or held request with no open watcher, then create a watcher, repoint the request and
    start it.
  anchor: services-platform:modules/tenants/services/deletion/operator-recovery.ts:resumeTenantDeletionSchedule
  seam: state_machine
  on_failure: Return errors; shared recovery handles unstarted committed jobs.
  left_inconsistent: Separate commits can leave an unreferenced watcher.
  observer: Job progress and service logs.
  provenance: observed
  source: services-platform:modules/tenants/services/deletion/operator-recovery.ts
- n: 3
  actor_or_service: Platform API (retryTenantPurge)
  action: Authorize purge retry on its bodyless staff route.
  anchor: services-platform:modules/tenants/aws/api/tenant-deletion-requests.routes.ts:retryTenantPurge
  seam: sync
  on_failure: Refuse missing staff authority.
  left_inconsistent: No new removal work starts.
  observer: Job progress and service logs.
  provenance: observed
  source: services-platform:modules/tenants/aws/api/tenant-deletion-requests.routes.ts
- n: 4
  actor_or_service: Operator recovery
  action: Require failed request and deleting workspace, create a linked retry, conditionally repoint the request
    and start it.
  anchor: services-platform:modules/tenants/services/deletion/operator-recovery.ts:retryTenantPurge
  seam: state_machine
  on_failure: Refuse a changed request or missing workspace; recover failed starts through jobs.
  left_inconsistent: A losing concurrent retry can leave a job created before repointing failed.
  observer: Job progress and service logs.
  provenance: observed
  source: services-platform:modules/tenants/services/deletion/operator-recovery.ts
branches: []
waits: []
evidence:
- 'services-platform at ad3f6a65: modules/tenants/services/deletion/operator-recovery.ts; aws/api/tenant-deletion-requests.routes.ts.'
- 'FND-PFM-TNL-052: recovery submission has a separate-commit race.'
controls: [TC-AVL-001, TC-AVL-004, TC-AVL-005, TC-AZ-001, TC-AZ-002, TC-AZ-003, TC-AZ-004, TC-AZ-005, TC-AZ-006, TC-DAT-001, TC-DAT-002, TC-DAT-003, TC-DAT-004, TC-DAT-006, TC-IAM-001, TC-IAM-002, TC-IAM-003, TC-IAM-004, TC-IAM-005, TC-IAM-006, TC-LOG-001, TC-LOG-002, TC-LOG-003, TC-LOG-004, TC-LOG-005, TC-MON-001, TC-MON-003, TC-PI-001, TC-PI-002, TC-PI-003, TC-PI-004, TC-PI-005, TC-TEN-001, TC-TEN-002, TC-TEN-003, TC-VEN-001, TC-VEN-002, TC-VEN-003]
---

# Recover A Stalled Deletion

## What It Does

Replace stopped deletion machinery under staff authority. Open watcher checks and conditional purge repointing restrict recovery. Separate job creation can still leave losing records under concurrency.

## The Flow

```mermaid
sequenceDiagram
  autonumber
  actor Staff as Rhythm staff
  participant API as Platform staff route
  participant Rec as Operator recovery
  participant DB as Platform database
  participant Jobs as Job engine
  alt watcher stopped
    Staff->>API: resume schedule
    Rec->>DB: scheduled or held, watcher not running?
    Rec->>Jobs: new watcher, request repointed, started
  else purge failed
    Staff->>API: retry purge
    Rec->>DB: failed and workspace deleting?
    Rec->>Jobs: linked retry, request executing under it, started
  end
```

## When Something Fails

Refuse missing staff authority. No recovery job is intentionally started. Return errors; shared recovery handles unstarted committed jobs. Separate commits can leave an unreferenced watcher. Refuse missing staff authority. No new removal work starts. Refuse a changed request or missing workspace; recover failed starts through jobs. A losing concurrent retry can leave a job created before repointing failed.

## Open Design Questions

Recovery commands lack reviewed version and reason, and their multi-commit job creation needs a concurrency policy.