--- id: WF-PFM-SEC-003 title: Watch a retirement request through its waiting period trigger: HQ staff request retirement of a disabled workspace. product_area: platform module: security revision: 1 status: draft design_intent: proposed implementation_evidence: observed inference: [] source_sha: services-platform: ad3f6a6582843008ed0f5a17b131878d15829af1 implements: - CAP-PFM-SEC-004 moves: - STATE-PFM-SEC-001 - STATE-PFM-SEC-003 crosses: - SVC-PFM-SEC-tenants - SVC-PFM-SEC-security idempotency: The request identifies its authoritative schedule job; a superseded watcher settles without destructive action. Notice slots are unique by kind and due time, and recorded delivered/suppressed notices are skipped. A successful send followed by failed outcome persistence can be sent again. Hold, release and cancel refuse inapplicable repeated decisions. concurrency: Request and workspace locks serialize competing decisions and the destructive boundary. Schedule recovery refuses while the old execution is still running and repoints the request to replacement work. hops: - n: 1 actor_or_service: HQ staff (request) action: Lock and validate workspace protections, typed confirmation, reason and open-request uniqueness; calculate eligibility and commit request, notice slots, schedule job and deletion-pending lifecycle. anchor: services-platform:modules/tenants/services/deletion/request-deletion.ts:requestTenantDeletion seam: sync on_failure: Refuse any unmet precondition without committing the request. left_inconsistent: No partial retirement request survives refusal. observer: The staff caller receives the specific rule refusal. provenance: observed source: services-platform:modules/tenants/services/deletion/request-deletion.ts (header) - n: 2 actor_or_service: Job engine action: Start the watcher after commit. anchor: '@rhythmsoftware/macroservice-jobs engine startJobOrLeaveToReconciler' seam: state_machine on_failure: Leave durable unstarted work for reconciliation; an operator must resume the stalled request. left_inconsistent: The request can remain valid without a live watcher. observer: Job monitoring exposes the stalled job; the inspected alarms do not prove watcher detection. provenance: documented source: services-platform:modules/tenants/services/deletion/request-deletion.ts (header); operator-recovery.ts - n: 3 actor_or_service: Schedule job (send due notices) action: Send due staff notices and return the next wake time. anchor: services-platform:modules/tenants/services/deletion/tenant-deletion-schedule-workflow.ts (SendDueNotices) seam: state_machine on_failure: Use six configured task retry attempts; undelivered gating notices block destruction and delivery failures feed the alarm. left_inconsistent: The disabled workspace remains intact; notice outcome persistence can lag delivery. observer: The request’s notice rows and failure alarm show outstanding delivery. provenance: observed source: services-platform:modules/tenants/aws/logic/tenant-deletion-notice.alarm.ts; modules/tenants/services/deletion/tenant-deletion-schedule-workflow.ts:NOTICE_TASK_RETRY; modules/tenants/services/deletion/notices/send-notices.ts:sendOneNotice - n: 4 actor_or_service: Schedule job (wait) action: Wait in the execution until the next notice or eligibility instant. anchor: services-platform:modules/tenants/services/deletion/tenant-deletion-schedule-workflow.ts (WaitForNextNotice) seam: schedule on_failure: A terminated watcher requires operator resume. left_inconsistent: A valid request may have no running watcher. observer: Operators inspect the job and request. provenance: observed source: services-platform:modules/tenants/services/deletion/tenant-deletion-schedule-workflow.ts (header) - n: 5 actor_or_service: Staff (hold, release, cancel) action: Apply a reasoned hold, extend eligibility on release, or cancel before execution. anchor: services-platform:modules/tenants/services/deletion/deletion-holds.ts seam: sync on_failure: Reject according to the current locked request state. left_inconsistent: A refused decision changes no request. observer: The caller and retained notices expose the decision. provenance: observed source: services-platform:modules/tenants/services/deletion/deletion-holds.ts (header) - n: 6 actor_or_service: Schedule job (point of no return) action: Recheck request, workspace, notices and current requester authority under locks, then create the purge job with executing/deleting state. anchor: services-platform:modules/tenants/services/deletion/begin-destruction.ts seam: state_machine on_failure: Wait on temporary blockers, hold on lost authority, or settle when the request can no longer proceed. left_inconsistent: No destructive state is committed before all checks pass. observer: Request state and notices expose the boundary result. provenance: observed source: services-platform:modules/tenants/services/deletion/begin-destruction.ts (header) - n: 7 actor_or_service: Schedule job (conclude) action: Record why the watcher concluded and settle its job. anchor: services-platform:modules/tenants/services/deletion/tenant-deletion-schedule-workflow.ts (ConcludeSchedule) seam: state_machine on_failure: Task retry applies; exhaustion can still fail the schedule job. left_inconsistent: Request decisions remain as committed even if job settlement fails. observer: Job monitoring records conclusion or failure. provenance: observed source: services-platform:modules/tenants/services/deletion/tenant-deletion-schedule-workflow.ts (header) branches: - Resume replaces the watcher without releasing holds, shortening eligibility or satisfying gating notices. - Disposable test workspaces use the short waiting policy outside production; production refuses that marker. waits: - Default eligibility is thirty days, or one day when never activated; watcher execution can last a year. - Temporary blockers at the destructive boundary are revisited after six hours. evidence: - 'services-platform at ad3f6a65: modules/tenants/services/deletion/request-deletion.ts, waiting-period.ts, notice-schedule.ts, deletion-holds.ts, tenant-deletion-schedule-workflow.ts, begin-destruction.ts, operator-recovery.ts; aws/logic/tenant-deletion-schedule.step-function.ts, tenant-deletion-notice.alarm.ts; docs/tenants/tenant-lifecycle-plan.md §5.' examined_at: '2026-10-23' examiner: codex controls: [TC-AVL-001, TC-AVL-004, TC-AVL-005, TC-AZ-001, TC-AZ-002, TC-AZ-003, TC-AZ-004, TC-AZ-005, TC-DAT-001, TC-DAT-002, TC-DAT-003, TC-DAT-004, TC-DAT-006, TC-IAM-001, TC-IAM-002, TC-IAM-003, TC-IAM-004, TC-IAM-005, TC-IAM-006, TC-LOG-001, TC-LOG-002, TC-LOG-003, TC-LOG-004, TC-LOG-005, TC-MON-001, TC-MON-002, TC-MON-003, TC-PI-001, TC-PI-002, TC-PI-003, TC-PI-004, TC-PI-005, TC-PRV-001, TC-PRV-002, TC-TEN-001, TC-TEN-002, TC-TEN-003, TC-VEN-001, TC-VEN-002, TC-VEN-003] --- # Watch a retirement request through its waiting period ## What It Does The schedule protects a cancellable waiting period and rechecks authority before destructive work starts. Its notice delivery is at least once where send success precedes outcome persistence; the design does not promise exactly-once mail. ## The Flow ```mermaid flowchart TD H1["Commit retirement request"] H2["Start watcher"] H3["Send due notices"] H4["Wait for next action"] H5["Apply staff decision"] H6["Recheck destruction authority"] H7["Settle watcher"] H1 --> H2 H2 --> H3 H3 -->|next notice| H4 H4 -->|wake| H3 H3 -->|eligible| H6 H6 -->|temporary blocker| H4 H6 -->|began or closed| H7 H5 -. hold or release .-> H4 H5 -. cancelled .-> H7 ``` The diagram separates the relevant success, alternative and failure paths. The numbered hop rows retain the detailed authority checks, delegated work and retry limits. ## When Something Fails A failed or unstarted watcher does not automatically acquire a replacement. Operator resume is the observed recovery path, and reliable stall detection remains an open finding. The request identifies its authoritative schedule job; a superseded watcher settles without destructive action. Notice slots are unique by kind and due time, and recorded delivered/suppressed notices are skipped. A successful send followed by failed outcome persistence can be sent again. Hold, release and cancel refuse inapplicable repeated decisions. Request and workspace locks serialize competing decisions and the destructive boundary. Schedule recovery refuses while the old execution is still running and repoints the request to replacement work. ## Open Design Questions The recorded findings retain any stronger planning expectation, missing writer or unproved recovery path. Observed behavior does not approve those outstanding product choices.