--- id: WF-PFM-TNL-003 title: Settle a failed build and tell staff trigger: Settle construction failure, retain completed resources and attempt staff communications. product_area: platform module: tenant-lifecycle revision: 1 status: draft design_intent: confirmed origin: services-platform:docs/tenants/tenant-lifecycle-plan.md#§4.4 No automatic compensation, A tenant stranded behind a failed job, §4.6 (approved 2026-10-09); PLAT-M23 D1 implementation_evidence: observed inference: [] source_sha: services-platform: ad3f6a6582843008ed0f5a17b131878d15829af1 implements: - CAP-PFM-TNL-003 moves: - STATE-PFM-TNL-001 - STATE-PFM-TNL-002 crosses: - SVC-PFM-TNL-tenants - SVC-PFM-NOT-notifications idempotency: Failure changes only a still-provisioning workspace. Claimed requester messages and keyed HQ alerts limit repeats without guaranteeing successful delivery. concurrency: Repair visits at most twenty-five candidates and continues after individual errors. An active workspace is not returned to failed-provisioning standing. hops: - n: 1 actor_or_service: Build job (record failure) or jobs reconciler action: Settle the attempt failed and invoke its registered domain consequence. anchor: services-platform:modules/jobs (onSettledFailure) seam: state_machine on_failure: The failure task has bounded retries; reconciliation can attempt settlement later. left_inconsistent: The job record can lag the stopped execution. observer: Job progress and service logs. provenance: documented source: services-platform:docs/jobs.md#§4 onSettledFailure - n: 2 actor_or_service: Failure consequence action: Fail only a workspace still provisioning and clear local availability facts. anchor: services-platform:modules/tenants/services/provisioning/provision-tenant-failure.ts:applyTenantProvisioningFailure seam: sync on_failure: Propagate errors for caller recovery, refusing a missing job. left_inconsistent: An already active workspace remains active. observer: Job progress and service logs. provenance: observed source: services-platform:modules/tenants/services/persistence/tenants.ts:failTenantProvisioning - n: 3 actor_or_service: Failure consequence action: Attempt the claimed requester failure notice with summary and job link. anchor: services-platform:modules/jobs/services/notices/notify-job-requester.ts:notifyJobRequester seam: sync on_failure: Record or log delivery failure without changing build outcome. left_inconsistent: The requester can miss the failure message. observer: Job progress and service logs. provenance: observed source: services-platform:modules/tenants/services/provisioning/provisioning-notice.ts:buildTenantProvisioningNotice - n: 4 actor_or_service: Failure consequence action: Attempt the keyed HQ failure notification. anchor: services-platform:modules/tenants/services/provisioning/publish-provisioning-alert.ts:publishProvisioningAlert seam: event on_failure: Log notification errors without failing settlement. left_inconsistent: The HQ channel can miss the outcome. observer: Job progress and service logs. provenance: observed source: services-platform:modules/tenants/services/provisioning/publish-provisioning-alert.ts - n: 5 actor_or_service: Stranded-provisioning repair action: After reconciliation, repair workspaces still provisioning behind their latest failed attempt. anchor: services-platform:modules/tenants/services/provisioning/repair-stranded-provisioning.ts:repairStrandedTenantProvisioning seam: schedule on_failure: Log an individual error and continue; later passes can retry. left_inconsistent: Unrepaired workspaces can remain stranded until another pass. observer: Job progress and service logs. provenance: observed source: services-platform:modules/tenants/services/provisioning/repair-stranded-provisioning.ts branches: - Workflow settlement, reconciliation and stranded-workspace repair use the registered failure consequence. waits: - Shared reconciliation runs at five-minute intervals. evidence: - 'services-platform at ad3f6a65: modules/tenants/services/provisioning/{provision-tenant-failure, repair-stranded-provisioning, provisioning-notice, publish-provisioning-alert}.ts; tenants-notifications.ts; docs/jobs.md.' controls: [TC-AVL-001, TC-AVL-004, TC-AVL-005, TC-AZ-001, TC-AZ-002, TC-AZ-003, TC-AZ-004, TC-AZ-005, TC-DAT-001, TC-DAT-002, TC-DAT-003, TC-DAT-004, TC-IAM-001, TC-IAM-002, TC-IAM-003, TC-IAM-004, TC-IAM-005, TC-IAM-006, TC-LOG-001, TC-LOG-004, TC-LOG-005, TC-MON-001, TC-MON-002, TC-PI-001, TC-PI-002, TC-PI-003, TC-PI-004, TC-PI-005, TC-PRV-001, TC-PRV-002, TC-TEN-001, TC-TEN-002, TC-TEN-003, TC-VEN-001, TC-VEN-002, TC-VEN-003] --- # Settle A Failed Build And Tell Staff ## What It Does Settle construction failure, retain completed resources and attempt staff communications. Failure changes only a still-provisioning workspace. Claimed requester messages and keyed HQ alerts limit repeats without guaranteeing successful delivery. ## The Flow ```mermaid sequenceDiagram autonumber participant Job as Build job or reconciler participant FC as Failure consequence participant DB as Platform database participant Mail as Requester notice participant HQ as HQ operations channel participant Rep as Repair pass Job->>FC: attempt settled failed FC->>DB: workspace provisioning -> failed (only if still provisioning) FC->>Mail: failure notice (claimed once) FC->>HQ: provisioning failed alert (once per job) Rep->>DB: find workspaces building behind a failed attempt Rep->>FC: apply the same consequence ``` ## When Something Fails The failure task has bounded retries; reconciliation can attempt settlement later. The job record can lag the stopped execution. Propagate errors for caller recovery, refusing a missing job. An already active workspace remains active. Record or log delivery failure without changing build outcome. The requester can miss the failure message. Log notification errors without failing settlement. The HQ channel can miss the outcome. Log an individual error and continue; later passes can retry. Unrepaired workspaces can remain stranded until another pass. ## Open Design Questions Outcome messaging can say construction failed even when activation committed. Delivery claims do not guarantee the requester or HQ received the message.