--- id: CAP-PFM-TNL-003 title: Understand and recover a workspace build feature: FEAT-PFM-TNL-01 product_area: platform module: tenant-lifecycle revision: 2 status: draft product_intent: confirmed implementation_evidence: observed supersedes: [] source_sha: services-platform: ad3f6a6582843008ed0f5a17b131878d15829af1 intent_sources: - 'services-platform:docs/tenants/tenant-lifecycle-plan.md#§4.1' - 'services-platform:docs/tenants/tenant-lifecycle-plan.md#§4.4' - 'services-platform:docs/tenants/tenant-lifecycle-plan.md#§4.6' - 'services-platform:docs/hq/rhythm-hq-plan.md#§D11' purpose: >- Staff inspect build progress and recover failed construction with its partial resources preserved. outcomes: - Provisioning snapshots explain completed steps and failures. - Retry preserves partial construction instead of compensating it away. - Requester notices and operations alerts report the result. - Repair resolves workspaces stranded behind settled jobs. concepts: - CON-PFM-TNL-002 actors: - ACT-PFM-004 - ACT-PFM-039 - ACT-PFM-012 surfaces: - console - api - job preconditions: [] postconditions: [] permissions: - key: hq.tenants.read where_checked: api scope: global - key: hq.tenants.provision where_checked: api scope: global rules: - id: BR-CAP-PFM-TNL-003-01 proved_by: [ATS-PFM-TNL-003-01, TST-PFM-TNL-056] statement: >- Authorized staff read the latest provisioning snapshot; customer administrators are refused and missing workspaces or absent provisioning jobs return not found. kind: permission enforced_at: - api - service must_prove: true provenance: test source: >- services-platform:modules/tenants/tests/e2e/tenant-provisioning.e2e.spec.js#may not watch the provisioning of their own tenant origin: >- services-platform:docs/tenants/tenant-lifecycle-plan.md#§4.1 (approved 2026-10-09) - id: BR-CAP-PFM-TNL-003-02 proved_by: [ATS-PFM-TNL-003-02, TST-PFM-TNL-056] statement: >- A failed build retains the workspace and its partial resources in provisioning-failed state. kind: lifecycle enforced_at: - service must_prove: true provenance: test source: >- services-platform:modules/tenants/tests/e2e/tenant-provisioning.e2e.spec.js#leaves the tenant saying its provisioning failed, which is what makes it retryable origin: >- services-platform:docs/tenants/tenant-lifecycle-plan.md#§4.4 No automatic compensation (approved 2026-10-09) - id: BR-CAP-PFM-TNL-003-03 proved_by: [ATS-PFM-TNL-003-03, TST-PFM-TNL-056] statement: >- Only failed provisioning can be retried; retry restores provisioning state and creates a linked job carrying completed steps. kind: lifecycle enforced_at: - api - service must_prove: true provenance: test source: >- services-platform:modules/tenants/tests/e2e/tenant-provisioning.e2e.spec.js#refuses to provision a tenant again whose provisioning did not fail origin: >- services-platform:docs/tenants/tenant-lifecycle-plan.md#§4.1 Retry (approved 2026-10-09) - id: BR-CAP-PFM-TNL-003-04 proved_by: [ATS-PFM-TNL-003-04, TST-PFM-TNL-083] statement: >- The requester’s outcome notice names the workspace, result, failure summary when applicable and job link; a handled notice is not resent and sending failure does not fail the build. kind: policy enforced_at: - service must_prove: true provenance: test source: >- services-platform:modules/tenants/tests/integration/services/provisioning/provisioning-notices.integration.spec.js#sends nothing a second time when the activate task runs again origin: >- services-platform:docs/tenants/tenant-lifecycle-plan.md#§4.6 (approved 2026-10-09); services-platform:docs/hq/rhythm-hq-plan.md#§D11 - id: BR-CAP-PFM-TNL-003-05 proved_by: [] statement: >- Failure notices explain that the workspace remains and that staff can choose retry or deletion without an automatic follow-up. kind: policy enforced_at: - service must_prove: false provenance: test source: >- services-platform:modules/tenants/tests/unit/services/provisioning/provisioning-notice.spec.js#says the two things the requester may do with it: retry it or delete it origin: >- services-platform:docs/tenants/tenant-lifecycle-plan.md#§4.6 (approved 2026-10-09) - id: BR-CAP-PFM-TNL-003-06 proved_by: [ATS-PFM-TNL-003-05, TST-PFM-TNL-055] statement: >- The HQ operations alert accompanies the requester’s e-mail, and alert delivery failure leaves the job’s result intact. kind: policy enforced_at: - service must_prove: true provenance: test source: >- services-platform:modules/tenants/tests/e2e/tenant-provisioning-alerts.e2e.spec.js#tells the requester by e-mail as well: the alert is beside the notice, not instead of it origin: >- services-platform:docs/tenants/tenant-lifecycle-plan.md#§4.6 (PLAT-M23 D1, approved) - id: BR-CAP-PFM-TNL-003-07 proved_by: [ATS-PFM-TNL-003-06] statement: >- After reconciliation, repair moves a stranded provisioning workspace to failed and handles its requester notice once; running jobs remain untouched. kind: lifecycle enforced_at: - service must_prove: true provenance: test source: >- services-platform:modules/tenants/tests/integration/services/provisioning/repair-stranded-provisioning.integration.spec.js#moves the stranded tenant to provisioning_failed, which makes it retryable and deletable again origin: >- services-platform:docs/tenants/tenant-lifecycle-plan.md#§4.4 A tenant stranded behind a failed job is repaired (approved) - id: BR-CAP-PFM-TNL-003-08 proved_by: [ATS-PFM-TNL-003-10] statement: >- Staff dispose of an unwanted failed build through a deletion request rather than automatic rollback. kind: policy enforced_at: - service must_prove: false provenance: documented source: >- services-platform:docs/tenants/tenant-lifecycle-plan.md#§4.4 Getting rid of a failed tenant origin: >- services-platform:docs/tenants/tenant-lifecycle-plan.md#§4.4 (approved 2026-10-09) scenarios: - id: SC-CAP-PFM-TNL-003-01 given: CRM became unreachable during construction. when: Staff retry after restoring the connection. then: >- The replacement attempt completes outstanding work and opens the workspace. provenance: test source: >- services-platform:modules/tenants/tests/e2e/tenant-provisioning.e2e.spec.js#leaves the tenant usable after the retry - id: SC-CAP-PFM-TNL-003-02 given: A never-started job is settled without updating its workspace. when: The repair pass examines it. then: >- The workspace becomes provisioning failed and its requester notice is handled. provenance: test source: >- services-platform:modules/tenants/tests/integration/lifecycle/reconciled-job-failure.integration.spec.js#tells the staff member who asked for the tenant that it failed, though no task of the job ever ran edge_cases: [] constraints: [] depends_on: - CAP-PFM-TNL-002 related: - CAP-PFM-TNL-001 - CAP-PFM-TNL-013 journeys: - JNY-PFM-TNL-001 - JNY-PFM-TNL-002 open_questions: [] evidence: - >- Inspected source: services-platform:docs/tenants/tenant-lifecycle-plan.md#§4.1 - >- Inspected source: services-platform:docs/tenants/tenant-lifecycle-plan.md#§4.4 - >- Inspected source: services-platform:docs/tenants/tenant-lifecycle-plan.md#§4.6 - 'Inspected source: services-platform:docs/hq/rhythm-hq-plan.md#§D11' - >- Inspected source: services-platform:modules/tenants/tests/e2e/tenant-provisioning.e2e.spec.js#may not watch the provisioning of their own tenant - >- Inspected source: services-platform:docs/tenants/tenant-lifecycle-plan.md#§4.1 (approved 2026-10-09) - >- Inspected source: services-platform:modules/tenants/tests/e2e/tenant-provisioning.e2e.spec.js#leaves the tenant saying its provisioning failed, which is what makes it retryable - >- Inspected source: services-platform:docs/tenants/tenant-lifecycle-plan.md#§4.4 No automatic compensation (approved 2026-10-09) - >- Inspected source: services-platform:modules/tenants/tests/e2e/tenant-provisioning.e2e.spec.js#refuses to provision a tenant again whose provisioning did not fail - >- Inspected source: services-platform:docs/tenants/tenant-lifecycle-plan.md#§4.1 Retry (approved 2026-10-09) - >- Inspected source: services-platform:modules/tenants/tests/integration/services/provisioning/provisioning-notices.integration.spec.js#sends nothing a second time when the activate task runs again - >- Inspected source: services-platform:docs/tenants/tenant-lifecycle-plan.md#§4.6 (approved 2026-10-09); services-platform:docs/hq/rhythm-hq-plan.md#§D11 - >- Inspected source: services-platform:modules/tenants/tests/unit/services/provisioning/provisioning-notice.spec.js#says the two things the requester may do with it: retry it or delete it - >- Inspected source: services-platform:docs/tenants/tenant-lifecycle-plan.md#§4.6 (approved 2026-10-09) - >- Inspected source: services-platform:modules/tenants/tests/e2e/tenant-provisioning-alerts.e2e.spec.js#tells the requester by e-mail as well: the alert is beside the notice, not instead of it - >- Inspected source: services-platform:docs/tenants/tenant-lifecycle-plan.md#§4.6 (PLAT-M23 D1, approved) - >- Inspected source: services-platform:modules/tenants/tests/integration/services/provisioning/repair-stranded-provisioning.integration.spec.js#moves the stranded tenant to provisioning_failed, which makes it retryable and deletable again - >- Inspected source: services-platform:docs/tenants/tenant-lifecycle-plan.md#§4.4 A tenant stranded behind a failed job is repaired (approved) - >- Inspected source: services-platform:docs/tenants/tenant-lifecycle-plan.md#§4.4 Getting rid of a failed tenant - >- Inspected source: services-platform:docs/tenants/tenant-lifecycle-plan.md#§4.4 (approved 2026-10-09) - >- Inspected source: services-platform:modules/tenants/tests/e2e/tenant-provisioning.e2e.spec.js#leaves the tenant usable after the retry - >- Inspected source: services-platform:modules/tenants/tests/integration/lifecycle/reconciled-job-failure.integration.spec.js#tells the staff member who asked for the tenant that it failed, though no task of the job ever ran controls: [TC-AVL-001, TC-AVL-004, TC-AZ-001, TC-AZ-002, TC-AZ-003, TC-AZ-004, TC-AZ-005, TC-DAT-001, TC-DAT-002, TC-DAT-003, TC-DAT-004, TC-IAM-001, TC-IAM-003, TC-IAM-006, TC-LOG-001, TC-LOG-004, TC-LOG-005, TC-MON-001, TC-MON-002, TC-PI-001, TC-PI-002, TC-PI-003, TC-PI-004, TC-PI-005, TC-PRV-001, TC-PRV-002, TC-TEN-001, TC-TEN-002, TC-TEN-003, TC-VEN-001, TC-VEN-002, TC-VEN-003] exercised_by: [FAS-PFM-TNL-01-01, FAS-PFM-TNL-01-02, FAS-PFM-TNL-01-03, FAS-PFM-TNL-01-04] suites: [TST-PFM-TNL-055, TST-PFM-TNL-056, TST-PFM-TNL-083] --- # Understand and recover a workspace build ## Purpose Staff inspect build progress and recover failed construction with its partial resources preserved. ## How It Works The administrator reads the latest build snapshot to see the completed work and failure explanation. Customer administrators cannot inspect or retry this staff operation. A failed build keeps the workspace and everything already created. After the cause is corrected, staff submit a retry. Rhythm records a new attempt linked to the failed job and carries its completed steps. The requester’s outcome notice links back to the job. The operations channel receives a separate alert. Notification problems do not reverse the build result. A repair pass also finds workspaces still provisioning after their job has settled failed and makes them available for staff recovery. ## Relationships - Open a workspace after its applications are ready: The construction job provides the result and retained steps that recovery inspects. - Request a complete customer workspace: The capabilities share the workspace’s recorded lifecycle decision. - Schedule deliberate workspace deletion: The capabilities share the workspace’s recorded lifecycle decision.