--- id: JNY-PFM-OPS-002 title: Rhythm staff recover failed workspace provisioning feature: FEAT-PFM-OPS-01 product_area: platform module: operations revision: 1 status: draft product_intent: confirmed implementation_evidence: partially-observed supersedes: [] source_sha: services-platform: ad3f6a6582843008ed0f5a17b131878d15829af1 macroservice-libraries: 54536a441935c6e09fdb7f96f0bec71c52ba4e03 react-console: baf6dae382d6f41dce976c7f848667ccc37e076f actor: ACT-PFM-004 goal: 'Follow a provisioning failure, learn its outcome and recover without repeating completed work.' capabilities: - CAP-PFM-OPS-006 - CAP-PFM-OPS-001 - CAP-PFM-OPS-002 - CAP-PFM-OPS-003 - CAP-PFM-OPS-004 screens: - /hq/tenants/provision - '/hq/jobs/{jobKind}/{jobId}' - /hq/jobs steps: - 'n': 1 surface: /hq/tenants/provision action: Finish the workspace preparation wizard. capability: CAP-PFM-OPS-006 expected: Submission opens the created job’s HQ page. provenance: test source: 'react-console:e2e/mock/hq-ops.mock.spec.ts#walks every step and lands on the job it started' - 'n': 2 surface: '/hq/jobs/{jobKind}/{jobId}' action: Follow the active run on its page. capability: CAP-PFM-OPS-006 expected: The page presents progress and history while refreshing active work. provenance: test source: 'react-console:e2e/mock/hq-ops.mock.spec.ts#shows a running job''s step, its timeline and its event log' - 'n': 3 surface: job action: Allow a permanent task refusal to end execution. capability: CAP-PFM-OPS-001 expected: The job retains the safe failure explanation and the owner applies the workspace consequence. provenance: test source: >- macroservice-libraries:packages/jobs/src/workflow/job-workflow.spec.ts#fails the job with the refusal’s own words and the execution’s correlation - 'n': 4 surface: job action: Attempt the declared requester notification. capability: CAP-PFM-OPS-003 expected: 'Delivery handling uses the job’s requester, environment and matching HQ page link.' provenance: observed source: 'services-platform:modules/jobs/services/notices/notify-job-requester.ts:notifyJobRequester' - 'n': 5 surface: '/hq/jobs/{jobKind}/{jobId}' action: Follow the message link to investigate failure. capability: CAP-PFM-OPS-006 expected: The detail page explains the failure and offers eligible provisioning recovery. provenance: test source: 'react-console:e2e/mock/hq-ops.mock.spec.ts#shows a failed job''s reason and retries its provisioning' - 'n': 6 surface: retryTenantProvisioning action: Confirm Retry Provisioning. capability: CAP-PFM-OPS-004 expected: The new attempt links to its predecessor and carries completed step keys. provenance: test source: >- macroservice-libraries:packages/jobs/src/engine/job-engine.spec.ts#answers the steps the prior job completed, which the start carries into the new execution alternates: - from_step: 3 condition: No execution starts or failure handling never completes. path: Reconciliation later applies the registered failure consequence. outcome: The workspace reaches recoverable failure and notification handling can run. provenance: test source: >- services-platform:modules/tenants/tests/integration/lifecycle/reconciled-job-failure.integration.spec.js#tells the staff member who asked for the tenant that it failed, though no task of the job ever ran - from_step: 5 condition: The staff caller lacks the retry button’s permission. path: Inspect the failure without a retry control. outcome: A permitted operator is needed to request recovery. provenance: observed source: 'react-console:components/console/hq/job/job-retry.tsx:JobRetry' failures: - from_step: 6 condition: The caller has the button permission but lacks backend provisioning authority. path: Display the operation’s refusal after confirmation. outcome: No retry job is created. provenance: observed source: >- services-platform:modules/tenants/aws/api/tenants.routes.ts:retryTenantProvisioning (HQ_TENANTS_PERMISSIONS.Provision) rules_in_play: - BR-CAP-PFM-OPS-001-06 - BR-CAP-PFM-OPS-002-07 - BR-CAP-PFM-OPS-003-03 - BR-CAP-PFM-OPS-003-06 - BR-CAP-PFM-OPS-004-01 - BR-CAP-PFM-OPS-004-02 - BR-CAP-PFM-OPS-004-08 assertions: - id: AS-JNY-PFM-OPS-002-01 statement: A completed failure notice suppresses delivery when its consequence is repeated sequentially. provenance: test source: >- services-platform:modules/tenants/tests/integration/lifecycle/reconciled-job-failure.integration.spec.js#sends the requester nothing a second time either - id: AS-JNY-PFM-OPS-002-02 statement: The provisioning failure consequence restores a recoverable workspace state. provenance: test source: >- services-platform:modules/tenants/tests/integration/lifecycle/reconciled-job-failure.integration.spec.js#moves the tenant to provisioning_failed, which is what makes it retryable and deletable again - id: AS-JNY-PFM-OPS-002-03 statement: Retry preserves predecessor identity and completed step keys. provenance: test source: >- macroservice-libraries:packages/jobs/src/engine/job-engine.spec.ts#creates the retry with lineage to the prior job e2e_policy: required e2e_reason: >- The outcome crosses submission, monitoring, messaging and recovery, requiring browser coverage alongside backend failure and delivery proofs. proved_by: [] --- # Rhythm staff recover failed workspace provisioning ## Story A Rhythm administrator submits workspace provisioning and follows the job page. Failure supplies an explanation and the owner’s recovery consequence, with requester notification attempted independently of job success. The administrator can confirm a supported retry that keeps the prior attempt’s history. This journey combines wizard, monitoring, notification and owning-module recovery evidence; it is not one already bound end-to-end proof.