--- id: WF-PFM-OPS-005 title: Retry a failed job without repeating finished steps product_area: platform module: operations revision: 1 status: draft design_intent: proposed implementation_evidence: observed inference: [] trigger: An owning module retries final work; the current HQ control calls workspace provisioning retry. source_sha: services-platform: ad3f6a6582843008ed0f5a17b131878d15829af1 macroservice-libraries: 54536a441935c6e09fdb7f96f0bec71c52ba4e03 react-console: baf6dae382d6f41dce976c7f848667ccc37e076f implements: - CAP-PFM-OPS-004 - CAP-PFM-OPS-006 moves: - STATE-PFM-OPS-001 - STATE-PFM-OPS-002 crosses: - SVC-PFM-OPS-jobs idempotency: >- The workspace state check normally refuses a later repeat after the move to provisioning. The submission token includes the current instant, so two concurrently admitted requests need not share a retry identity. concurrency: >- The initial workspace check and state write are not one serialized decision. Concurrent admitted requests can create separate retry runs. hops: - 'n': 1 actor_or_service: Rhythm HQ job page action: >- Offer confirmed retry when the failed job’s kind contains provisioning and its subject has a workspace, subject to the console jobs-retry permission. anchor: 'react-console:components/console/hq/job/job-retry.tsx:JobRetry' seam: sync on_failure: Keep an unsuccessful action’s dialog open with the returned message. left_inconsistent: No business change is made by the presentation step. observer: Dialog is the observation surface. provenance: test source: 'react-console:e2e/mock/hq-ops.mock.spec.ts#shows a failed job''s reason and retries its provisioning' - 'n': 2 actor_or_service: Workspace lifecycle action: >- Require workspace provisioning authority, check failed provisioning, find the prior run and move the workspace back to provisioning. anchor: >- services-platform:modules/tenants/services/provisioning/submit-tenant-provisioning.ts:retryTenantProvisioning seam: sync on_failure: Refuse absent or ineligible workspace state before changing it. left_inconsistent: An admitted state write precedes new-job execution. observer: Refusal response is the observation surface. provenance: observed source: >- services-platform:modules/tenants/services/provisioning/submit-tenant-provisioning.ts:retryTenantProvisioning - 'n': 3 actor_or_service: Job engine action: 'Create a linked retry, copy completed steps and start execution carrying the bypass keys.' anchor: 'macroservice-libraries:packages/jobs/src/engine/job-engine.ts:retryJob' seam: state_machine on_failure: Propagate a strict start error even after job submission commits. left_inconsistent: A provisioning workspace can have a pending retry with no recorded execution until recovery succeeds. observer: 'Returned start error, followed by job detail and workspace state is the observation surface.' provenance: test source: >- macroservice-libraries:packages/jobs/src/engine/job-engine.spec.ts#answers the steps the prior job completed, which the start carries into the new execution - 'n': 4 actor_or_service: Rhythm HQ job page action: Invalidate the relevant workspace and job reads after the retry call. anchor: 'react-console:components/console/hq/job/job-retry.tsx:JobRetry' seam: sync on_failure: A refresh failure can leave displayed data stale. left_inconsistent: Committed recovery is unaffected by client cache refresh. observer: Page is the observation surface. provenance: observed source: 'react-console:components/console/hq/job/job-retry.tsx:JobRetry' branches: - condition: An owning module must point its own records at the new job before its first task runs path: >- Workspace purge creates its retry inside the business transaction, commits its row changes, then starts execution. waits: [] evidence: - >- macroservice-libraries at 54536a4: engine/job-engine.ts (retryJob, createRetryJob, requireRetryablePriorJob, copyCompletedStepsFromPrior). - >- services-platform at ad3f6a6: modules/tenants/services/provisioning/submit-tenant-provisioning.ts:retryTenantProvisioning; modules/tenants/aws/api/tenants.routes.ts:retryTenantProvisioning. - 'react-console at baf6dae: components/console/hq/job/job-retry.tsx; lib/hq/permissions.ts.' - >- codex examination 2026-10-23: checked the pinned source anchors, planning references and enforcement boundaries; unresolved limits are recorded in module findings. controls: [TC-AVL-001, TC-AVL-004, TC-AVL-005, TC-AZ-001, TC-AZ-002, TC-AZ-003, TC-AZ-004, TC-AZ-005, TC-AZ-006, TC-DAT-001, TC-DAT-003, TC-DAT-004, TC-IAM-001, TC-IAM-002, TC-IAM-003, TC-IAM-004, TC-IAM-005, TC-IAM-006, TC-LOG-001, TC-LOG-002, TC-LOG-003, TC-LOG-004, TC-LOG-005, TC-MON-001, TC-PI-001, TC-PI-002, TC-PI-003, TC-PI-004, TC-PI-005, TC-TEN-001, TC-TEN-002, TC-TEN-003, TC-VEN-001, TC-VEN-002, TC-VEN-003] --- # Retry a failed job without repeating finished steps ## What It Does Retry creates another run and preserves the final predecessor. Completed predecessor steps are copied and passed as bypass keys to orchestration. Platform’s retryable kinds are workspace provisioning and purge; the HQ job control invokes provisioning retry. ## The Flow ```mermaid flowchart TD H0["Offer confirmed retry when the failed job’s kind contains provisioning and its subject has a workspace, subject to the console jobs-retry permission."] H1["Require workspace provisioning authority, check failed provisioning, find the prior run and move the workspace back to provisioning."] H0 --> H1 H2["Create a linked retry, copy completed steps and start execution carrying the bypass keys."] H1 --> H2 H3["Invalidate the relevant workspace and job reads after the retry call."] H2 --> H3 ``` ## When Something Fails Invalid workspace state is refused. A strict start failure reports an error after the workspace and job changes have committed; a later reconciler can fail the unstarted job and restore recoverable workspace state. The console jobs-retry permission differs from the provisioning permission required by the called route. ## Open Design Questions Product must settle which audiences can manage which kinds before general retry or stop controls are exposed.