--- id: CAP-PFM-OPS-004 title: Recover or deliberately stop background work feature: FEAT-PFM-OPS-01 product_area: platform module: operations revision: 1 status: draft product_intent: proposed implementation_evidence: partially-observed supersedes: [] source_sha: services-platform: ad3f6a6582843008ed0f5a17b131878d15829af1 macroservice-libraries: 54536a441935c6e09fdb7f96f0bec71c52ba4e03 react-console: baf6dae382d6f41dce976c7f848667ccc37e076f intent_sources: - 'services-platform:docs/hq/rhythm-hq-plan.md#D12 (retry where the kind offers one; cancel hidden)' purpose: Provide recovery and stopping semantics that preserve the history of the original work. outcomes: - Retry can retain completed work while creating a separate attempt. - Requested abort is distinguished from ordinary failure. concepts: - CON-PFM-OPS-001 actors: - ACT-PFM-004 - ACT-PFM-006 surfaces: - console - job preconditions: - The kind declares support for the requested retry or cancellation. postconditions: - 'Retry produces a linked new job, while cancellation records the request before stopping execution.' permissions: - key: hq.jobs.retry where_checked: console scope: global - key: hq.tenants.provision where_checked: api scope: global rules: - id: BR-CAP-PFM-OPS-004-01 proved_by: [ATS-PFM-OPS-001-10, ATS-PFM-OPS-004-01, ATS-PFM-OPS-004-09, ATS-PFM-OPS-004-10, TST-PFM-OPS-005] statement: >- Retry creation requires a settled job of a retryable kind and creates a new job linked to the prior execution. kind: validation enforced_at: - service must_prove: true provenance: test source: >- macroservice-libraries:packages/jobs/src/engine/job-engine.spec.ts#refuses to create a retry of a kind that is not retryable origin: 'services-platform:docs/hq/rhythm-hq-plan.md#L. Verification plan (retry lineage)' - id: BR-CAP-PFM-OPS-004-02 proved_by: [ATS-PFM-OPS-001-10, ATS-PFM-OPS-004-02] statement: A retry carries forward completed step keys so its workflow can bypass work already completed. kind: calculation enforced_at: - service must_prove: true provenance: test source: >- macroservice-libraries:packages/jobs/src/engine/job-engine.spec.ts#answers the steps the prior job completed, which the start carries into the new execution origin: none - id: BR-CAP-PFM-OPS-004-03 proved_by: [] statement: >- Workspace provisioning and workspace purge are the retryable Platform kinds; the other registered kinds do not permit retry. kind: policy enforced_at: - service must_prove: false provenance: observed source: >- services-platform:modules/tenants/services/provisioning/provision-tenant-workflow.ts:retryable; modules/tenants/services/deletion/tenant-purge-workflow.ts:retryable origin: none - id: BR-CAP-PFM-OPS-004-04 proved_by: [ATS-PFM-OPS-004-03, TST-PFM-OPS-003] statement: 'Cancellation requires a reason, an unsettled job and a kind that declares cancellation support.' kind: validation enforced_at: - service must_prove: true provenance: test source: >- macroservice-libraries:packages/jobs/src/engine/job-cancellation.spec.ts#refuses a job whose type is not cancellable origin: none - id: BR-CAP-PFM-OPS-004-05 proved_by: [ATS-PFM-OPS-004-04, TST-PFM-OPS-003] statement: >- The cancellation request commits to the job and its history before execution is stopped; an unstarted job settles cancelled immediately, and repeat requests add no second request event. kind: lifecycle enforced_at: - service must_prove: true provenance: test source: >- macroservice-libraries:packages/jobs/src/engine/job-cancellation.spec.ts#records the request on the root and in its history, then stops the execution with the cause origin: none - id: BR-CAP-PFM-OPS-004-06 proved_by: [ATS-PFM-OPS-004-05, ATS-PFM-OPS-004-11, TST-PFM-OPS-003] statement: >- Reconciliation treats an execution aborted after a cancellation request as cancelled without applying failure consequences; an unrequested abort fails normally. kind: lifecycle enforced_at: - service must_prove: true provenance: test source: >- macroservice-libraries:packages/jobs/src/engine/job-cancellation.spec.ts#settles an execution aborted on request as cancelled, applying no failure consequence origin: none - id: BR-CAP-PFM-OPS-004-07 proved_by: [] statement: >- Platform exposes no job-cancellation route or screen despite cancellation support in the eligibility population kind. kind: policy enforced_at: - api must_prove: false provenance: observed source: >- services-platform:modules/jobs/aws/api/jobs.routes.ts (read-only surface); modules/eligibility/services/population/population-workflow.ts:cancellable origin: none - id: BR-CAP-PFM-OPS-004-08 proved_by: [ATS-PFM-OPS-004-06, ATS-PFM-OPS-004-08, ATS-PFM-OPS-006-05, TST-PFM-OPS-054] statement: >- The HQ job page offers confirmed retry for failed workspace provisioning under the jobs retry permission, while the called workspace operation requires provisioning authority. kind: permission enforced_at: - console - api must_prove: true provenance: test source: 'react-console:e2e/mock/hq-ops.mock.spec.ts#shows a failed job''s reason and retries its provisioning' origin: 'services-platform:docs/hq/rhythm-hq-plan.md#D12' scenarios: - id: SC-CAP-PFM-OPS-004-01 given: A retryable job has completed earlier steps before failing. when: Its owner creates a retry. then: The new job identifies its predecessor and carries the completed step keys. provenance: inferred source: >- macroservice-libraries:packages/jobs/src/engine/job-engine.spec.ts#creates the retry with lineage to the prior job note: >- This illustrative example combines behavior established by the cited test and the capability’s implementation evidence; it is not an independently bound end-to-end proof. - id: SC-CAP-PFM-OPS-004-02 given: A cancellable job is executing. when: An authorized engine caller records a stop reason and requests cancellation. then: An aborted execution settles cancelled without invoking the business failure consequence. provenance: inferred source: >- macroservice-libraries:packages/jobs/src/engine/job-cancellation.spec.ts#settles an execution aborted on request as cancelled, applying no failure consequence note: >- This illustrative example combines behavior established by the cited test and the capability’s implementation evidence; it is not an independently bound end-to-end proof. edge_cases: [] constraints: [] depends_on: - CAP-PFM-OPS-001 related: - CAP-PFM-OPS-006 - CAP-PFM-SEC-003 - CAP-PFM-SEC-004 journeys: - JNY-PFM-OPS-002 open_questions: - OQ-PFM-OPS-003 evidence: - >- macroservice-libraries at 54536a4: engine/job-engine.ts (retryJob, createRetryJob, cancelJob, settleCancelledJob), CHANGELOG 0.5.0; job-engine and job-cancellation spec suites. - >- services-platform at ad3f6a6: retryable and cancellable declarations of every Platform job type; modules/tenants/aws/api/tenants.routes.ts:retryTenantProvisioning (permission); aws/platform.service.ts HQ_JOBS_PERMISSIONS. - >- react-console at baf6dae: components/console/hq/job/job-retry.tsx, lib/hq/permissions.ts; e2e/mock/hq-ops.mock.spec.ts. - 'Permission keys this capability covers: hq.jobs.retry (no operation requires it; FND-PFM-OPS-004).' - >- Examination: proposed acceptance matrices retain candidate test references; incomplete proof is recorded by FND-PFM-OPS-028. controls: [TC-AVL-001, TC-AVL-004, TC-AZ-001, TC-AZ-002, TC-AZ-003, TC-AZ-004, TC-AZ-005, TC-DAT-001, TC-DAT-003, TC-DAT-004, TC-IAM-001, TC-IAM-003, TC-IAM-006, TC-LOG-001, TC-LOG-002, TC-LOG-003, TC-LOG-004, TC-LOG-005, TC-MON-001, TC-PI-001, TC-PI-002, TC-PI-004, TC-PI-005, TC-TEN-001, TC-TEN-002, TC-TEN-003, TC-VEN-001, TC-VEN-002, TC-VEN-003] exercised_by: [FAS-PFM-OPS-01-03, FAS-PFM-OPS-01-05] suites: [TST-PFM-OPS-003, TST-PFM-OPS-005, TST-PFM-OPS-054] --- # Recover or deliberately stop background work ## Purpose Provide recovery and stopping semantics that preserve the history of the original work. ## How It Works The shared engine supplies retry and cancellation operations to owning modules. Retrying creates a separate job and preserves the link to the earlier attempt. Completed step keys let the workflow avoid repeating successful work. An HQ operator can retry failed workspace provisioning after confirming the action. The button’s permission differs from the permission the receiving operation checks, which can cause a refusal. Cancellation records the request before attempting to stop execution. A requested abort then settles as cancelled rather than as an ordinary failure. No current Platform route or screen exposes that cancellation flow to a person. ## Relationships The job record supports reconciliation, notices and both monitors; owning modules retain responsibility for the work itself.