--- id: ATS-PFM-OPS-004 scope: CAP-PFM-OPS-004 product_area: platform module: operations revision: 2 status: proposed approved_by: null approved_at: null source_sha: services-platform: ad3f6a6582843008ed0f5a17b131878d15829af1 macroservice-libraries: 54536a441935c6e09fdb7f96f0bec71c52ba4e03 react-console: baf6dae382d6f41dce976c7f848667ccc37e076f scenarios: - id: ATS-PFM-OPS-004-01 title: Restrict retry creation level: unit proves: - BR-CAP-PFM-OPS-004-01 - INV-DOM-PFM-OPS-001-10 preconditions: 'Running, nonretryable failed and retryable failed jobs exist.' steps: Request a retry for each. expected: The first two requests fail and the last creates a linked job. negative: true data_needs: Controlled records and caller contexts matching this scenario’s preconditions. bound_to: - TST-PFM-OPS-005 provenance: test source: >- macroservice-libraries:packages/jobs/src/engine/job-engine.spec.ts#refuses to create a retry of a kind that is not retryable evidence_candidates: - >- macroservice-libraries:packages/jobs/src/engine/job-engine.spec.ts#refuses to create a retry of a kind that is not retryable category: validation binding_evidence: - >- macroservice-libraries:packages/jobs/src/engine/job-engine.spec.ts#refuses to create a retry of a kind that is not retryable - id: ATS-PFM-OPS-004-02 title: Carry completed work into retry level: unit proves: - BR-CAP-PFM-OPS-004-02 preconditions: A failed job completed its first three steps. steps: Create and start a retry. expected: The copied completion keys let execution continue without repeating those steps. negative: false data_needs: Controlled records and caller contexts matching this scenario’s preconditions. bound_to: [] provenance: proposed source: >- macroservice-libraries:packages/jobs/src/engine/job-engine.spec.ts#answers the steps the prior job completed, which the start carries into the new execution evidence_candidates: - >- macroservice-libraries:packages/jobs/src/engine/job-engine.spec.ts#answers the steps the prior job completed, which the start carries into the new execution category: rule binding_evidence: - >- macroservice-libraries:packages/jobs/src/engine/job-engine.spec.ts#answers the steps the prior job completed, which the start carries into the new execution binding_review: >- The engine suite checks carried completion keys; an end-to-end bypass sequence is not established by that binding. - id: ATS-PFM-OPS-004-03 title: Reject invalid cancellation requests level: unit proves: - BR-CAP-PFM-OPS-004-04 preconditions: Noncancellable and final jobs exist and another request lacks a reason. steps: Ask the engine to cancel each case. expected: Every invalid cancellation is refused. negative: true data_needs: Controlled records and caller contexts matching this scenario’s preconditions. bound_to: - TST-PFM-OPS-003 provenance: test source: >- macroservice-libraries:packages/jobs/src/engine/job-cancellation.spec.ts#refuses a job whose type is not cancellable evidence_candidates: - >- macroservice-libraries:packages/jobs/src/engine/job-cancellation.spec.ts#refuses a job whose type is not cancellable category: validation binding_evidence: - >- macroservice-libraries:packages/jobs/src/engine/job-cancellation.spec.ts#refuses a job whose type is not cancellable - id: ATS-PFM-OPS-004-04 title: Commit the stop request first level: unit proves: - BR-CAP-PFM-OPS-004-05 preconditions: A cancellable job is running and another has not started. steps: Cancel each and repeat the requests. expected: >- History precedes execution stop, the unstarted job settles cancelled and repeated requests add nothing. negative: false data_needs: Controlled records and caller contexts matching this scenario’s preconditions. bound_to: - TST-PFM-OPS-003 provenance: test source: >- macroservice-libraries:packages/jobs/src/engine/job-cancellation.spec.ts#records the request on the root and in its history, then stops the execution with the cause evidence_candidates: - >- macroservice-libraries:packages/jobs/src/engine/job-cancellation.spec.ts#records the request on the root and in its history, then stops the execution with the cause category: transition binding_evidence: - >- macroservice-libraries:packages/jobs/src/engine/job-cancellation.spec.ts#records the request on the root and in its history, then stops the execution with the cause - id: ATS-PFM-OPS-004-05 title: Distinguish requested from accidental abort level: unit proves: - BR-CAP-PFM-OPS-004-06 preconditions: 'Two executions abort, only one after a recorded cancellation request.' steps: Reconcile both. expected: >- The requested abort settles cancelled without a business failure consequence; the other fails normally. negative: false data_needs: Controlled records and caller contexts matching this scenario’s preconditions. bound_to: - TST-PFM-OPS-003 provenance: test source: >- macroservice-libraries:packages/jobs/src/engine/job-cancellation.spec.ts#settles an execution aborted on request as cancelled, applying no failure consequence evidence_candidates: - >- macroservice-libraries:packages/jobs/src/engine/job-cancellation.spec.ts#settles an execution aborted on request as cancelled, applying no failure consequence category: transition binding_evidence: - >- macroservice-libraries:packages/jobs/src/engine/job-cancellation.spec.ts#settles an execution aborted on request as cancelled, applying no failure consequence - id: ATS-PFM-OPS-004-06 title: Confirm provisioning recovery in HQ level: browser proves: - BR-CAP-PFM-OPS-004-08 preconditions: Staff have retry authority and a failed workspace-provisioning job. steps: Open the detail page and confirm Retry Provisioning. expected: >- With the mocked service response, confirmation changes the page to the returned running attempt; the browser check does not establish the backend provisioning permission. negative: false data_needs: Controlled records and caller contexts matching this scenario’s preconditions. bound_to: - TST-PFM-OPS-054 provenance: test source: 'react-console:e2e/mock/hq-ops.mock.spec.ts#shows a failed job''s reason and retries its provisioning' evidence_candidates: - >- react-console:e2e/mock/hq-ops.mock.spec.ts#shows a failed job's reason and retries its provisioning category: permission binding_evidence: - >- react-console:e2e/mock/hq-ops.mock.spec.ts#shows a failed job's reason and retries its provisioning - id: ATS-PFM-OPS-004-07 title: Preserve the retry’s predecessor level: unit proves: - AS-JNY-PFM-OPS-002-03 preconditions: A retryable job has failed after completing steps. steps: Create a new retry record. expected: The record links to its predecessor and preserves completed step keys. negative: false data_needs: Controlled records and caller contexts matching this scenario’s preconditions. bound_to: - TST-PFM-OPS-005 provenance: test source: >- macroservice-libraries:packages/jobs/src/engine/job-engine.spec.ts#creates the retry with lineage to the prior job evidence_candidates: - >- macroservice-libraries:packages/jobs/src/engine/job-engine.spec.ts#creates the retry with lineage to the prior job category: invariant binding_evidence: - >- macroservice-libraries:packages/jobs/src/engine/job-engine.spec.ts#creates the retry with lineage to the prior job - id: ATS-PFM-OPS-004-08 title: Confirm recovery from the inspected failure category: interaction level: api proves: - BR-CAP-PFM-OPS-004-08 - BR-CAP-PFM-OPS-006-06 exercises: - CAP-PFM-OPS-004 - CAP-PFM-OPS-006 preconditions: Staff can inspect a failed provisioning run and hold both required action permissions. steps: 'Open the failure, confirm retry, and inspect the successor.' expected: The page follows the newly started attempt while the earlier failure remains inspectable. negative: false data_needs: Isolated workspace records and caller contexts for both sides of the interaction. bound_to: [] provenance: proposed source: >- react-console:e2e/mock/hq-ops.mock.spec.ts#shows a failed job's reason and retries its provisioning; react-console:e2e/mock/hq-ops.mock.spec.ts#shows a running job's step, its timeline and its event log; CAP-PFM-OPS-006; WF-PFM-OPS-003; WF-PFM-OPS-005; WF-PFM-OPS-006 - id: ATS-PFM-OPS-004-09 title: Retain lifecycle failure evidence category: interaction level: api proves: - BR-CAP-PFM-OPS-004-01 exercises: - CAP-PFM-OPS-004 - CAP-PFM-SEC-003 preconditions: A workspace lifecycle run reaches an interrupted execution outcome. steps: Reconcile it and inspect the lifecycle record alongside the job record. expected: Both records identify the failed work without silently reopening its final history. negative: false data_needs: Isolated workspace records and caller contexts for both sides of the interaction. bound_to: [] provenance: proposed source: >- macroservice-libraries:packages/jobs/src/engine/job-engine.spec.ts#refuses to create a retry of a kind that is not retryable; CAP-PFM-SEC-003; WF-PFM-OPS-003; WF-PFM-OPS-005; WF-PFM-OPS-006 - id: ATS-PFM-OPS-004-10 title: Preserve retirement scope category: interaction level: api proves: - BR-CAP-PFM-OPS-004-01 exercises: - CAP-PFM-OPS-004 - CAP-PFM-SEC-004 preconditions: A workspace is deleting and its deletion jobs must remain inspectable. steps: Purge operational resources and inspect the retained deletion runs. expected: Retirement removes owned resources while its retained work still explains the deletion. negative: false data_needs: Isolated workspace records and caller contexts for both sides of the interaction. bound_to: [] provenance: proposed source: >- macroservice-libraries:packages/jobs/src/engine/job-engine.spec.ts#refuses to create a retry of a kind that is not retryable; CAP-PFM-SEC-004; WF-PFM-OPS-003; WF-PFM-OPS-005; WF-PFM-OPS-006 - id: ATS-PFM-OPS-004-11 title: Distinguish requester evidence after requested and accidental abort category: interaction level: api proves: - BR-CAP-PFM-OPS-004-06 - BR-CAP-PFM-OPS-003-06 exercises: - CAP-PFM-OPS-002 - CAP-PFM-OPS-003 - CAP-PFM-OPS-004 workflows: - WF-PFM-OPS-003 preconditions: 'Separate stale provisioning runs abort, with only one carrying a recorded stop request.' steps: Reconcile both runs and inspect the declared failure consequence and requester evidence. expected: >- The accidental abort follows its failure consequence and requester explanation; the requested stop retains cancellation evidence without invoking that consequence. negative: true data_needs: >- Isolated workspace and run-owned lifecycle records, with a second linked workspace where shared company evidence is checked. bound_to: [] provenance: proposed source: >- services-platform:modules/jobs/aws/logic/jobs-reconciler.lambda.ts:createJobsReconcilerFunction; macroservice-libraries:packages/jobs/src/reconciler/job-reconciler.ts:handleCandidate; services-platform:modules/jobs/tests/integration/lifecycle/job-reconciliation/job-reconciliation.integration.spec.js#halts the job only on the third pass; macroservice-libraries:packages/jobs/src/reconciler/job-reconciler.ts:settleFromReconciliation; services-platform:modules/jobs/tests/integration/lifecycle/job-reconciliation/job-reconciliation.integration.spec.js#corrects the job to failed, never succeeded, keeping the progress its steps earned; services-platform:modules/jobs/tests/integration/lifecycle/job-reconciliation/job-reconciliation.integration.spec.js#raises one alert and changes nothing; services-platform:modules/jobs/aws/lambda/RECONCILER.handler.ts:handler; services-platform:modules/jobs/aws/lambda/RECONCILER.handler.ts:handler evidence: - >- Pinned capability rules, object invariants, state transitions and workflows are reviewed for testing obligations. - >- Executable bindings refer only to existing suites; no module executable suite runs during this documentation job. coverage: rule: covered transition: covered validation: covered permission: covered edge: 'gap: FND-PFM-OPS-056' failure: 'gap: FND-PFM-OPS-056' boundary: 'gap: FND-PFM-OPS-056' concurrency: 'gap: FND-PFM-OPS-056' invariant: covered interaction: covered coverage_notes: edge: 'Empty, missing and unusual inputs need dedicated proof.' failure: Injected dependency failures need dedicated proof. boundary: Exact limits and cross-scope boundaries need dedicated proof. concurrency: >- No controlled overlapping execution proves the capability’s read or write behavior while related state changes. related_capabilities: - CAP-PFM-OPS-006 - CAP-PFM-SEC-003 - CAP-PFM-SEC-004 workflow_interactions: - WF-PFM-OPS-003 - WF-PFM-OPS-005 - WF-PFM-OPS-006 coverage_obligations: finding: FND-PFM-OPS-056 uncovered_must_prove: [] object_invariants_requiring_explicit_trace: - INV-DOM-PFM-OPS-001-01 - INV-DOM-PFM-OPS-001-02 - INV-DOM-PFM-OPS-001-03 - INV-DOM-PFM-OPS-001-04 - INV-DOM-PFM-OPS-001-05 - INV-DOM-PFM-OPS-001-06 - INV-DOM-PFM-OPS-001-07 - INV-DOM-PFM-OPS-001-08 - INV-DOM-PFM-OPS-001-09 - INV-DOM-PFM-OPS-001-11 - INV-DOM-PFM-OPS-001-12 related_scenarios: - capabilities: - CAP-PFM-OPS-004 - CAP-PFM-OPS-006 scenario: ATS-PFM-OPS-004-08 - capabilities: - CAP-PFM-OPS-004 - CAP-PFM-SEC-003 scenario: ATS-PFM-OPS-004-09 - capabilities: - CAP-PFM-OPS-004 - CAP-PFM-SEC-004 scenario: ATS-PFM-OPS-004-10 unbound_scenarios: - ATS-PFM-OPS-004-02 - ATS-PFM-OPS-004-08 - ATS-PFM-OPS-004-09 - ATS-PFM-OPS-004-10 - ATS-PFM-OPS-004-11 transition_review: - model: STATE-PFM-OPS-001 transition: 1 from: pending to: running provenance: unknown source: FND-PFM-OPS-056 note: >- Existing cases may exercise this move, but exhaustive transition-to-scenario trace is not established. - model: STATE-PFM-OPS-001 transition: 2 from: running to: succeeded provenance: unknown source: FND-PFM-OPS-056 note: >- Existing cases may exercise this move, but exhaustive transition-to-scenario trace is not established. - model: STATE-PFM-OPS-001 transition: 3 from: running to: failed provenance: unknown source: FND-PFM-OPS-056 note: >- Existing cases may exercise this move, but exhaustive transition-to-scenario trace is not established. - model: STATE-PFM-OPS-001 transition: 4 from: running to: failed provenance: unknown source: FND-PFM-OPS-056 note: >- Existing cases may exercise this move, but exhaustive transition-to-scenario trace is not established. - model: STATE-PFM-OPS-001 transition: 5 from: running to: succeeded provenance: unknown source: FND-PFM-OPS-056 note: >- Existing cases may exercise this move, but exhaustive transition-to-scenario trace is not established. - model: STATE-PFM-OPS-001 transition: 6 from: pending to: failed provenance: unknown source: FND-PFM-OPS-056 note: >- Existing cases may exercise this move, but exhaustive transition-to-scenario trace is not established. - model: STATE-PFM-OPS-001 transition: 7 from: pending to: cancelled provenance: unknown source: FND-PFM-OPS-056 note: >- Existing cases may exercise this move, but exhaustive transition-to-scenario trace is not established. - model: STATE-PFM-OPS-001 transition: 8 from: running to: cancelled provenance: unknown source: FND-PFM-OPS-056 note: >- Existing cases may exercise this move, but exhaustive transition-to-scenario trace is not established. - model: STATE-PFM-OPS-002 transition: 1 from: queued to: running provenance: unknown source: FND-PFM-OPS-056 note: >- Existing cases may exercise this move, but exhaustive transition-to-scenario trace is not established. - model: STATE-PFM-OPS-002 transition: 2 from: running to: completed provenance: unknown source: FND-PFM-OPS-056 note: >- Existing cases may exercise this move, but exhaustive transition-to-scenario trace is not established. - model: STATE-PFM-OPS-002 transition: 3 from: running to: skipped provenance: unknown source: FND-PFM-OPS-056 note: >- Existing cases may exercise this move, but exhaustive transition-to-scenario trace is not established. - model: STATE-PFM-OPS-002 transition: 4 from: running to: retrying provenance: unknown source: FND-PFM-OPS-056 note: >- Existing cases may exercise this move, but exhaustive transition-to-scenario trace is not established. - model: STATE-PFM-OPS-002 transition: 5 from: retrying to: running provenance: unknown source: FND-PFM-OPS-056 note: >- Existing cases may exercise this move, but exhaustive transition-to-scenario trace is not established. - model: STATE-PFM-OPS-002 transition: 6 from: running to: failed provenance: unknown source: FND-PFM-OPS-056 note: >- Existing cases may exercise this move, but exhaustive transition-to-scenario trace is not established. - model: STATE-PFM-OPS-002 transition: 7 from: retrying to: failed provenance: unknown source: FND-PFM-OPS-056 note: >- Existing cases may exercise this move, but exhaustive transition-to-scenario trace is not established. - model: STATE-PFM-OPS-002 transition: 8 from: queued to: completed provenance: unknown source: FND-PFM-OPS-056 note: >- Existing cases may exercise this move, but exhaustive transition-to-scenario trace is not established. permission_review: - key: hq.jobs.retry where_checked: console provenance: unknown source: FND-PFM-OPS-056 note: >- Scenario-level audience checks do not establish every grant combination at each enforcement surface. - key: hq.tenants.provision where_checked: api provenance: unknown source: FND-PFM-OPS-056 note: >- Scenario-level audience checks do not establish every grant combination at each enforcement surface. gap_detail: >- Failed execution stop, failed retry startup and concurrent stop-versus-completion need proof beyond sequential engine cases. workflow_scenarios: WF-PFM-OPS-003: - ATS-PFM-OPS-004-11 WF-PFM-OPS-005: - ATS-PFM-OPS-004-06 --- # Recover or deliberately stop background work: Capability Test Specification ## Coverage The category map distinguishes authored scenarios from unresolved proof obligations. A bound scenario names an existing executable suite; its recorded result remains unrun. | Rule | Proof status | |---|---| | `BR-CAP-PFM-OPS-004-01` | `ATS-PFM-OPS-004-01`, `ATS-PFM-OPS-004-09`, `ATS-PFM-OPS-004-10` | | `BR-CAP-PFM-OPS-004-02` | `ATS-PFM-OPS-004-02` | | `BR-CAP-PFM-OPS-004-04` | `ATS-PFM-OPS-004-03` | | `BR-CAP-PFM-OPS-004-05` | `ATS-PFM-OPS-004-04` | | `BR-CAP-PFM-OPS-004-06` | `ATS-PFM-OPS-004-05` | | `BR-CAP-PFM-OPS-004-08` | `ATS-PFM-OPS-004-06`, `ATS-PFM-OPS-004-08` | ## Interactions With Related Capabilities Confirm recovery from the inspected failure. The page follows the newly started attempt while the earlier failure remains inspectable. Retain lifecycle failure evidence. Both records identify the failed work without silently reopening its final history. Preserve retirement scope. Retirement removes owned resources while its retained work still explains the deletion. Distinguish requester evidence after requested and accidental abort. The accidental abort follows its failure consequence and requester explanation; the requested stop retains cancellation evidence without invoking that consequence. ## Evidence and Boundaries Pinned assertion sources support the listed executable bindings. Proposed cases retain the proof required beyond those boundaries. Structured obligations name incomplete object, transition and permission trace; row-level concurrency proof does not establish complete workflow serialization.