--- id: CAP-PFM-OPS-001 title: Complete background work with a durable record feature: FEAT-PFM-OPS-01 product_area: platform module: operations revision: 1 status: draft product_intent: confirmed implementation_evidence: observed supersedes: [] source_sha: services-platform: ad3f6a6582843008ed0f5a17b131878d15829af1 macroservice-libraries: 54536a441935c6e09fdb7f96f0bec71c52ba4e03 intent_sources: - 'services-platform:docs/hq/rhythm-hq-plan.md#K8 (plan approved 2026-10-11)' - >- services-platform:docs/hq/rhythm-hq-plan.md#L. Verification plan (durable job persistence, retry lineage and idempotent submission) purpose: Give long-running work a persistent account that operators can inspect and use for recovery. outcomes: - Repeated intentions resolve to the existing job rather than another submission. - 'Operators can inspect completed work, interruptions and failure explanations.' - Failure handling includes the owning record’s declared recovery consequence. concepts: - CON-PFM-OPS-001 actors: - ACT-PFM-006 - ACT-PFM-004 surfaces: - job preconditions: - >- The owning module registers the job kind, step catalog, execution workflow, recovery declarations and failure consequence. postconditions: - Submission preserves the steps and event history and the job eventually records one final status. permissions: [] rules: - id: BR-CAP-PFM-OPS-001-01 proved_by: [ATS-PFM-OPS-001-01, TST-PFM-OPS-005] statement: >- Repeated intentions identify one job; a concurrent uniqueness conflict resolves to the existing job outside a caller transaction but aborts an open caller transaction for the caller to retry. kind: invariant enforced_at: - service - database must_prove: true provenance: observed source: 'macroservice-libraries:packages/jobs/src/engine/job-engine.ts:createJob' origin: 'services-platform:docs/hq/rhythm-hq-plan.md#L. Verification plan (approved K8, 2026-10-11)' examples: positive: A repeated workspace-provisioning submission returns the existing job. negative: A unique submission key prevents two job records for the same intention. - id: BR-CAP-PFM-OPS-001-02 proved_by: [ATS-PFM-OPS-001-02] statement: >- Creating a job records its queued steps and submission event; starting its execution inside the submitting transaction is refused. kind: lifecycle enforced_at: - service must_prove: true provenance: test source: >- macroservice-libraries:packages/jobs/src/engine/job-engine.spec.ts#refuses submitJob inside a transaction: the execution starts after the commit origin: none - id: BR-CAP-PFM-OPS-001-03 proved_by: [ATS-PFM-OPS-001-03, ATS-PFM-OPS-001-09] statement: >- When execution cannot start after submission commits, the API-facing start helper leaves the job pending for reconciliation and preserves the successful submission response. kind: policy enforced_at: - service must_prove: true provenance: observed source: 'macroservice-libraries:packages/jobs/src/engine/job-engine.ts:startJobOrLeaveToReconciler' origin: none examples: positive: A repeated workspace-provisioning submission returns the existing job. negative: A unique submission key prevents two job records for the same intention. - id: BR-CAP-PFM-OPS-001-04 proved_by: [ATS-PFM-OPS-001-04, ATS-PFM-OPS-001-10, ATS-PFM-OPS-001-13, ATS-PFM-OPS-001-14, ATS-PFM-OPS-001-16, ATS-PFM-OPS-003-13, TST-PFM-OPS-005, TST-PFM-OPS-006, TST-PFM-OPS-007] statement: >- A final job or step cannot transition to another status; late reports for settled jobs are discarded and logged, and another settlement is refused. kind: invariant enforced_at: - service must_prove: true provenance: test source: >- macroservice-libraries:packages/jobs/src/engine/job-engine.spec.ts#refuses settling an already terminal job origin: none - id: BR-CAP-PFM-OPS-001-05 proved_by: [ATS-PFM-OPS-001-05, TST-PFM-OPS-005] statement: 'Job progress cannot decrease, and settlement timestamps are present only on final jobs.' kind: invariant enforced_at: - service - database must_prove: true provenance: test source: >- macroservice-libraries:packages/jobs/src/engine/job-engine.spec.ts#refuses progress that would move backwards origin: none - id: BR-CAP-PFM-OPS-001-06 proved_by: [ATS-PFM-OPS-001-06, TST-PFM-OPS-019] statement: >- A permanent task refusal ends the job with its safe explanation; transient failures are retried without copying their details into the job or execution history. kind: policy enforced_at: - service must_prove: true provenance: test source: >- macroservice-libraries:packages/jobs/src/workflow/job-workflow.spec.ts#lets no detail of the transient error reach the job or the execution state origin: none - id: BR-CAP-PFM-OPS-001-07 proved_by: [] statement: >- Asynchronous job inputs contain identifying references and request context, with secrets and customer content retained outside the execution payload. kind: policy enforced_at: - service must_prove: false provenance: documented source: 'services-platform:docs/jobs.md#What may cross the asynchronous boundary' origin: none - id: BR-CAP-PFM-OPS-001-08 proved_by: [] statement: >- The monitor obtains the job’s step catalog from its published history, preserving the definition used for that execution rather than substituting the current declaration. kind: policy enforced_at: - service must_prove: false provenance: test source: >- services-platform:modules/jobs/tests/integration/services/job-snapshot/job-snapshot.integration.spec.js#takes that catalog from the published event rather than the declaration, which the event itself carries origin: none - id: BR-CAP-PFM-OPS-001-09 proved_by: [ATS-PFM-OPS-001-07, TST-PFM-OPS-077] statement: >- Each registered job kind must declare its business failure consequence or explicitly declare that failure changes nothing outside the job. kind: policy enforced_at: - service must_prove: true provenance: test source: >- services-platform:modules/jobs/tests/unit/job-failure-consequences.spec.js#declares a failure consequence, or states there is none, for every registered kind origin: none - id: BR-CAP-PFM-OPS-001-10 proved_by: [ATS-PFM-OPS-001-08, ATS-PFM-OPS-001-11, ATS-PFM-OPS-001-12, ATS-PFM-OPS-001-17] statement: >- Workspace deletion removes its job records, steps, events and notices while preserving the deletion’s own scheduled and executing jobs. kind: lifecycle enforced_at: - service - database must_prove: true provenance: observed source: 'services-platform:modules/jobs/services/persistence/tenant-purges.ts:JOB_PURGE_RESOURCES' origin: none - id: BR-CAP-PFM-OPS-001-11 proved_by: [] statement: >- Default progress rounds the fraction of settled steps to a percentage capped at one hundred; declared progress weights instead use the highest settled-step weight. kind: calculation enforced_at: - service must_prove: true provenance: observed source: 'macroservice-libraries:packages/jobs/src/engine/job-engine.ts:deriveDeclaredProgress' origin: none - id: BR-CAP-PFM-OPS-001-12 proved_by: [ATS-PFM-OPS-001-15, ATS-PFM-OPS-002-12, TST-PFM-OPS-004] statement: >- Execution continuation records the successor handle and continuation history together, accepts replay of the same move and refuses a source execution that does not match the job. kind: invariant enforced_at: - service must_prove: true provenance: observed source: 'macroservice-libraries:packages/jobs/src/engine/job-engine.ts:recordJobContinuation' origin: none scenarios: - id: SC-CAP-PFM-OPS-001-01 given: A provisioning request already has a recorded submission identity. when: The caller repeats the request using that identity. then: The existing job is returned without another job record. provenance: inferred source: >- macroservice-libraries:packages/jobs/src/engine/job-engine.spec.ts#answers the job the identity already names when the submission is repeated note: >- This illustrative example combines behavior established by the cited test and the capability’s implementation evidence; it is not an independently bound end-to-end proof. - id: SC-CAP-PFM-OPS-001-02 given: An unexpected database fault interrupts a running task. when: The task exhausts its workflow retries. then: The recorded failure remains generic and the technical error is confined to logs. provenance: inferred source: >- macroservice-libraries:packages/jobs/src/workflow/job-workflow.spec.ts#lets no detail of the transient error reach the job or the execution state note: >- This illustrative example combines behavior established by the cited test and the capability’s implementation evidence; it is not an independently bound end-to-end proof. edge_cases: - id: SC-CAP-PFM-OPS-001-03 given: A job has already completed successfully. when: A delayed report marks one of its steps running. then: The engine ignores and logs the report without adding history or raising a regression alert. provenance: inferred source: >- macroservice-libraries:packages/jobs/src/engine/job-engine.spec.ts#raises no terminal-regression alert for a late running report against a succeeded job note: >- This illustrative example combines behavior established by the cited test and the capability’s implementation evidence; it is not an independently bound end-to-end proof. constraints: - 'Cloud workflows own sequencing, retry and timeouts; the engine maintains the observable record.' depends_on: [] related: - CAP-PFM-OPS-002 - CAP-PFM-OPS-004 - CAP-PFM-SEC-003 - CAP-PFM-SEC-004 journeys: - JNY-PFM-OPS-002 open_questions: [] evidence: - >- macroservice-libraries at 54536a4: packages/jobs README, CHANGELOG, persistence/job-persistence-columns.ts, engine/job-engine.ts, engine/job-type-registry.ts, task/job-task-handler.ts, task/job-task-errors.ts, workflow/job-workflow.ts and their spec suites. - >- services-platform at ad3f6a6: docs/jobs.md §1 to §4, §7; modules/jobs/services/job-types/ensure-platform-job-types-registered.ts; modules/jobs/services/persistence/tenant-purges.ts; modules/jobs/tests/unit/job-failure-consequences.spec.js; job-snapshot integration suite. - >- services-platform at ad3f6a6, tables named by this capability: the job root, step and event tables of every Platform family (owned by their modules) through the engine contract. - >- Examination: proposed acceptance matrices retain candidate test references; incomplete proof is recorded by FND-PFM-OPS-028. controls: [TC-AVL-001, TC-AVL-004, TC-AVL-005, TC-AZ-001, TC-AZ-002, TC-AZ-003, TC-AZ-004, TC-CRY-001, TC-CRY-002, TC-CRY-003, TC-CRY-004, TC-DAT-001, TC-DAT-002, TC-DAT-003, TC-DAT-004, TC-IAM-001, TC-IAM-002, TC-IAM-004, TC-IAM-005, TC-IAM-006, TC-LOG-002, TC-LOG-003, TC-LOG-004, TC-LOG-005, TC-MON-001, TC-MON-002, TC-PI-001, TC-PI-002, TC-PI-003, TC-PI-004, TC-PI-005, TC-SEC-001, TC-SEC-002, TC-SEC-003, TC-SEC-004, TC-TEN-001, TC-TEN-002, TC-TEN-003, TC-VEN-001, TC-VEN-002, TC-VEN-003] exercised_by: [FAS-PFM-OPS-01-01, FAS-PFM-OPS-01-02, FAS-PFM-OPS-01-03, FAS-PFM-OPS-01-05] suites: [TST-PFM-OPS-004, TST-PFM-OPS-005, TST-PFM-OPS-006, TST-PFM-OPS-007, TST-PFM-OPS-019, TST-PFM-OPS-077] --- # Complete background work with a durable record ## Purpose Give long-running work a persistent account that operators can inspect and use for recovery. ## How It Works The owning module registers the kind of work, its visible steps and its recovery behavior. A submitting operation creates the job beside the records that require it. Execution starts after that transaction commits. The shared engine records step reports and updates the job summary together. A monitor can then show attempts and outcomes without reconstructing them from logs. When execution cannot begin, the committed job remains available for reconciliation. Permanent refusals provide an operator-safe explanation, while unexpected technical details stay outside the public job record. Recovery belongs to the owning workflow and the shared reconciliation contract. Submission inside a caller transaction can require that caller to retry a conflicting transaction. Progress combines settled steps with declared weights, and execution continuation keeps a traceable successor rather than losing the original handle. ## Relationships The job record supports reconciliation, notices and both monitors; owning modules retain responsibility for the work itself.