--- id: STATE-PFM-OPS-001 title: Background job lifecycle product_area: platform module: operations revision: 1 status: draft object: DOM-PFM-OPS-001 design_intent: proposed implementation_evidence: observed source_sha: services-platform: ad3f6a6582843008ed0f5a17b131878d15829af1 macroservice-libraries: 54536a441935c6e09fdb7f96f0bec71c52ba4e03 states: - name: pending meaning: Submission exists with queued steps but no step report; an execution handle may already exist. - name: running meaning: A step report has begun the recorded run. - name: succeeded meaning: The run has a successful final outcome and full progress. - name: failed meaning: The recorded run has failed; recovery creates another job. - name: cancelled meaning: A recorded stop request has been observed and the run is final. transitions: - from: pending to: running trigger: The task handler records the first step report. guard: The job remains open and the step exists. side_effects: - Record the running step. - Stamp the first start time. - Append the step-running event. who: the job engine on behalf of the running task provenance: observed source: 'macroservice-libraries:packages/jobs/src/engine/job-engine.ts:applyStepTransition' - from: running to: succeeded trigger: The settling task finishes its work. guard: The root has not already settled. side_effects: - Append success history. - Set progress to one hundred. - Stamp settlement time. - A notifying kind invokes delivery through its success path. who: the job engine on behalf of the settling task provenance: observed source: 'macroservice-libraries:packages/jobs/src/engine/job-engine.ts:settleJob' - from: running to: failed trigger: A permanent refusal or exhausted retry budget reaches failure handling. guard: The job remains eligible for failed settlement. side_effects: - Record step failure. - Apply the registered effect before ordinary failed settlement. - Append failure history. - Retain a safe failure summary. who: the job engine through the workflow failure task provenance: test source: >- macroservice-libraries:packages/jobs/src/workflow/job-workflow.spec.ts#lets no detail of the transient error reach the job or the execution state - from: running to: failed trigger: >- Reconciliation discovers terminal or repeatedly missing execution, or corrects success with unfinished in-flight steps. guard: The candidate passed the silence threshold and remains open. side_effects: - Fail running and retrying steps. - 'Apply the effect before explicit failure, or after success corrected to failure.' - Append reconciliation history. who: the reconciler provenance: test source: >- services-platform:modules/jobs/tests/integration/lifecycle/job-reconciliation/job-reconciliation.integration.spec.js#settles the job failed, with the reconciled - from: running to: succeeded trigger: Reconciliation observes successful execution without running or retrying steps. guard: 'The job remains open, passed the silence threshold and has no active recorded successor.' side_effects: - Set successful progress to one hundred. - Append reconciliation history. who: the reconciler provenance: observed source: 'macroservice-libraries:packages/jobs/src/reconciler/job-reconciler.ts:applyReconciledOutcome' - from: pending to: failed trigger: An eligible reconciliation pass finds no execution handle. guard: The job is still open after passing the silence threshold. side_effects: - Keep unstarted steps queued. - Record the never-started explanation. - Apply the declared business failure effect. who: the reconciler provenance: test source: >- services-platform:modules/jobs/tests/integration/lifecycle/job-reconciliation/job-reconciliation.integration.spec.js#settles it failed, saying the job never started - from: pending to: cancelled trigger: Cancellation is requested before an execution handle exists. guard: The kind supports cancellation and the reason is nonblank. side_effects: - Record cancellation request and immediate cancelled outcome together. who: the job engine on a caller’s request provenance: test source: >- macroservice-libraries:packages/jobs/src/engine/job-cancellation.spec.ts#refuses a job whose type is not cancellable - from: running to: cancelled trigger: An execution abort or observing task confirms the recorded cancellation request. guard: The job carries a recorded cancellation request. side_effects: - Fail in-flight steps as stopped. - Omit the business failure effect. - Append cancelled history. who: the reconciler or the observing task provenance: test source: >- macroservice-libraries:packages/jobs/src/engine/job-cancellation.spec.ts#settles an execution aborted on request as cancelled, applying no failure consequence invalid: - from: succeeded to: running why: A final job cannot return to active execution; late reports leave the record unchanged. - from: failed to: succeeded why: A late successful step report for a failed job is ignored and raises an operational alert. - from: failed to: pending why: Retry creates a linked job instead of reopening the failed record. - from: cancelled to: running why: A cancelled job has a permanent recorded outcome. - from: running to: pending why: Ordinary step reporting advances pending to running and provides no return to pending. concurrency: >- Transactional version checks arbitrate competing task and reconciler writes. Losing writes reread and derive the change again within a bounded retry budget. Handle recording patches the root rather than replacing concurrent step progress. Platform reserves one reconciler invocation. invariants: - INV-DOM-PFM-OPS-001-02 - INV-DOM-PFM-OPS-001-03 - INV-DOM-PFM-OPS-001-04 - INV-DOM-PFM-OPS-001-05 - INV-DOM-PFM-OPS-001-07 - INV-DOM-PFM-OPS-001-10 evidence: - >- macroservice-libraries at 54536a4: engine/job-engine.ts (applyStepTransition, settleJob, recordStepFailure, settleJobFromReconciliation, cancelJob, settleCancelledJob), reconciler/job-reconciler.ts, persistence/job-persistence-columns.ts. - 'services-platform at ad3f6a6: modules/jobs/tests/integration/lifecycle/job-reconciliation.' - >- codex examination 2026-10-23: checked the pinned source anchors, planning references and enforcement boundaries; unresolved limits are recorded in module findings. --- # Background job lifecycle ## The Lifecycle ```mermaid stateDiagram-v2 [*] --> pending: submission recorded pending --> running: initial step report pending --> failed: no recorded execution pending --> cancelled: unstarted cancellation running --> succeeded: settling task succeeds running --> failed: failed execution or recovery running --> succeeded: confirmed execution success running --> cancelled: recorded stop observed succeeded --> [*] failed --> [*] cancelled --> [*] ``` ## Transitions Explained Submission starts the recorded job pending with all steps queued. The first report makes it running. A settling task records success; failure handling records failure. Cancellation becomes final when its recorded request is observed. Reconciliation can close silent runs from execution evidence, including failing a job that has no recorded execution handle. ## What Cannot Happen Final jobs retain their outcome. A retry creates a separate run, and a late successful step cannot replace a recorded failure. Reconciliation treats running or retrying steps as incompatible with execution success. Its corrected-failure path applies the business effect after settlement rather than before it. ## Concurrency and Timing Platform invokes reconciliation every five minutes and selects jobs silent for fifteen minutes. Missing executions require three trailing probe misses. Candidate bounds, failed probes and failed invocations can extend recovery beyond the usual first eligible pass.