--- id: STATE-PFM-DJE-001 title: Data job lifecycle object: DOM-PFM-DJE-001 product_area: platform module: data-job-engine revision: 1 status: draft design_intent: confirmed origin: >- services-platform:docs/proposals/rhythm-data-job-engine-plan.md §6.7 States and §6.10 Cancellation and pause (approved 2026-10-12); PLAT-M19 Product Accepted 2026-10-22 implementation_evidence: observed source_sha: macroservice-libraries: 54536a441935c6e09fdb7f96f0bec71c52ba4e03 services-crm: a8d1364f1dd44b723bb5f9acba12f983e9ec8d69 services-platform: ad3f6a6582843008ed0f5a17b131878d15829af1 states: - name: accepted meaning: >- Declared admission vocabulary; submission currently creates a validating or queued job directly. - name: validating meaning: Checks are running without business-record writes. - name: validated meaning: >- Preview processing has ended; approval remains available until its recorded deadline. - name: queued meaning: Work awaits execution or file admission. - name: running meaning: Batch processing has begun. - name: paused meaning: >- New work is held by an operator request, excessive refusals or exhausted transient attempts; prior commits remain. - name: cancelling meaning: Cancellation awaits completion of work already in flight. - name: completing meaning: >- Declared settlement vocabulary; current batch settlement writes the resulting state directly. - name: succeeded meaning: Processing ends with only applied or skipped outcomes. - name: completed_with_rejections meaning: 'Processing ends with at least one rejection, dependency block or conflict.' - name: cancelled meaning: >- Cancellation ends with committed work retained and unfinished positions counted as not attempted. - name: failed meaning: >- A systemic processing problem ends the job without compensating committed work. - name: expired meaning: A late approval has closed the preview’s application window. transitions: - from: accepted to: validating trigger: Submit a preview guard: Submission validation succeeds side_effects: - plan and enqueue validation batches; begin impact accumulation who: 'Submitter (console, integration or data run)' provenance: observed source: >- macroservice-libraries:packages/data-jobs/src/submit-data-job.ts:submitDataJob - from: accepted to: queued trigger: Submit direct application guard: The operation is not deletion side_effects: - plan batches and queue work; request dispatch after commit who: Submitter provenance: test source: >- services-crm:modules/data-jobs/tests/integration/services/jobs/contact-delete.integration.spec.js#refuses a delete submitted to be applied, because a delete is validated first - from: validating to: validated trigger: Settle the final validation batch guard: No unfinished batch remains side_effects: - >- save combined impact, digest and captured versions; expose a deletion confirmation phrase who: Batch runner provenance: observed source: >- macroservice-libraries:packages/data-jobs/src/run-data-job-batch.ts:settleJob - from: validated to: queued trigger: Approve the saved preview guard: >- Validation is within its 24-hour window and the digest agrees; deletion also requires the matching phrase side_effects: - >- queue the preserved input for application; recalculate finished-batch progress who: Approver holding the job's permissions provenance: test source: >- services-crm:modules/data-jobs/tests/integration/services/jobs/contact-data-jobs.integration.spec.js#refuses an apply of the validated update whose digest is not the one the validation answered - from: validated to: expired trigger: Attempt late approval guard: The preview deadline has passed side_effects: - commit expiry before returning an application refusal who: Approver invoking the expiry check provenance: observed source: >- macroservice-libraries:packages/data-jobs/src/job-control.ts:applyValidatedJob - from: queued to: running trigger: Begin the first apply batch guard: The job permits execution side_effects: - record the processing start who: Worker (CRM) or the submitting request (Platform) provenance: observed source: >- macroservice-libraries:packages/data-jobs/src/run-data-job-batch.ts:runDataJobBatch - from: running to: succeeded trigger: Finish apply processing guard: All positions applied or skipped and none remain unattempted side_effects: - save outcome totals and finish time who: 'Batch runner (bulk lane, the settle task)' provenance: observed source: >- macroservice-libraries:packages/data-jobs/src/data-job-states.ts:finalDataJobState - from: running to: completed_with_rejections trigger: Finish apply processing with unresolved outcomes guard: 'A rejection, block or conflict remains' side_effects: - save totals and expose refusals who: Batch runner provenance: observed source: >- macroservice-libraries:packages/data-jobs/src/data-job-states.ts:finalDataJobState - from: running to: paused trigger: Hold further processing guard: >- An operator requests pause, transient attempts are exhausted, or the refusal rate exceeds its configured threshold side_effects: - record the pause reason; operator pause removes queued work who: Operator or batch runner provenance: observed source: >- macroservice-libraries:packages/data-jobs/src/rejection-guard.ts:rejectionGuardVerdict - from: queued to: paused trigger: Pause before execution guard: The job is queued side_effects: - remove queued work; file orchestration waits who: Operator provenance: test source: >- macroservice-libraries:packages/data-jobs/src/tests/integration/bulk-data-jobs.integration.spec.ts#holds a job paused before its source was read, and runs it to the end once resumed - from: paused to: running trigger: Continue held work guard: The job is paused side_effects: - >- requeue pending, abandoned and transiently failed batches; clear the pause reason who: Operator holding the job's permissions provenance: observed source: 'macroservice-libraries:packages/data-jobs/src/job-control.ts:resumeDataJob' - from: running to: cancelling trigger: Cancel active work guard: At least one batch remains in flight side_effects: - >- remove queued work; abandon pending batches and count their unattempted positions who: Operator provenance: observed source: 'macroservice-libraries:packages/data-jobs/src/job-control.ts:cancelDataJob' - from: queued to: cancelling trigger: Cancel queued work guard: The job has not reached a final state side_effects: - >- remove queued work and abandon pending batches; settle immediately when none are in flight who: Operator holding the job's permissions provenance: observed source: >- macroservice-libraries:packages/data-jobs/src/data-job-states.ts:DATA_JOB_TRANSITIONS - from: paused to: cancelling trigger: Cancel held work guard: The job is paused side_effects: - abandon pending batches and count their unattempted positions who: Operator holding the job's permissions provenance: observed source: 'macroservice-libraries:packages/data-jobs/src/job-control.ts:cancelDataJob' - from: validated to: cancelling trigger: Discard an unused preview guard: The job is validated side_effects: - settle cancellation without any preview business writes who: Operator holding the job's permissions provenance: observed source: >- macroservice-libraries:packages/data-jobs/src/data-job-states.ts:DATA_JOB_TRANSITIONS - from: cancelling to: cancelled trigger: Finish cancellation guard: No batch remains in flight side_effects: - record completion and report committed and unattempted totals who: Batch runner or the cancel itself provenance: test source: >- services-crm:modules/data-jobs/tests/integration/services/jobs/contact-data-jobs.integration.spec.js#ends the job cancelled, not failed, once the batch it waited for has finished - from: completed_with_rejections to: queued trigger: Retry dependency-blocked items guard: Blocked items remain and no applied rollback prevents retry side_effects: - >- requeue batches containing blocked items; receipts protect earlier writes who: Operator holding the job's permissions provenance: test source: >- macroservice-libraries:packages/data-jobs/src/tests/integration/keyed-data-jobs.integration.spec.ts#blocks the child, and a retry after the parent is loaded applies it and finishes the job - from: running to: failed trigger: End work after a systemic problem guard: The problem belongs to processing rather than one item side_effects: - >- mark remaining work unattempted; retain commits and save safe failure detail who: 'Batch execution, file failure handling or reconciliation' provenance: test source: >- services-crm:modules/data-jobs/tests/integration/services/jobs/bulk-data-job-failure.integration.spec.js#fails the data job, so its status read answers failed and a Data Run polling it stops - from: completing to: succeeded trigger: Complete declared settlement guard: No unresolved or unattempted outcome remains side_effects: - record outcome totals who: Batch runner provenance: observed source: >- macroservice-libraries:packages/data-jobs/src/data-job-states.ts:DATA_JOB_TRANSITIONS - from: validating to: paused trigger: Pause a preview through the guard guard: Unfinished batches remain and the refusal-rate threshold is exceeded side_effects: - 'record guard pause directly, outside the declared transition table' who: Batch settlement provenance: observed source: >- macroservice-libraries:packages/data-jobs/src/run-data-job-batch.ts:settleJob invalid: - from: succeeded to: queued why: >- A fully successful job has no unfinished work to retry; correcting input creates another job. - from: completed_with_rejections to: queued why: >- Retry requires blocked work and is unavailable once a qualifying rollback has applied. - from: validated to: running why: Application approval queues the preserved input before batch execution. - from: expired to: queued why: Expired previews require a new validation. - from: cancelled to: running why: Cancelled work cannot return to processing. - from: failed to: running why: Failed work has no resume transition. concurrency: >- Controls lock the job row while checking their requested move. A batch that has already begun may finish after pause or cancellation. Receipt keys and conditional batch completion prevent repeated delivery from duplicating item writes or finished-batch progress. Automatic settlement locks the job but does not consult the transition table. invariants: - INV-DOM-PFM-DJE-001-05 - INV-DOM-PFM-DJE-001-06 - INV-DOM-PFM-DJE-001-07 - INV-DOM-PFM-DJE-001-10 evidence: - >- macroservice-libraries at 54536a44: packages/data-jobs/src/data-job-states.ts, job-control.ts, run-data-job-batch.ts, rejection-guard.ts, bulk-data-job-tasks.ts. - >- services-crm at a8d1364f: tests/integration/services/jobs/contact-data-jobs.integration.spec.js, bulk-data-job-failure.integration.spec.js. --- # Data Job Lifecycle ## The Lifecycle ```mermaid stateDiagram-v2 [*] --> validating [*] --> queued validating --> validated: settle the final validation batch validated --> queued: approve the saved preview validated --> expired: attempt late approval queued --> running: begin the first apply batch running --> succeeded: finish apply processing running --> completed_with_rejections: finish apply processing with unresolved outcomes running --> paused: hold further processing queued --> paused: pause before execution paused --> running: continue held work running --> cancelling: cancel active work queued --> cancelling: cancel queued work paused --> cancelling: cancel held work validated --> cancelling: discard an unused preview cancelling --> cancelled: finish cancellation completed_with_rejections --> queued: retry dependency-blocked items running --> failed: end work after a systemic problem validating --> paused: pause a preview through the guard ``` ## Transitions Explained Submission creates validating or queued work directly; accepted is declared vocabulary. Validation saves impact and versions without business writes. Approval checks the saved digest, deadline and any deletion phrase before requeuing the same input. Settlement writes the resulting state directly rather than persisting completing. Operator pause, retry and cancellation use the transition table. Automatic refusal-rate handling can pause a validation even though the declared table excludes that move. Blocked work may reopen a job that completed with rejections. ## What Cannot Happen Controls refuse final-state resumes and late preview application. Operator pause is unavailable during validation, but the automatic guard has a separate path. Cancellation preserves earlier business commits. ## Concurrency and Timing Controls and settlement use the job lock, while execution stops at batch boundaries. Preview expiry becomes visible when a late approval or explicit expiry action records it; no scheduled sweep closes previews.