--- id: DOM-PFM-DJE-001 title: Data Job realises: CON-PFM-DJE-001 product_area: platform module: data-job-engine revision: 1 status: draft design_intent: confirmed origin: >- services-platform:docs/proposals/rhythm-data-job-engine-plan.md (approved by Andrew 2026-10-12: §5.3 a library with per-service tables, §6 the shared job contract, §9.5 receipts); PLAT-M19 decisions D3 to D6 (services-platform:docs/milestones/PLAT-M19-data-job-engine-foundations.md#Decisions, Product Accepted 2026-10-22) implementation_evidence: observed source_sha: macroservice-libraries: 54536a441935c6e09fdb7f96f0bec71c52ba4e03 services-platform: ad3f6a6582843008ed0f5a17b131878d15829af1 services-crm: a8d1364f1dd44b723bb5f9acba12f983e9ec8d69 react-console: baf6dae382d6f41dce976c7f848667ccc37e076f owned_by: SVC-PFM-DJE-engine readable_by: - SVC-PFM-DJE-crm-data-jobs - SVC-PFM-DJE-data-jobs lifecycle: STATE-PFM-DJE-001 custom_fields: none identity: >- The hosting service assigns a job identity within a workspace. Items retain their submission positions and batches retain their sequence numbers. An optional submission key identifies repeated requests by the same caller; CRM purges old keys after 30 days, while Platform has no purge. attributes: - name: record kind kind: enum required: true computed: false notes: >- The registered record category selects the domain operation and its authority checks. - name: operation kind: enum required: true computed: false notes: >- The requested change is update, upsert, insert or delete. Delete declarations cannot offer editable fields or compensation. - name: contract version kind: number required: true computed: false notes: >- The domain accepts a numbered input contract; unsupported versions stop submission. - name: mode kind: enum required: true computed: false notes: >- Validation gathers a preview without saving business records. Apply performs the requested writes. - name: lane kind: enum required: true computed: true notes: >- Request-carried input uses the interactive lane with a 500-item default limit; CRM uploaded input uses the bulk lane. - name: purpose kind: enum required: true computed: false notes: >- Operational work is ordinary processing. An initial import requires a workspace being loaded and may omit only declared side effects. - name: state kind: enum required: true computed: true notes: >- Processing assigns the job state and, where applicable, the reason work paused. - name: submitter kind: reference required: true computed: true notes: >- The initiating person supplies the execution identity. CRM resolves that person’s current grants for each delivery. - name: frozen input kind: collection required: true computed: false notes: >- Saved item positions retain identities, values, optional versions, producer revisions and groups. CRM file jobs retain exact object versions instead of item rows. - name: common change kind: collection required: false computed: false notes: >- A selection may share one patch, constrained to fields that the domain operation permits. - name: conflict policy kind: enum required: false computed: false notes: >- Keyed deliveries default to reject intervening edits; overwrite is an explicit alternative. - name: producer and delivery kind: text required: false computed: false notes: The producing system and delivery description identify the input’s origin. - name: captured versions kind: collection required: false computed: true notes: >- Interactive validation stores the observed record version for each input position. - name: impact and impact fingerprint kind: computed required: false computed: true notes: >- Validation combines batch impacts and records their digest. Deletion adds a confirmation phrase based on the counted impact. - name: counters kind: computed required: true computed: true notes: >- Outcome totals and finished-batch progress summarize durable processing evidence. - name: related jobs kind: reference required: false computed: false notes: >- Correction, revalidation and rollback links connect later work to its original job. - name: bulk progress kind: computed required: false computed: true notes: >- File processing records source, planning, admission and settlement progress and any wait for capacity. - name: timestamps and failure summary kind: date required: true computed: true notes: >- Lifecycle instants record submission and processing progress; optional finish and validation times identify settlement and approval age. Failure text is intended for operators. - name: batches kind: collection required: true computed: true notes: >- Batches retain contiguous input ranges or chunk references, processing state and delivery attempts. Grouping may enlarge a batch beyond its target size. - name: receipts kind: collection required: false computed: true notes: >- A receipt identifies an item that committed or was skipped and records its destination, outcome and any version written. - name: refusals kind: collection required: false computed: true notes: >- Each refusal attempt records a reason, relevant field paths and safe explanatory text without including input values. relationships: - to: DOM-PFM-DJE-003 cardinality: >- many-to-many: jobs can consume multiple uploads, and uploads can serve multiple jobs ownership: The job holds upload references; Platform owns upload storage. cross_service: true - to: DOM-PFM-DJE-004 cardinality: 'one-to-many: a keyed job can read and change multiple mappings' ownership: The adopting service owns durable mappings independently of job retention. cross_service: false - to: DOM-PFM-DJE-005 cardinality: 'many-to-one: a run dataset attempt identifies at most one job' ownership: The run stores the job identity; the destination service owns execution. cross_service: true - to: DOM-PFM-DJE-006 cardinality: 'many-to-one: bulk admissions consume the shared capacity decision' ownership: >- The sentinel owns capacity publication; admission only consumes its factor. cross_service: true - to: DOM-PFM-DJE-001 cardinality: 'one-to-many: subsequent jobs can compensate or revalidate an earlier job' ownership: The adopting service owns the original and related jobs. cross_service: false invariants: - id: INV-DOM-PFM-DJE-001-01 statement: >- Submission preserves the input used by later processing, so an approval does not replace the saved change with a new request. enforced_at: - service - database provenance: test source: >- services-crm:modules/data-jobs/tests/integration/services/jobs/contact-data-jobs.integration.spec.js#freezes the patch on every item, so a later apply writes what was submitted and not what is asked for then - id: INV-DOM-PFM-DJE-001-02 statement: >- A unique item receipt commits with its domain write, preventing a repeated delivery of that job position from writing twice. enforced_at: - service - database provenance: test source: >- services-crm:modules/data-jobs/tests/integration/services/queue/data-job-worker.integration.spec.js#changes each contact once between them, whichever delivery got there first - id: INV-DOM-PFM-DJE-001-03 statement: >- Refusal evidence contains reason codes, field paths, optional constraint names and safe messages without storing business values. enforced_at: - service provenance: test source: >- services-crm:modules/data-jobs/tests/integration/services/jobs/contact-data-jobs.integration.spec.js#names the rule a refused value broke, and quotes no value of the record - id: INV-DOM-PFM-DJE-001-04 statement: >- Settlement derives outcomes from durable evidence and counts remaining positions as not attempted; it does not impose a database equality against the declared item total. enforced_at: - service provenance: observed source: >- macroservice-libraries:packages/data-jobs/src/run-data-job-batch.ts:settleJob - id: INV-DOM-PFM-DJE-001-05 statement: >- Controls consult the transition table, while automatic settlement can pause validating work outside that table. enforced_at: - service provenance: observed source: >- macroservice-libraries:packages/data-jobs/src/job-control.ts; macroservice-libraries:packages/data-jobs/src/run-data-job-batch.ts:settleJob - id: INV-DOM-PFM-DJE-001-06 statement: >- Approval must echo the validation digest before its 24-hour deadline, and deletion must also supply the impact-derived confirmation text. enforced_at: - service provenance: test source: >- services-crm:modules/data-jobs/tests/integration/services/jobs/contact-delete.integration.spec.js#refuses an apply whose confirmation names another number of contacts - id: INV-DOM-PFM-DJE-001-07 statement: >- The engine refuses direct application of a delete and requires an initial validation. enforced_at: - service provenance: test source: >- services-crm:modules/data-jobs/tests/integration/services/jobs/contact-delete.integration.spec.js#refuses a delete submitted to be applied, because a delete is validated first - id: INV-DOM-PFM-DJE-001-08 statement: >- Workspace row security confines engine records to their workspace and conceals foreign jobs as missing. enforced_at: - database - service provenance: test source: >- macroservice-libraries:packages/data-jobs/src/tests/integration/tenant-isolation.integration.spec.ts#does not let another tenant see a job, or learn that it exists - id: INV-DOM-PFM-DJE-001-09 statement: >- A commit group writes or refuses all of its collected members together, but distant or resumed file repetitions can escape that grouping. enforced_at: - service provenance: test source: >- macroservice-libraries:packages/data-jobs/src/tests/integration/group-refusals.integration.spec.ts#writes none of the group, rejects the other two as group_member_refused, and applies the items of no group; FND-PFM-DJE-041 - id: INV-DOM-PFM-DJE-001-10 statement: >- CRM resolves the submitter’s authority for each batch; revoked authority fails the job without reversing earlier commits. enforced_at: - service provenance: test source: >- services-crm:modules/data-jobs/tests/integration/services/queue/data-job-worker.integration.spec.js#fails the whole job with the code, not just the batch - id: INV-DOM-PFM-DJE-001-11 statement: >- Interactive submission commits the job, saved items, planned batches, queued work and any submission key together. enforced_at: - service - database provenance: documented source: >- macroservice-libraries:packages/data-jobs/README.md#A worked example: updating contacts in bulk - id: INV-DOM-PFM-DJE-001-12 statement: >- Every rollback item must carry an expected record version so that applying the compensation cannot discard a subsequent edit. enforced_at: - service provenance: test source: >- macroservice-libraries:packages/data-jobs/src/tests/integration/rollback-and-retry.integration.spec.ts#refuses a rollback plan with an unversioned item, and writes no rollback job physical: - data_jobs - data_job_items - data_job_batches - queued_data_job_batches - data_job_receipts - data_job_rejections - data_job_submissions - 'bulk_data_jobs (CRM, the bulk lane''s durable job)' - bulk_data_job_steps (CRM) - bulk_data_job_events (CRM) evidence: - >- macroservice-libraries at 54536a44: packages/data-jobs/README.md; src/data-job-states.ts, data-job-columns.ts, submit-data-job.ts, job-control.ts, run-data-job-batch.ts, row-level-security.ts, define-data-operation.ts. - >- services-crm at a8d1364f: modules/data-jobs/db/data-jobs.table.ts, bulk-data-jobs.table.ts; services/jobs/contact-data-jobs.ts; docs/data-jobs.md. - >- services-platform at ad3f6a65: modules/data-jobs/db/data-jobs.table.ts; docs/data-jobs/platform-data-operations.md; docs/proposals/rhythm-data-job-engine-plan.md §5.3, §6, §9.5; docs/milestones/PLAT-M19-data-job-engine-foundations.md. - >- react-console at baf6dae3: lib/api/data-jobs/types.ts, rules.ts (what the console reads of a job). - >- services-platform at ad3f6a65: docs/persistence.md, docs/row-level-security.md (the data-job tables row), docs/queued-work.md §2, docs/record-history.md, docs/standards/record-write-contracts.md#W4 (the standards the job's storage and writes follow). --- # Data Job ## What It Models A data job records one requested change across a set of records. Staff use its saved input, progress and outcomes to review a preview, follow an import and account for partial completion. Record outcomes belong to the job as receipts and refusals. The shared library defines the model, while CRM and Platform keep separate instances beside the business records they administer. ## Identity and Attributes Item positions remain stable after submission, allowing an operator to relate a result to the original input. The job combines caller-supplied change data with system-computed lifecycle state, impact, progress and evidence. A submission key recognizes repeated requests while its ledger entry exists; simultaneous first requests can produce a conflict for the losing caller. ## Relationships Completed uploads supply file input. Producer mappings connect external keys to saved records, and migration steps retain the identities of their child jobs. File admission reads the shared capacity factor. Later jobs can correct, revalidate or compensate an earlier job. ## Invariants The structured invariant rows identify the transactional boundaries and their enforcement limits. Item receipts protect redelivery of one job; they do not establish uniqueness across independent jobs using the same new producer key. Controls and automatic settlement also have different transition enforcement. ## How It Is Stored and Shared Each adopter stores the shared engine family under workspace row security. CRM adds durable execution records for its file workflow. The record owner performs each business write and supplies history and change events. CRM job inspection uses its API; Platform migration work can call its local job services directly. These engine records enable no custom fields. ## Open Design Questions Evidence retention remains unenforced, and Platform has neither a submission-key purge nor scheduled recovery for interrupted request execution. Job completion has no notification event. File grouping, producer-key concurrency and the remaining recovery findings require attention before stronger guarantees are accepted.