--- id: SVC-PFM-DJE-data-jobs title: Platform Data Jobs product_area: platform module: data-job-engine repository: services-platform source_module: modules/data-jobs revision: 1 status: draft design_intent: confirmed origin: >- services-platform:docs/proposals/rhythm-data-job-engine-plan.md (approved 2026-10-12: §6.2 uploads, §12 new-workspace loading, §14 routes, §16 security); PLAT-M19 Product Accepted 2026-10-22. The data-run routes follow PLAT-M22, which is not yet Product Accepted. implementation_evidence: observed inference: [] source_sha: services-platform: ad3f6a6582843008ed0f5a17b131878d15829af1 macroservice-libraries: 54536a441935c6e09fdb7f96f0bec71c52ba4e03 owns: - DOM-PFM-DJE-003 - DOM-PFM-DJE-005 does_not_own: - >- The library defines hosted jobs and producer mappings; Platform stores their instances. - >- Metadata retains ownership of field definitions, custom records and their validation. - 'Workspace lifecycle retains loading, reset and activation authority.' operations: - operation_id: getDataJobsHealth intent: Expose function readiness and deployed-build information. access: public permission: 'none: public health route' realises: [] published_to_submissions: 'not published: operational readiness has no submission contract' - operation_id: openDataJobUpload intent: Begin a workspace input transfer and return its prescribed part sizing. access: console permission: platform.data-jobs.upload realises: - CAP-PFM-DJE-011 published_to_submissions: 'not published: file transfer for job input, not a business capture' - operation_id: presignDataJobUploadParts intent: Issue short-lived signed addresses for requested input parts. access: console permission: platform.data-jobs.upload realises: - CAP-PFM-DJE-011 published_to_submissions: 'not published: file transfer for job input, not a business capture' - operation_id: getDataJobUpload intent: Inspect the session and multipart parts available for resumption. access: console permission: platform.data-jobs.upload realises: - CAP-PFM-DJE-011 published_to_submissions: 'not published: file transfer for job input, not a business capture' - operation_id: completeDataJobUpload intent: >- Assemble input, validate available integrity evidence and save storage facts. access: console permission: platform.data-jobs.upload realises: - CAP-PFM-DJE-011 published_to_submissions: 'not published: file transfer for job input, not a business capture' - operation_id: abortDataJobUpload intent: Abandon multipart input while preserving settled-session restrictions. access: console permission: platform.data-jobs.upload realises: - CAP-PFM-DJE-011 published_to_submissions: 'not published: file transfer for job input, not a business capture' - operation_id: listDataJobUploads intent: Page the workspace’s upload sessions with an optional state filter. access: console permission: platform.data-jobs.upload realises: - CAP-PFM-DJE-011 published_to_submissions: 'not published: file transfer for job input, not a business capture' - operation_id: registerDataRun intent: Check an uploaded package and save dependency-ordered migration steps. access: system permission: 'hq.data-runs.manage (Rhythm staff, system scope)' realises: - CAP-PFM-DJE-016 published_to_submissions: 'not published: Rhythm staff operation outside any workspace intake' - operation_id: listDataRuns intent: Page the workspace’s registered migration attempts. access: system permission: 'hq.data-runs.manage (Rhythm staff, system scope)' realises: - CAP-PFM-DJE-016 published_to_submissions: 'not published: Rhythm staff operation outside any workspace intake' - operation_id: getDataRun intent: Inspect migration progress and advance one pass when the run is active. access: system permission: >- hq.data-runs.manage (Rhythm staff, system scope); the Platform step keys are evaluated for in-process datasets realises: - CAP-PFM-DJE-016 - CAP-PFM-DJE-017 published_to_submissions: 'not published: Rhythm staff operation outside any workspace intake' - operation_id: listDataRunSteps intent: Read recorded dataset progress without advancing execution. access: system permission: 'hq.data-runs.manage (Rhythm staff, system scope)' realises: - CAP-PFM-DJE-016 published_to_submissions: 'not published: Rhythm staff operation outside any workspace intake' - operation_id: startDataRun intent: Begin or advance staff-driven migration execution. access: system permission: >- hq.data-runs.manage (Rhythm staff, system scope); the Platform step keys are evaluated for in-process datasets realises: - CAP-PFM-DJE-016 published_to_submissions: 'not published: Rhythm staff operation outside any workspace intake' - operation_id: pauseDataRun intent: Hold further dataset submissions while active children continue. access: system permission: 'hq.data-runs.manage (Rhythm staff, system scope)' realises: - CAP-PFM-DJE-016 published_to_submissions: 'not published: Rhythm staff operation outside any workspace intake' - operation_id: resumeDataRun intent: Continue a paused migration under current staff authority. access: system permission: >- hq.data-runs.manage (Rhythm staff, system scope); the Platform step keys are evaluated for in-process datasets realises: - CAP-PFM-DJE-016 published_to_submissions: 'not published: Rhythm staff operation outside any workspace intake' - operation_id: cancelDataRun intent: Request child cancellation and settle the parent while retaining its load. access: system permission: >- hq.data-runs.manage (Rhythm staff, system scope); the Platform step keys are evaluated for in-process datasets realises: - CAP-PFM-DJE-016 - CAP-PFM-DJE-017 published_to_submissions: 'not published: Rhythm staff operation outside any workspace intake' - operation_id: submitCustomFieldDataJob intent: Create a custom field job from the input forms this collection supports. access: console permission: >- platform.custom-fields.bulk and one of platform.custom-fields.create or platform.custom-fields.update realises: - CAP-PFM-DJE-001 - CAP-PFM-DJE-018 published_to_submissions: >- not published: staff bulk administration of existing records, not a capture a Submission Form makes - operation_id: listCustomFieldDataJobs intent: Page accessible custom field jobs using the supported filters. access: console permission: >- platform.custom-fields.bulk and one of platform.custom-fields.create or platform.custom-fields.update realises: - CAP-PFM-DJE-005 published_to_submissions: 'not published: a read of job evidence, not a business operation' - operation_id: getCustomFieldDataJob intent: 'Inspect the custom field job’s state, counters and lifecycle details.' access: console permission: >- platform.custom-fields.bulk and one of platform.custom-fields.create or platform.custom-fields.update realises: - CAP-PFM-DJE-005 published_to_submissions: 'not published: a read of job evidence, not a business operation' - operation_id: readCustomFieldsBySourceKey intent: >- Resolve producer keys to current saved custom fields through this collection’s authority checks. access: console permission: >- platform.custom-fields.bulk and one of platform.custom-fields.create or platform.custom-fields.update realises: - CAP-PFM-DJE-015 published_to_submissions: 'not published: a read of job evidence, not a business operation' - operation_id: listCustomFieldDataJobResults intent: Page recorded item outcomes for the custom field job. access: console permission: >- platform.custom-fields.bulk and one of platform.custom-fields.create or platform.custom-fields.update realises: - CAP-PFM-DJE-005 published_to_submissions: 'not published: a read of job evidence, not a business operation' - operation_id: listCustomFieldDataJobRejections intent: Page refusal evidence for the custom field job in input-position order. access: console permission: >- platform.custom-fields.bulk and one of platform.custom-fields.create or platform.custom-fields.update realises: - CAP-PFM-DJE-005 published_to_submissions: 'not published: a read of job evidence, not a business operation' - operation_id: applyCustomFieldDataJob intent: >- Approve a validated custom field job using its saved impact digest and any required deletion phrase. access: console permission: >- platform.custom-fields.bulk and one of platform.custom-fields.create or platform.custom-fields.update realises: - CAP-PFM-DJE-002 - CAP-PFM-DJE-018 published_to_submissions: >- not published: staff bulk administration of existing records, not a capture a Submission Form makes - operation_id: pauseCustomFieldDataJob intent: Hold queued or active custom field work at its batch boundary. access: console permission: >- platform.custom-fields.bulk and one of platform.custom-fields.create or platform.custom-fields.update realises: - CAP-PFM-DJE-004 published_to_submissions: >- not published: staff bulk administration of existing records, not a capture a Submission Form makes - operation_id: resumeCustomFieldDataJob intent: Requeue unfinished batches from a paused custom field job. access: console permission: >- platform.custom-fields.bulk and one of platform.custom-fields.create or platform.custom-fields.update realises: - CAP-PFM-DJE-004 published_to_submissions: >- not published: staff bulk administration of existing records, not a capture a Submission Form makes - operation_id: cancelCustomFieldDataJob intent: Abandon future custom field batches and retain committed changes. access: console permission: >- platform.custom-fields.bulk and one of platform.custom-fields.create or platform.custom-fields.update realises: - CAP-PFM-DJE-004 published_to_submissions: >- not published: staff bulk administration of existing records, not a capture a Submission Form makes - operation_id: retryCustomFieldDataJob intent: Take up eligible paused or dependency-blocked custom field work. access: console permission: >- platform.custom-fields.bulk and one of platform.custom-fields.create or platform.custom-fields.update realises: - CAP-PFM-DJE-004 published_to_submissions: >- not published: staff bulk administration of existing records, not a capture a Submission Form makes - operation_id: submitCustomObjectRecordDataJob intent: >- Create a custom object record job from the input forms this collection supports. access: console permission: >- platform.custom-records.create, scoped by custom object, with platform.custom-records.update evaluated realises: - CAP-PFM-DJE-001 - CAP-PFM-DJE-018 published_to_submissions: >- not published: staff bulk administration of existing records, not a capture a Submission Form makes - operation_id: listCustomObjectRecordDataJobs intent: Page accessible custom object record jobs using the supported filters. access: console permission: >- platform.custom-records.create, scoped by custom object, with platform.custom-records.update evaluated realises: - CAP-PFM-DJE-005 published_to_submissions: 'not published: a read of job evidence, not a business operation' - operation_id: getCustomObjectRecordDataJob intent: >- Inspect the custom object record job’s state, counters and lifecycle details. access: console permission: >- platform.custom-records.create, scoped by custom object, with platform.custom-records.update evaluated realises: - CAP-PFM-DJE-005 published_to_submissions: 'not published: a read of job evidence, not a business operation' - operation_id: listCustomObjectRecordDataJobResults intent: Page recorded item outcomes for the custom object record job. access: console permission: >- platform.custom-records.create, scoped by custom object, with platform.custom-records.update evaluated realises: - CAP-PFM-DJE-005 published_to_submissions: 'not published: a read of job evidence, not a business operation' - operation_id: listCustomObjectRecordDataJobRejections intent: >- Page refusal evidence for the custom object record job in input-position order. access: console permission: >- platform.custom-records.create, scoped by custom object, with platform.custom-records.update evaluated realises: - CAP-PFM-DJE-005 published_to_submissions: 'not published: a read of job evidence, not a business operation' - operation_id: applyCustomObjectRecordDataJob intent: >- Approve a validated custom object record job using its saved impact digest and any required deletion phrase. access: console permission: >- platform.custom-records.create, scoped by custom object, with platform.custom-records.update evaluated realises: - CAP-PFM-DJE-002 - CAP-PFM-DJE-018 published_to_submissions: >- not published: staff bulk administration of existing records, not a capture a Submission Form makes - operation_id: pauseCustomObjectRecordDataJob intent: Hold queued or active custom object record work at its batch boundary. access: console permission: >- platform.custom-records.create, scoped by custom object, with platform.custom-records.update evaluated realises: - CAP-PFM-DJE-004 published_to_submissions: >- not published: staff bulk administration of existing records, not a capture a Submission Form makes - operation_id: resumeCustomObjectRecordDataJob intent: Requeue unfinished batches from a paused custom object record job. access: console permission: >- platform.custom-records.create, scoped by custom object, with platform.custom-records.update evaluated realises: - CAP-PFM-DJE-004 published_to_submissions: >- not published: staff bulk administration of existing records, not a capture a Submission Form makes - operation_id: cancelCustomObjectRecordDataJob intent: Abandon future custom object record batches and retain committed changes. access: console permission: >- platform.custom-records.create, scoped by custom object, with platform.custom-records.update evaluated realises: - CAP-PFM-DJE-004 published_to_submissions: >- not published: staff bulk administration of existing records, not a capture a Submission Form makes - operation_id: retryCustomObjectRecordDataJob intent: Take up eligible paused or dependency-blocked custom object record work. access: console permission: >- platform.custom-records.create, scoped by custom object, with platform.custom-records.update evaluated realises: - CAP-PFM-DJE-004 published_to_submissions: >- not published: staff bulk administration of existing records, not a capture a Submission Form makes events: [] jobs: - name: in-request batch runner kind: 'synchronous, after commit' does: >- Claim and execute queued workspace batches after request commit; there is no external transport or scheduled recovery. - name: platform.data-jobs and platform.data-operations lifecycle participants kind: workspace lifecycle steps does: >- Remove engine rows during reset while retaining package evidence, and purge uploads, runs and storage during deletion. integrations: - system: Data-jobs storage bucket direction: outbound contract: >- Use workspace-scoped multipart objects and prefix retention for input, chunks and exports. - system: CRM data-job routes direction: outbound contract: >- Call CRM dataset submit, progress and cancel surfaces with the current staff token. - system: Workspace lifecycle direction: inbound contract: >- Consult destination loading eligibility and implement lifecycle reset and deletion participants. consistency: - rule: >- Upload bookkeeping follows storage operations across separate persistence boundaries. kind: eventual compensation: >- Bucket abandonment cleans incomplete transfers; storage-first assembly has no established session repair. - rule: Registration commits the run and all ordered steps together. kind: transactional compensation: Registration rollback removes partial parent and step records. - rule: Staff-driven passes update parent progress after child work has executed. kind: eventual compensation: >- Parent failure retains the load; reset removes it rather than automatic compensation. - rule: Local batch processing begins after the request’s submission commit. kind: eventual compensation: >- Request interruption can lose claimed work because no recovery worker is present. evidence: - >- services-platform at ad3f6a65: modules/data-jobs/aws/api/data-jobs.routes.ts, data-runs.routes.ts, custom-field-data-jobs.routes.ts, custom-object-record-data-jobs.routes.ts, *.schemas.ts; aws/logic/data-jobs-api.lambda.ts; aws/resources/data-jobs-storage.s3.ts; services/**; db/**; docs/data-jobs/README.md, uploads.md, data-runs.md, platform-data-operations.md, tenant-reset.md. - >- The 36 operations above are every route the data-jobs function serves at the pinned commit; they match the module's indexed operation catalogue. controls: [TC-AVL-001, TC-AVL-004, TC-AVL-005, TC-AZ-001, TC-AZ-002, TC-AZ-003, TC-AZ-004, TC-AZ-005, TC-AZ-006, TC-CHG-001, TC-CHG-002, TC-CHG-003, TC-CHG-004, TC-CHG-005, TC-CHG-006, TC-CHG-007, TC-CRY-001, TC-CRY-002, TC-CRY-003, TC-CRY-004, TC-DAT-001, TC-DAT-002, TC-DAT-003, TC-DAT-004, TC-DAT-005, TC-DAT-006, TC-IAM-001, TC-IAM-002, TC-IAM-003, TC-IAM-004, TC-IAM-005, TC-IAM-006, TC-LOG-001, TC-LOG-002, TC-LOG-003, TC-LOG-004, TC-LOG-005, TC-MON-001, TC-PI-001, TC-PI-002, TC-PI-003, TC-PI-004, TC-PI-005, TC-PRV-001, TC-PRV-002, TC-PRV-004, TC-TEN-001, TC-TEN-002, TC-TEN-003, TC-TEN-004, TC-VEN-001, TC-VEN-002, TC-VEN-003] --- # Platform Data Jobs ## What This Service Is For Platform supplies input-file transfer and migration-package coordination. It also adopts the engine for its own field definitions and custom records, using bounded request execution. ## Boundary Uploads and migration runs belong to this service contract. Hosted engine records use the shared designs, while metadata retains its business records and saves. Workspace lifecycle remains responsible for loading and release. ## Operations in Words Upload authority covers opening, signing, reading, completing, aborting and listing transfers. Staff system authority covers run registration, inspection and driving controls; reading a running parent can advance it. Custom-field jobs include keyed reconciliation, while custom-record jobs omit that lookup. Neither supports rollback or publishes a Submissions binding. ## Events and Jobs Local requests run engine batches after commit. Lifecycle participants remove engine and package records according to reset or deletion intent. The module declares no business event or model invocation. ## Consistency Guarantees Run registration is atomic within the database. Storage completion, child submission, parent progress and local batch execution have separate boundaries. Interrupted requests and unrecorded remote submissions remain recovery gaps.