---
id: WF-PFM-DJE-001
title: Submit a data job and queue its batches
trigger: >-
  A caller submits record input or a selected-record patch to a CRM data-job
  collection.
product_area: platform
module: data-job-engine
revision: 1
status: draft
design_intent: confirmed
origin: >-
  services-platform:docs/proposals/rhythm-data-job-engine-plan.md §6.4 frozen
  target sets, §6.5 idempotency, §9.2 the small bulk update end to end (approved
  2026-10-12); services-platform:docs/queued-work.md §2 (the queued-work
  standard)
implementation_evidence: observed
inference: []
source_sha:
  macroservice-libraries: 54536a441935c6e09fdb7f96f0bec71c52ba4e03
  services-crm: a8d1364f1dd44b723bb5f9acba12f983e9ec8d69
  services-platform: ad3f6a6582843008ed0f5a17b131878d15829af1
implements:
  - CAP-PFM-DJE-001
  - CAP-PFM-DJE-002
  - CAP-PFM-DJE-007
  - CAP-PFM-DJE-018
moves:
  - STATE-PFM-DJE-001
  - STATE-PFM-DJE-002
crosses:
  - SVC-PFM-DJE-crm-data-jobs
  - SVC-PFM-DJE-engine
hops:
  - 'n': 1
    actor_or_service: Record collection's data-job route (CRM)
    action: >-
      Check bulk authority and an admitted record permission, then choose
      interactive or file processing.
    anchor: 'services-crm:modules/data-jobs/aws/api/data-jobs.routes.ts:dataJobsRoutes'
    seam: sync
    on_failure: Return an authorization or input refusal before submission.
    left_inconsistent: No job is created.
    observer: The request response presents the refusal.
    provenance: observed
    source: >-
      services-crm:modules/data-jobs/aws/api/data-jobs.routes.ts:dataJobAuthorization
  - 'n': 2
    actor_or_service: Engine submission
    action: >-
      Validate the declaration, input version, submit permissions, patch fields,
      item limits and distinct identities; resolve any saved submission key.
    anchor: >-
      macroservice-libraries:packages/data-jobs/src/submit-data-job.ts:submitDataJob
    seam: sync
    on_failure: Return a named refusal for invalid input or mismatched key reuse.
    left_inconsistent: Submission has not persisted new work.
    observer: 'Input refusals identify a field, bound or reason in the response.'
    provenance: test
    source: >-
      services-crm:modules/data-jobs/tests/integration/services/jobs/contact-data-jobs.integration.spec.js#answers
      the second submission with the first job, having written nothing
  - 'n': 3
    actor_or_service: Engine submission
    action: >-
      Preserve positioned input and group-aware batches, then commit job, items,
      queued work and any submission key together.
    anchor: >-
      macroservice-libraries:packages/data-jobs/src/submit-data-job.ts:submitDataJob
    seam: sync
    on_failure: Roll back the submission transaction.
    left_inconsistent: No partial job family remains.
    observer: Failed submission returns an error without adding a job to the list.
    provenance: documented
    source: >-
      macroservice-libraries:packages/data-jobs/README.md#A worked example:
      updating contacts in bulk
  - 'n': 4
    actor_or_service: CRM dispatcher nudge
    action: Request immediate workspace dispatch after commit.
    anchor: >-
      services-crm:modules/data-jobs/services/queue/data-job-dispatch-nudge.ts:nudgeDataJobDispatcher
    seam: event
    on_failure: Ignore the failed nudge and rely on the minute schedule.
    left_inconsistent: Committed queue rows wait for scheduled dispatch.
    observer: Dispatch logging records the attempt.
    provenance: documented
    source: >-
      services-crm:modules/data-jobs/services/queue/data-job-dispatch.ts (module
      comment)
  - 'n': 5
    actor_or_service: CRM dispatcher
    action: >-
      Lease queued entries, publish batch messages outside the transaction and
      delete accepted entries.
    anchor: >-
      macroservice-libraries:packages/data-jobs/src/batch-queue-dispatcher.ts:dispatchDataJobBatches
    seam: schedule
    on_failure: Retain refused publications for lease expiry and later dispatch.
    left_inconsistent: Accepted messages can repeat if their acknowledgement is lost.
    observer: Dispatch logs show publication while job progress awaits processing.
    provenance: observed
    source: >-
      services-crm:modules/data-jobs/services/queue/data-job-dispatch.ts:dispatchQueuedDataJobBatches
  - 'n': 6
    actor_or_service: CRM data-job worker
    action: Resolve the submitter’s context and process the delivered batch.
    anchor: >-
      services-crm:modules/data-jobs/services/queue/data-job-worker.ts:runQueuedDataJobBatches
    seam: queue
    on_failure: >-
      Report unsuccessful messages for delayed redelivery and eventual
      dead-letter handling.
    left_inconsistent: Unbegun work remains pending.
    observer: >-
      Queue alarms identify exhausted delivery and saved job failure provides
      detail.
    provenance: documented
    source: >-
      services-crm:modules/data-jobs/services/queue/data-job-worker.ts (module
      comment)
idempotency: >-
  A saved key and request hash recognize repeated submission by the same caller.
  Missing keys create new jobs. A simultaneous first submission can lose the
  ledger insert and return a conflict naming the winner. Dialog application
  approves its existing validation rather than submitting another job.
concurrency: >-
  Submission uniqueness is database-backed. Dispatch claims lease queue entries
  with skip-locked selection; loss between publication and acknowledgement can
  still repeat a message. Workspace security requires separate dispatch
  transactions.
branches:
  - Uploaded input selects file orchestration at the same collection address.
  - >-
    Platform adopts the engine through request execution instead of CRM
    transport.
  - A completed preview remains available for approval.
waits:
  - A lost nudge leaves work for the next minute dispatch.
evidence:
  - >-
    macroservice-libraries at 54536a44: src/submit-data-job.ts,
    validate-data-job-request.ts, batch-plan.ts, batch-queue-dispatcher.ts;
    README.md.
  - >-
    services-crm at a8d1364f: aws/api/data-jobs.routes.ts,
    data-jobs.idempotency.ts; services/jobs/contact-data-jobs.ts;
    services/queue/data-job-dispatch.ts, data-job-dispatch-nudge.ts,
    data-job-worker.ts; aws/logic/data-jobs.lambda.ts.
controls: [TC-AVL-001, TC-AVL-004, TC-AVL-005, TC-AZ-001, TC-AZ-002, TC-AZ-003, TC-AZ-004, TC-CHG-001, TC-CHG-002, TC-CHG-003, TC-CHG-004, TC-CHG-005, TC-CHG-006, TC-CHG-007, TC-CRY-001, TC-CRY-002, TC-CRY-003, TC-CRY-004, TC-DAT-001, TC-DAT-002, TC-DAT-003, TC-DAT-004, TC-IAM-001, TC-IAM-002, TC-IAM-004, TC-IAM-005, TC-IAM-006, TC-LOG-001, TC-LOG-002, TC-LOG-003, TC-LOG-004, TC-LOG-005, TC-MON-001, TC-MON-002, TC-PI-001, TC-PI-002, TC-PI-003, TC-PI-004, TC-PI-005, TC-PRV-001, TC-PRV-002, TC-PRV-004, TC-TEN-001, TC-TEN-002, TC-TEN-003, TC-TEN-004, TC-VEN-001, TC-VEN-002, TC-VEN-003]
---

# Submit A Data Job And Queue Its Batches

## What It Does

Submission records the requested change and durable work before handing batches to transport. The route chooses request-carried or uploaded input, and the engine preserves the accepted input for later execution.

## The Flow

```mermaid
flowchart TD
  H1["Record collection's data-job route (CRM)"] --> H2
  H2["Engine submission"] --> H3
  H3["Engine submission"] --> H4
  H4["CRM dispatcher nudge"] --> H5
  H5["CRM dispatcher"] --> H6
  H6["CRM data-job worker"]
```

## When Something Fails

Validation failures create no job. Transport failure leaves durable queue work for another dispatch; acknowledgement loss can duplicate delivery, which item receipts handle. Committed business work is never automatically reversed.

## Open Design Questions

The recorded findings identify the remaining limits; this examination does not authorize stronger behavior or change the planning rules.