--- id: WF-PFM-MTD-012 title: Load custom records and links from a file trigger: The Data Job Engine validates submitted files before apply and writes accepted lines through ordinary persistence. product_area: platform module: metadata revision: 1 status: draft design_intent: proposed implementation_evidence: observed inference: [] source_sha: services-platform: ad3f6a6582843008ed0f5a17b131878d15829af1 implements: - CAP-PFM-MTD-016 - CAP-PFM-MTD-012 - CAP-PFM-MTD-018 moves: - STATE-PFM-MTD-005 - STATE-PFM-MTD-010 crosses: - SVC-PFM-MTD-associations - SVC-PFM-MTD-custom-objects idempotency: A repeated submission key resolves to the existing job. Crosswalk source keys distinguish create from correction, the impact digest guards apply and claimed batches have one active runner. concurrency: A repeated submission key resolves to the existing job. Crosswalk source keys distinguish create from correction, the impact digest guards apply and claimed batches have one active runner. hops: - n: 1 actor_or_service: Data job routes (link surface on the relationship service; record surface on the data-jobs function) action: Link or record job routes validate submission reach and transactionally store job, items, batches and idempotency facts. anchor: services-platform:modules/associations/aws/api/association-data-jobs.routes.ts:associationDataJobRoutes seam: sync on_failure: Reject invalid submission before any job is committed. left_inconsistent: No job or business record change is retained. observer: The initiating person or service and its operation logs. provenance: test source: 'services-platform:tests/acceptance/custom-objects/journeys/data-operations/CO-ACC-0404.spec.js#step 5: a clerk with the create key only has the known key refuse' - n: 2 actor_or_service: Data job engine, in the submitting request action: The engine claims validation batches and evaluates ordinary write rules in process, including grants, types, conditions, references and whole-file uniqueness. anchor: services-platform:modules/custom-objects/services/persistence/custom-object-records.ts:checkRecordWrite seam: sync on_failure: Exhausted transient failures pause work; systemic failures produce an operator-safe failure summary. left_inconsistent: Validation changes job state and results, not records or links. observer: The initiating person or service and its operation logs. provenance: test source: 'services-platform:tests/acceptance/custom-objects/journeys/data-operations/CO-ACC-0400.spec.js#step 3: the rejections name each bad line and its property, never' - n: 3 actor_or_service: Submitter action: The submitter applies the validated impact by returning its digest. anchor: services-platform:modules/associations/aws/api/association-data-jobs.routes.ts:associationDataJobRoutes seam: sync on_failure: Refuse missing or stale digests and retain the validated job. left_inconsistent: No apply work has begun. observer: The initiating person or service and its operation logs. provenance: test source: 'services-platform:tests/acceptance/custom-objects/journeys/data-operations/CO-ACC-0400.spec.js#step 4: an apply with a stale digest is refused 409 impact_digest' - n: 4 actor_or_service: Data job engine, in the applying request action: The in-process engine prepares attachments before batch transactions and applies lines through the ordinary record or link writes. anchor: services-platform:modules/associations/services/persistence/associations.ts:createAssociation seam: sync on_failure: Record line rejections; persistent transient batch failures pause the job, and cancellation settles after current work. left_inconsistent: Committed batches remain; omitted link attributes can clear dates and reopen ended connections. observer: The initiating person or service and its operation logs. provenance: test source: 'services-platform:tests/acceptance/custom-objects/journeys/data-operations/CO-ACC-0400.spec.js#step 6: every created boat reads back as the file said, and the i' - n: 5 actor_or_service: Submitter action: The submitter uses pause, resume, retry or cancel to control remaining job batches. anchor: services-platform:modules/associations/aws/api/association-data-jobs.routes.ts:associationDataJobRoutes seam: sync on_failure: Reject controls incompatible with the job’s current state. left_inconsistent: Existing committed outcomes remain and are reported on cancellation. observer: The initiating person or service and its operation logs. provenance: test source: 'services-platform:tests/acceptance/custom-objects/journeys/data-operations/CO-ACC-0400.spec.js#step 9: the list is newest first with the three jobs, and a resen' branches: - An unknown crosswalk key may adopt the object record already carrying the matching external identity. - Archived load targets require restoration before correction. waits: - A validated file waits for explicit apply with its impact digest. evidence: - 'services-platform at ad3f6a65: modules/associations/aws/api/association-data-jobs.routes.ts; modules/data-jobs/services/jobs/custom-field-data-jobs.ts; modules/data-jobs/services/operations/association-import.ts, custom-object-record-import.ts (read for their contract only; owned by the data job engine module); modules/custom-objects/services/persistence/custom-object-records.ts (recordWriteRules, checkRecordWrite, prepareRecordFiles); docs/data-jobs/platform-data-operations.md; docs/custom-objects.md#Import, export and reset' controls: [TC-AVL-001, TC-AVL-004, TC-AVL-005, TC-AZ-001, TC-AZ-002, TC-AZ-003, TC-AZ-004, TC-DAT-001, TC-DAT-002, TC-DAT-003, TC-DAT-004, TC-IAM-001, TC-IAM-002, TC-IAM-004, TC-IAM-005, TC-LOG-001, TC-LOG-002, TC-LOG-003, TC-LOG-004, TC-LOG-005, TC-MON-001, TC-MON-002, TC-PI-001, TC-PI-002, TC-PI-003, TC-PI-004, TC-PI-005, TC-PRV-001, TC-PRV-002, TC-TEN-001, TC-TEN-002, TC-TEN-003, TC-TEN-004, TC-VEN-001, TC-VEN-002, TC-VEN-003] --- # Load custom records and links from a file ## What It Does The Data Job Engine validates submitted files before apply and writes accepted lines through ordinary persistence. Batch commits survive later failure or cancellation. ## The Flow ```mermaid sequenceDiagram participant Sub as Submitter or data run participant Routes as Data job routes participant Engine as Data job engine (in request) participant Write as Record store or relationship service Sub->>Routes: submit (idempotency key) Routes->>Engine: job, items, batches committed Engine->>Write: prevalidate each line (no writes) Engine-->>Sub: validated impact, digest, rejections Sub->>Routes: apply with digest Routes->>Engine: run batches Engine->>Write: ordinary create or update per line Write-->>Engine: outcome per line Engine-->>Sub: results, rejections, final state ``` ## When Something Fails Exhausted transient failures pause work; systemic failures produce an operator-safe failure summary. Validation changes job state and results, not records or links. Record line rejections; persistent transient batch failures pause the job, and cancellation settles after current work. Committed batches remain; omitted link attributes can clear dates and reopen ended connections. Reject controls incompatible with the job’s current state. Existing committed outcomes remain and are reported on cancellation. ## Open Design Questions The Data Job Engine owns batch lifecycle and controls; this module owns the contributed write contracts.