---
id: CAP-PFM-DJE-012
title: Import records using their producer identities
feature: FEAT-PFM-DJE-04
product_area: platform
module: data-job-engine
revision: 1
status: draft
product_intent: confirmed
implementation_evidence: partially-observed
supersedes:
  - CAP-DJE-ENG-006
  - CAP-DJE-ENG-008
  - RUL-DJE-ENG-020
source_sha:
  macroservice-libraries: 54536a441935c6e09fdb7f96f0bec71c52ba4e03
  services-crm: a8d1364f1dd44b723bb5f9acba12f983e9ec8d69
  react-console: baf6dae382d6f41dce976c7f848667ccc37e076f
intent_sources:
  - 'services-platform:docs/milestones/PLAT-M19-data-job-engine-foundations.md#D1'
  - 'services-platform:docs/milestones/PLAT-M19-data-job-engine-foundations.md#D5'
  - 'services-platform:docs/milestones/plans/PLAT-M19-plan.md#S12'
  - 'services-platform:docs/milestones/plans/PLAT-M19-plan.md#S13'
  - 'services-platform:docs/milestones/plans/PLAT-M19-plan.md#S14'
  - 'services-crm:docs/milestones/CRM-M7-data-operations.md#D-38'
  - 'services-crm:docs/milestones/CRM-M7-data-operations.md#E-11'
  - 'services-crm:docs/milestones/CRM-M7-data-operations.md#M2-D13'
purpose: >-
  Staff and integrations deliver keyed input that can be reconciled, corrected and completed when its
  dependencies arrive.
outcomes:
  - Producer identities support repeatable import behavior.
  - Records and their owned details commit as one unit.
  - Missing references produce recoverable blocked outcomes.
concepts:
  - CON-PFM-DJE-004
  - CON-PFM-DJE-001
actors:
  - ACT-PFM-036
  - ACT-PFM-035
  - ACT-PFM-037
surfaces:
  - console
  - api
  - job
preconditions: []
postconditions: []
permissions:
  - key: crm.contacts.bulk
    where_checked: api
    scope: tenant
  - key: crm.contacts.update
    where_checked: api
    scope: tenant
  - key: crm.organizations.bulk
    where_checked: api
    scope: tenant
  - key: crm.organizations.update
    where_checked: api
    scope: tenant
  - key: crm.activities.bulk
    where_checked: api
    scope: tenant
  - key: crm.regions.bulk
    where_checked: api
    scope: tenant
  - key: crm.activity-types.bulk
    where_checked: api
    scope: tenant
rules:
  - id: BR-CAP-PFM-DJE-012-01
    proved_by: [ATS-PFM-DJE-006-10, ATS-PFM-DJE-012-01, ATS-PFM-DJE-012-13, ATS-PFM-DJE-012-14, ATS-PFM-DJE-013-06, TST-PFM-DJE-042, TST-PFM-DJE-132]
    statement: >-
      Insert creates an unmapped key and skips a mapped key; upsert creates or replaces a whole record;
      update patches supplied fields and skips satisfied contact changes.
    kind: policy
    enforced_at:
      - service
    must_prove: true
    provenance: test
    source: >-
      services-crm:modules/data-jobs/tests/integration/services/operations/record-imports.integration.spec.js#skips
      every record of the same file delivered a second time, rather than making a second organization
    origin: 'services-platform:docs/milestones/plans/PLAT-M19-plan.md#S13'
  - id: BR-CAP-PFM-DJE-012-02
    proved_by: [ATS-PFM-DJE-012-02, ATS-PFM-DJE-012-15, ATS-PFM-DJE-014-11, TST-PFM-DJE-029]
    statement: >-
      Keyed inserts skip existing mappings; upserts skip lower revisions as stale and equal revisions as
      duplicates, applying higher or unspecified revisions; patches use their separate revision rules.
    kind: policy
    enforced_at:
      - service
    must_prove: true
    provenance: test
    source: >-
      macroservice-libraries:packages/data-jobs/src/tests/integration/keyed-data-jobs.integration.spec.ts#skips
      a delivery a newer one has overtaken, so a late file never undoes the current record
    origin: 'services-platform:docs/milestones/plans/PLAT-M19-plan.md#S13'
  - id: BR-CAP-PFM-DJE-012-03
    proved_by: [ATS-PFM-DJE-003-14, ATS-PFM-DJE-006-10, ATS-PFM-DJE-012-03, ATS-PFM-DJE-012-11, ATS-PFM-DJE-012-13, ATS-PFM-DJE-012-15, ATS-PFM-DJE-012-16, ATS-PFM-DJE-014-11, ATS-PFM-DJE-015-06, TST-PFM-DJE-029]
    statement: >-
      A successful record write and its producer mapping commit together; a refused creation leaves no
      mapping and corrected input can create it.
    kind: invariant
    enforced_at:
      - service
      - database
    must_prove: true
    provenance: test
    source: >-
      macroservice-libraries:packages/data-jobs/src/tests/integration/keyed-data-jobs.integration.spec.ts#writes
      no crosswalk row for an item it refused, so the corrected delivery of that key is a create
    origin: none
  - id: BR-CAP-PFM-DJE-012-04
    proved_by: [ATS-PFM-DJE-004-10, ATS-PFM-DJE-012-04, ATS-PFM-DJE-012-12, ATS-PFM-DJE-012-17, ATS-PFM-DJE-016-15, TST-PFM-DJE-029, TST-PFM-DJE-046, TST-PFM-DJE-047]
    statement: >-
      An unresolved required producer reference blocks its input record, allowing retry after the
      dependency is loaded without uploading the original file again.
    kind: policy
    enforced_at:
      - service
    must_prove: true
    provenance: test
    source: >-
      macroservice-libraries:packages/data-jobs/src/tests/integration/keyed-data-jobs.integration.spec.ts#blocks
      the child, and a retry after the parent is loaded applies it and finishes the job
    origin: 'services-platform:docs/milestones/plans/PLAT-M19-plan.md#S12'
  - id: BR-CAP-PFM-DJE-012-05
    proved_by: [ATS-PFM-DJE-003-14, ATS-PFM-DJE-012-05, ATS-PFM-DJE-012-11, ATS-PFM-DJE-012-14, ATS-PFM-DJE-013-06, TST-PFM-DJE-132]
    statement: >-
      A record and its owned details commit together; an invalid owned address prevents both record
      creation and mapping.
    kind: invariant
    enforced_at:
      - service
      - database
    must_prove: true
    provenance: test
    source: >-
      services-crm:modules/data-jobs/tests/integration/services/operations/record-imports.integration.spec.js#writes
      nothing at all of the rejected organization: not the row, not its addresses, not its mapping
    origin: none
  - id: BR-CAP-PFM-DJE-012-06
    proved_by: [ATS-PFM-DJE-012-06, TST-PFM-DJE-049, TST-PFM-DJE-132]
    statement: >-
      Imported contacts and organizations receive workspace-assigned numbers, ignoring supplied numbers
      and defaulting external identity to the producer key; configuration items retain their supplied
      identifiers.
    kind: policy
    enforced_at:
      - service
    must_prove: true
    provenance: test
    source: >-
      services-crm:modules/data-jobs/tests/integration/services/operations/record-imports.integration.spec.js#keeps
      each producer key as its organization’s external id, the record sending none of its own
    origin: none
  - id: BR-CAP-PFM-DJE-012-07
    proved_by: [ATS-PFM-DJE-012-07, ATS-PFM-DJE-012-17, ATS-PFM-DJE-016-15, TST-PFM-DJE-122]
    statement: >-
      Initial import requires a sandbox workspace in loading state, suppresses only declared side effects
      and preserves validation and derived values while allowing archived options as historical input.
    kind: policy
    enforced_at:
      - service
    must_prove: true
    provenance: test
    source: >-
      services-crm:modules/data-jobs/tests/integration/services/jobs/initial-import-guarantees.integration.spec.js#keeps
      the archived option an initial import carries, as history
    origin: 'services-platform:docs/milestones/PLAT-M19-data-job-engine-foundations.md#D5'
  - id: BR-CAP-PFM-DJE-012-08
    proved_by: [ATS-PFM-DJE-012-08, TST-PFM-DJE-132]
    statement: >-
      Hard deletion removes associated producer mappings so later input can create the record again;
      merge redirects mappings to the survivor.
    kind: policy
    enforced_at:
      - service
      - database
    must_prove: true
    provenance: test
    source: >-
      services-crm:modules/data-jobs/tests/integration/services/operations/record-imports.integration.spec.js#takes
      the keys of the deleted records with them
    origin: none
  - id: BR-CAP-PFM-DJE-012-09
    proved_by: [ATS-PFM-DJE-012-09, TST-PFM-DJE-069, TST-PFM-DJE-122]
    statement: >-
      Record files are prepared before the write transaction; an unsavable file rejects its record while
      storage outages receive transient handling.
    kind: policy
    enforced_at:
      - service
    must_prove: true
    provenance: test
    source: >-
      services-crm:modules/data-jobs/tests/integration/services/jobs/initial-import-guarantees.integration.spec.js#applies
      every organization whose logo could be saved, and rejects the two that could not be written
    origin: none
  - id: BR-CAP-PFM-DJE-012-10
    proved_by: [ATS-PFM-DJE-012-10]
    statement: >-
      The product should offer an automatic-role import option that defaults off; the recorded decision
      is not implemented in the scoped request contracts.
    kind: policy
    enforced_at: []
    must_prove: true
    provenance: documented
    source: 'services-crm:docs/milestones/CRM-M7-data-operations.md#M2-D13'
    origin: 'services-crm:docs/milestones/CRM-M7-data-operations.md#M2-D13'
  - id: BR-CAP-PFM-DJE-012-11
    proved_by: []
    statement: >-
      A correcting file job can identify the earlier job and expose that relationship in both job
      details.
    kind: policy
    enforced_at:
      - api
    must_prove: false
    provenance: documented
    source: 'services-crm:docs/data-jobs.md#The bulk lane'
    origin: none
  - id: BR-CAP-PFM-DJE-012-12
    proved_by: []
    statement: >-
      Imports resolve identity by producer key; a new key can create another person without raising a
      duplicate candidate.
    kind: policy
    enforced_at:
      - service
    must_prove: false
    provenance: documented
    source: 'services-crm:docs/data-jobs.md#What is not built'
    origin: none
  - id: BR-CAP-PFM-DJE-012-13
    proved_by: []
    statement: >-
      Initial activity import preserves historical creation, change and owner information and marks
      migration; other imported records use the current submitter’s stamps.
    kind: policy
    enforced_at:
      - service
    must_prove: false
    provenance: documented
    source: 'services-crm:docs/data-jobs.md#Historical import'
    origin: none
scenarios:
  - id: SC-CAP-PFM-DJE-012-01
    given: 'A producer supplies an organization file through insert, repeated insert, upsert and update jobs.'
    when: The engine processes each operation.
    then: >-
      The passes respectively create, skip mapped identities, replace full records and patch supplied
      values.
    provenance: test
    source: 'react-console:acceptance/crm/playwright/journeys/data-operations/CRM-ACC-0643.spec.ts#CRM-ACC-0643'
  - id: SC-CAP-PFM-DJE-012-02
    given: A file places branches before their head office.
    when: The administrator loads the file and then retries blocked input.
    then: 'Branches initially wait for the head office, then attach after it arrives.'
    provenance: test
    source: 'react-console:acceptance/crm/playwright/journeys/data-operations/CRM-ACC-0647.spec.ts#CRM-ACC-0647'
edge_cases: []
constraints: []
depends_on: []
related:
  - CAP-PFM-DJE-006
  - CAP-PFM-DJE-013
  - CAP-PFM-DJE-014
  - CAP-PFM-DJE-015
journeys:
  - JNY-PFM-DJE-001
  - JNY-PFM-DJE-004
  - JNY-PFM-DJE-005
open_questions:
  - OQ-PFM-DJE-005
evidence:
  - >-
    Reviewed evidence: macroservice-libraries (54536a44): README.md (Keyed writes and the ID crosswalk);
    src/keyed-item-decision.ts, apply-keyed-item.ts, data-id-mappings.ts, group-dependencies.ts; tests
    keyed-data-jobs.integration.spec.ts, group-dependencies.integration.spec.ts,
    prepared-data-jobs.integration.spec.ts.
  - >-
    Reviewed evidence: services-crm (a8d1364f): docs/data-jobs.md (Imports; The crosswalk; Historical
    import; What an import does not keep); services/imports/*; services/operations/contact-import.ts,
    organization-import.ts, configuration-imports.ts, activity-import.ts, record-files.ts;
    services/jobs/initial-import-gate.ts; tests record-imports.integration.spec.js,
    initial-import-guarantees.integration.spec.js, activity-imports.integration.spec.js,
    contact-upsert-affiliations.integration.spec.js.
  - >-
    Reviewed evidence: react-console (baf6dae3): acceptance/crm/journeys/data-operations CRM-ACC-0640 to
    0650.
controls: [TC-AVL-001, TC-AVL-004, TC-AZ-001, TC-AZ-002, TC-AZ-003, TC-AZ-004, TC-CHG-001, TC-CHG-002, TC-CHG-003, TC-CHG-004, TC-CHG-005, TC-CHG-006, TC-CHG-007, TC-CRY-001, TC-CRY-002, TC-CRY-003, TC-CRY-004, TC-DAT-001, TC-DAT-002, TC-DAT-003, TC-DAT-004, TC-IAM-001, TC-IAM-002, TC-IAM-004, TC-IAM-005, TC-LOG-004, TC-LOG-005, TC-MON-001, TC-PI-001, TC-PI-002, TC-PI-003, TC-PI-004, TC-PI-005, TC-PRV-001, TC-PRV-002, TC-PRV-004, TC-TEN-001, TC-TEN-002, TC-TEN-003, TC-TEN-004, TC-VEN-001, TC-VEN-002, TC-VEN-003]
exercised_by: [FAS-PFM-DJE-04-01, FAS-PFM-DJE-04-02, FAS-PFM-DJE-04-03, FAS-PFM-DJE-04-04, FAS-PFM-DJE-04-05]
suites: [TST-PFM-DJE-029, TST-PFM-DJE-042, TST-PFM-DJE-046, TST-PFM-DJE-047, TST-PFM-DJE-049, TST-PFM-DJE-069, TST-PFM-DJE-122, TST-PFM-DJE-132]
---

# Import records using their producer identities

## Purpose

Staff and integrations deliver keyed input that can be reconciled, corrected and completed when its dependencies arrive.

## How It Works

A producer names each input record with its own key and supplies the values for the selected import operation. References can name other producer keys. Rhythm stores the link from each successful key to its destination record. Insert, upsert and patch have distinct repeat-delivery behavior. Owned details are written with their record so an invalid detail does not leave partial creation behind. Missing required references produce blocked outcomes. Staff can retry those outcomes after loading the dependencies. Initial imports use the loading workspace context and declared suppression choices. File preparation takes place before record writes. The automatic-role option remains a recorded product requirement, while duplicate-detection behavior still needs Product’s decision.

## Relationships

- Conflict protection governs later staff edits and revision order.
- Spreadsheet mapping prepares typed input for file processing.
- Producer-key lookup provides reconciliation after import.