--- id: DOM-PFM-DJE-004 title: Source Key realises: CON-PFM-DJE-004 product_area: platform module: data-job-engine revision: 1 status: draft design_intent: confirmed origin: >- services-platform:docs/proposals/rhythm-data-job-engine-plan.md §6.3 matching keys and ID mapping, §6.9 optimistic concurrency (approved by Andrew 2026-10-12); services-platform:docs/standards/record-write-contracts.md#W4 (D-37, D-38) implementation_evidence: observed source_sha: macroservice-libraries: 54536a441935c6e09fdb7f96f0bec71c52ba4e03 services-crm: a8d1364f1dd44b723bb5f9acba12f983e9ec8d69 services-platform: ad3f6a6582843008ed0f5a17b131878d15829af1 services-membership: 5994c270e8a37a30f848e99ba5b5ee6b87c08893 owned_by: SVC-PFM-DJE-engine readable_by: - SVC-PFM-DJE-crm-data-jobs - SVC-PFM-DJE-data-jobs custom_fields: none identity: >- A producer mapping is unique by workspace, producing system, record kind and external key. Identical external text in another system or record category denotes a separate mapping. attributes: - name: source system kind: text required: true computed: false notes: The system name establishes the producer namespace used for matching. - name: record kind kind: enum required: true computed: false notes: The registered record category keeps keys for different kinds separate. - name: producer key kind: text required: true computed: false notes: The external producer supplies its stable record identity. - name: destination record kind: reference required: true computed: true notes: >- The mapping points to the destination that the keyed write created or updated. - name: last applied revision kind: text required: false computed: true notes: >- The last source revision supports numeric, date or lexical delivery ordering. - name: producer-written version kind: number required: false computed: true notes: >- Only a producer write advances its saved destination version; an unchanged result does not adopt a later staff version. - name: last job and timestamps kind: reference required: false computed: true notes: >- First and latest job identities and the latest applied time retain mapping history without foreign keys to retained jobs. relationships: - to: DOM-PFM-DJE-001 cardinality: 'many-to-many: successive jobs can write the same mapping' ownership: The mapping keeps first and last job identifiers without job foreign keys. cross_service: false - to: DOM-PFM-DJE-005 cardinality: 'many-to-many: run reconciliation looks up mappings for multiple datasets' ownership: >- Services own their mappings; workspace reset removes the mappings associated with the load. cross_service: true invariants: - id: INV-DOM-PFM-DJE-004-01 statement: >- The domain write and producer mapping share a transaction, so refusing an item rolls back its mapping too. enforced_at: - service - database provenance: test source: >- macroservice-libraries:packages/data-jobs/src/tests/integration/keyed-data-jobs.integration.spec.ts#writes no crosswalk row for an item it refused, so the corrected delivery of that key is a create - id: INV-DOM-PFM-DJE-004-02 statement: >- Existing mappings are locked, but an unseen key can be created by concurrent jobs because insertion replaces the mapping on conflict. enforced_at: - service - database provenance: observed source: >- macroservice-libraries:packages/data-jobs/src/data-id-mappings.ts:sqlDataIdMappingStore; macroservice-libraries:packages/data-jobs/src/apply-keyed-item.ts:applyDataJobItem - id: INV-DOM-PFM-DJE-004-03 statement: >- The mapping retains the destination version written by its producer rather than adopting an unrelated staff edit. enforced_at: - service provenance: documented source: >- macroservice-libraries:packages/data-jobs/README.md#Keyed writes and the ID crosswalk - id: INV-DOM-PFM-DJE-004-04 statement: >- Whole-record deliveries skip equal revisions as duplicates and lower revisions as stale, while an equal-revision patch can still apply. enforced_at: - service provenance: observed source: >- macroservice-libraries:packages/data-jobs/src/keyed-item-decision.ts:decideKeyedItem - id: INV-DOM-PFM-DJE-004-05 statement: >- Key resolution is confined to the current workspace and registered record category. enforced_at: - database - service provenance: test source: >- services-crm:modules/data-jobs/tests/integration/services/jobs/records-by-source-key.integration.spec.js#looks a key up under its own object type only: a contact key is not an organization - id: INV-DOM-PFM-DJE-004-06 statement: >- Record deletion removes its mappings, merge repoints them to the survivor, and reset clears the load’s mappings. enforced_at: - service - database provenance: test source: >- services-crm:modules/data-jobs/tests/integration/services/operations/record-imports.integration.spec.js#takes the keys of the deleted records with them physical: - data_id_mappings evidence: - >- macroservice-libraries at 54536a44: packages/data-jobs/README.md (Keyed writes and the ID crosswalk); src/data-id-mapping-columns.ts, data-id-mappings.ts, keyed-item-decision.ts, apply-keyed-item.ts. - >- services-crm at a8d1364f: services/persistence/data-id-mappings.ts; services/jobs/records-by-source-key.ts; export/import-crosswalk.dataset.ts. - >- services-platform at ad3f6a65: services/persistence/data-id-mappings.ts; services/jobs/records-by-source-key.ts. - >- services-membership at 5994c270: export/import-crosswalk.dataset.ts (answers that it does not apply). --- # Source Key ## What It Models A source key links an external producer’s record identity to its Rhythm destination. Recurring deliveries use that link to find an existing record, and dependent datasets use it to resolve references. The mapping remains useful after the originating job is retired. ## Identity and Attributes The key combines workspace, producer, record category and external identity. The row keeps the destination, delivery revision, producer-written version and job attribution. Staff do not maintain these links through an editing screen. ## Relationships Keyed jobs consult and update mappings; migration reconciliation reads them through the owning service. Multiple runs can share the same mappings. Job attribution is retained as text without a foreign key, allowing mappings to survive job retention. ## Invariants A successful item commits its destination and mapping together. Existing-key locking and revision comparison protect ordinary repeat deliveries. They leave a first-delivery race: two jobs can create separate destinations and the mapping can end on the last writer. ## How It Is Stored and Shared CRM and Platform each store their mappings under workspace row security. Their application roles may delete mappings, queue entries and submission-ledger rows according to the engine grants. Reconciliation coverage follows the implemented record surfaces; Membership has no mapping store. No custom-field capability is declared for the mapping itself. ## Open Design Questions New-key concurrency needs a protection mechanism. Duplicate detection for imported contacts remains a product choice, and lexical revision ordering requires producer care. Platform custom records and associations still lack producer-key reconciliation endpoints.