--- id: MOD-CRM-DQ title: Duplicate Review and Record Merging product_area: crm module: data-quality revision: 1 status: draft source_sha: services-crm: a8d1364f1dd44b723bb5f9acba12f983e9ec8d69 react-console: baf6dae382d6f41dce976c7f848667ccc37e076f repositories: - name: services-crm paths: - modules/duplicates - modules/merges - 'aws/crm.service.ts (duplicates, merges and duplicate-block override permissions)' - docs/merging - docs/milestones/CRM-M4-duplicates-and-merge.md - docs/milestones/CRM-M6-access-and-audit.md (decision D-21 only) - tests/integration/db/merge-coverage.integration.spec.ts - tests/aws/duplicate-rescans.synth.ts - tests/aws/record-merges.synth.ts - name: react-console paths: - 'app/(console)/console/[tenantId]/crm/dataQuality' - 'app/(console)/console/[tenantId]/crm/merges' - components/duplicates - components/console/duplicates - components/records/create/duplicate-block-refusal.tsx - lib/duplicates - acceptance/crm/journeys/merging - acceptance/crm/playwright/journeys/merging - acceptance/crm/inventory (area MRG) owns: - 'Comparison keys, contact and organization candidates and their review groups' - 'Workspace duplicate configuration, creation exceptions and background re-check runs and batches' - Working merge previews and suggested survivor identity - 'Execution attempts, durable-job records, progress and permanent original and resulting values' - 'Absorbed identities, survivor aliases and identity reuse protection' - CRM candidate and declined-suggestion functions for Platform registration does_not_own: - record: 'Contact, organization and affiliation records, their creation refusals and merged-record route responses' owner: MOD-CRM-ENT - record: Tasks and activity records referenced by participants owner: MOD-CRM-ACT - record: Region assignment following record changes owner: MOD-CRM-REG - record: 'Workspace initialization, purge and export coordination' owner: MOD-PFM-TNL - record: 'Portal accounts, login authorization, security audit and login participant execution' owner: MOD-PFM-SEC - record: 'Durable-job infrastructure, shared audit storage and job-monitoring pages' owner: MOD-PFM-OPS - record: Attention delivery and responsibility assignment owner: MOD-PFM-ELG - record: Bulk record creation and update jobs owner: MOD-PFM-DJE - record: Search and saved-view identity resolution owner: MOD-PFM-QRY - record: Application-owned references and participant changes outside CRM owner: Their own modules (for example MOD-ACC-TRX for posted accounting documents) hosted_screens: - route: '/console/{tenantId}/crm/dataQuality' purpose: Overview of duplicate work - route: '/console/{tenantId}/crm/dataQuality/duplicates/{objectType}' purpose: Contact or organization review queue - route: '/console/{tenantId}/crm/dataQuality/duplicates/{objectType}/groups/{groupId}' purpose: Shared selection of a review group - route: '/console/{tenantId}/crm/dataQuality/dismissed' purpose: Previously dismissed review work - route: '/console/{tenantId}/crm/dataQuality/merges' purpose: Workspace merge history - route: '/console/{tenantId}/crm/dataQuality/merges/{mergeId}' purpose: One merge outcome - route: '/console/{tenantId}/crm/merges/new' purpose: Participant selection for a new plan - route: '/console/{tenantId}/crm/merges/plans/{planId}' purpose: 'Preview decisions, confirmation and execution progress' - route: '/console/{tenantId}/configuration/crm/duplicateRules' purpose: Workspace matching configuration - route: Possible-duplicates indicator and Merge action on the contact and organization pages purpose: Entry from a record to duplicate review or merging supersedes: - react-console acceptance inventory area MRG (CAP-CRM-MRG-001 to 023; RUL-CRM-MRG-001 to 076) - react-console acceptance journeys CRM-ACC-0230 to CRM-ACC-0289 (as journey and scenario seeds) evidence: [] coverage_catalog: >- ### Pinned Source Coverage | Measure | Covered | Ratio | |---|---|---| | HTTP operations declared in the two modules' route files (17 duplicates, 13 merges; module health routes excluded) | 30 of 30 | 30/30 | | Table files in `modules/duplicates/db` and `modules/merges/db` (16 physical tables), each named in a capability's evidence | 11 of 11 | 11/11 | | Permission keys of this module in `aws/crm.service.ts` (duplicates read, review, configure; merges read; contacts and organizations merge; the two duplicate-block override keys) | 8 of 8 | 8/8 | | End-to-end specification files of the two modules | 7 of 7 | 7/7 | | Integration and unit files (74 integration, 28 unit), indexed for the referenced evidence | 102 indexed | not exhaustively read | | Execution components and scheduled work | 8 of 8 | 8/8 | | Console pages under crm/dataQuality and crm/merges | 8 of 8 | 8/8 | | Acceptance inventory capabilities of area MRG named in a `supersedes` | 23 of 23 | 23/23 | | Acceptance inventory rules of area MRG folded into rule rows or findings | 76 of 76 | 76/76 | | Merging journeys CRM-ACC-0230 to 0289 bound or absorbed | 60 of 60 | 60/60 | ### Operation Ownership | Operation | Capability | |---|---| | `listDuplicateGroups` | CAP-CRM-DQ-007 | | `getDuplicateGroup` | CAP-CRM-DQ-007 | | `dismissDuplicateGroup` | CAP-CRM-DQ-008 | | `restoreDuplicateGroupDismissal` | CAP-CRM-DQ-008 | | `getDuplicateSummary` | CAP-CRM-DQ-011 | | `listDuplicateCandidates` | CAP-CRM-DQ-007 | | `addManualDuplicateCandidate` | CAP-CRM-DQ-009 | | `getDuplicateCandidate` | CAP-CRM-DQ-007, CAP-CRM-DQ-002 | | `listRecordDuplicateCandidates` | CAP-CRM-DQ-010 | | `dismissDuplicateCandidate` | CAP-CRM-DQ-008 | | `restoreDuplicateCandidate` | CAP-CRM-DQ-008 | | `checkForDuplicates` | CAP-CRM-DQ-004 | | `startDuplicateRescan` | CAP-CRM-DQ-005 | | `listDuplicateRescans` | CAP-CRM-DQ-005 | | `getDuplicateRescan` | CAP-CRM-DQ-005 | | `listDuplicateRules` | CAP-CRM-DQ-003 | | `updateDuplicateRule` | CAP-CRM-DQ-003 | | `createMergePlan` | CAP-CRM-DQ-012 | | `listMergePlans` | CAP-CRM-DQ-012 | | `getMergeLimits` | CAP-CRM-DQ-012 | | `getMergePlan` | CAP-CRM-DQ-012, CAP-CRM-DQ-013 | | `updateMergePlan` | CAP-CRM-DQ-013 | | `validateMergePlan` | CAP-CRM-DQ-013 | | `executeMergePlan` | CAP-CRM-DQ-016 | | `listRecordMerges` | CAP-CRM-DQ-019 | | `getRecordMerge` | CAP-CRM-DQ-016, CAP-CRM-DQ-019 | | `getRecordMergeDetails` | CAP-CRM-DQ-019 | | `listMergesIntoRecord` | CAP-CRM-DQ-019 | | `getMergeIntoRecord` | CAP-CRM-DQ-019 | | `getMergedRecordAlias` | CAP-CRM-DQ-018 | ### Storage Coverage | Table file | Capabilities | |---|---| | `match-keys.table.ts (contact_match_keys, organization_match_keys)` | CAP-CRM-DQ-001 | | `duplicate-candidates.table.ts (contact_duplicate_candidates, organization_duplicate_candidates)` | CAP-CRM-DQ-001, CAP-CRM-DQ-002, CAP-CRM-DQ-008, CAP-CRM-DQ-009 | | `duplicate-groups.table.ts (contact_duplicate_groups, organization_duplicate_groups)` | CAP-CRM-DQ-007 | | `duplicate-rules.table.ts (duplicate_rules)` | CAP-CRM-DQ-003 | | `duplicate-block-overrides.table.ts (duplicate_block_overrides)` | CAP-CRM-DQ-004 | | `duplicate-rescans.table.ts (duplicate_rescans)` | CAP-CRM-DQ-005 | | `duplicate-rescan-batches.table.ts (duplicate_rescan_batches)` | CAP-CRM-DQ-005 | | `record-merge-plans.table.ts (record_merge_plans)` | CAP-CRM-DQ-012, CAP-CRM-DQ-013 | | `record-merges.table.ts (record_merges)` | CAP-CRM-DQ-016, CAP-CRM-DQ-019 | | `record-merge-jobs.table.ts (record_merge_jobs, record_merge_job_steps, record_merge_job_events)` | CAP-CRM-DQ-016 | | `merged-record-aliases.table.ts (merged_record_aliases)` | CAP-CRM-DQ-018 | ### Permission Coverage | Permission key | Capabilities | |---|---| | `crm.duplicates.read` | CAP-CRM-DQ-002, CAP-CRM-DQ-004, CAP-CRM-DQ-007, CAP-CRM-DQ-010, CAP-CRM-DQ-011 | | `crm.duplicates.review` | CAP-CRM-DQ-008, CAP-CRM-DQ-009 | | `crm.duplicates.configure` | CAP-CRM-DQ-003, CAP-CRM-DQ-005 | | `crm.contacts.merge` | CAP-CRM-DQ-010, CAP-CRM-DQ-012 to CAP-CRM-DQ-016 | | `crm.organizations.merge` | CAP-CRM-DQ-010, CAP-CRM-DQ-012, CAP-CRM-DQ-013, CAP-CRM-DQ-015 to CAP-CRM-DQ-017 | | `crm.merges.read` | CAP-CRM-DQ-016, CAP-CRM-DQ-018, CAP-CRM-DQ-019 | | `crm.contacts.create-despite-duplicate-block` | CAP-CRM-DQ-004 (the refusal itself: CAP-CRM-ENT-001) | | `crm.organizations.create-despite-duplicate-block` | CAP-CRM-DQ-004 (the refusal itself: CAP-CRM-ENT-002) | ### End-to-End File Coverage | Spec file | Capabilities | |---|---| | `duplicates/tests/e2e/duplicates.e2e.spec.js` | CAP-CRM-DQ-003, 004, 005, 007, 008, 009, 010 | | `duplicates/tests/e2e/duplicate-groups.e2e.spec.js` | CAP-CRM-DQ-007, 008, 011 | | `duplicates/tests/e2e/organization-block-rules.e2e.spec.js` | CAP-CRM-DQ-004 | | `merges/tests/e2e/merges.e2e.spec.js` | CAP-CRM-DQ-012, 013, 014, 016, 017, 018, 019 | | `merges/tests/e2e/merged-records.e2e.spec.js` | CAP-CRM-DQ-010, 018 | | `merges/tests/e2e/merged-from-history.e2e.spec.js` | CAP-CRM-DQ-019 | | `merges/tests/e2e/business-events.e2e.spec.js` | CAP-CRM-DQ-016 | ### Execution Coverage | Piece | Capabilities | |---|---| | Committed contact and organization change processing | CAP-CRM-DQ-001 | | Re-check delivery, failed-message holding and batch processing | CAP-CRM-DQ-005 | | Re-check publication after creation and on the minute schedule | CAP-CRM-DQ-005 | | Merge job workflow with progress publication, execution and failure settlement | CAP-CRM-DQ-016 | | Five-minute job reconciliation and abandoned-plan expiry | CAP-CRM-DQ-012, CAP-CRM-DQ-016 | | Committed contact-merge event publication through the shared dispatcher | CAP-CRM-DQ-016 | | Declared daily attention refresh for duplicate groups, currently held | CAP-CRM-DQ-011 | | CRM candidate selection and contact creation functions for registration | CAP-CRM-DQ-006 | ### Hosted Pages and Their Capabilities | Route | Capabilities | |---|---| | `/console/{tenantId}/crm/dataQuality` | CAP-CRM-DQ-011 | | `/console/{tenantId}/crm/dataQuality/duplicates/{objectType}` | CAP-CRM-DQ-007 | | `/console/{tenantId}/crm/dataQuality/duplicates/{objectType}/groups/{groupId}` | CAP-CRM-DQ-007, CAP-CRM-DQ-008 | | `/console/{tenantId}/crm/dataQuality/dismissed` | CAP-CRM-DQ-008 | | `/console/{tenantId}/crm/dataQuality/merges` | CAP-CRM-DQ-019 | | `/console/{tenantId}/crm/dataQuality/merges/{mergeId}` | CAP-CRM-DQ-019 | | `/console/{tenantId}/crm/merges/new` | CAP-CRM-DQ-012 | | `/console/{tenantId}/crm/merges/plans/{planId}` | CAP-CRM-DQ-012 to CAP-CRM-DQ-016 | Inventory mappings remain in capability supersedes fields; differences are recorded in the findings register. --- # Duplicate Review and Record Merging ## Purpose Data Quality helps staff recognize repeated people and organizations, review suggestions and carry out informed combinations. It keeps matching evidence, review decisions, working previews and permanent outcomes connected. Older identities retain a route to the survivor and an explanation of the change. ## Boundary The module owns duplicate comparison, review and merge coordination. Entities retains ownership of the compared records and their ordinary lifecycle operations. Activities and regions own their related work, while Platform owns accounts, jobs, audit infrastructure, attention, bulk operations, search and workspace lifecycle. Other applications participate through functions they own rather than having CRM rewrite their records directly. ## Features | Feature | Name | Active Concepts | Capabilities | Journeys | Intent | |---|---|---|---|---|---| | `FEAT-CRM-DQ-01` | Locate and Prevent Duplicate Records | 0 | 6 | 1 | confirmed | | `FEAT-CRM-DQ-02` | Review Suspected Duplicate Records | 1 | 5 | 1 | confirmed | | `FEAT-CRM-DQ-03` | Prepare a Reviewed Merge | 1 | 4 | 1 | confirmed | | `FEAT-CRM-DQ-04` | Execute Record Combinations | 1 | 2 | 2 | confirmed | | `FEAT-CRM-DQ-05` | Follow Absorbed Identities and Outcomes | 0 | 2 | 1 | confirmed | ## Capabilities | Capability | Feature | Outcome | Intent | Implementation | |---|---|---|---|---| | `CAP-CRM-DQ-001` | `FEAT-CRM-DQ-01` | Detect duplicate records after changes | confirmed | observed | | `CAP-CRM-DQ-002` | `FEAT-CRM-DQ-01` | Explain the strength of a duplicate suggestion | confirmed | observed | | `CAP-CRM-DQ-003` | `FEAT-CRM-DQ-01` | Set the response to duplicate matches | confirmed | observed | | `CAP-CRM-DQ-004` | `FEAT-CRM-DQ-01` | Check drafts and account for block overrides | confirmed | observed | | `CAP-CRM-DQ-005` | `FEAT-CRM-DQ-01` | Refresh duplicate evidence across a record type | confirmed | partially-observed | | `CAP-CRM-DQ-006` | `FEAT-CRM-DQ-01` | Reuse a contact during portal registration | proposed | partially-observed | | `CAP-CRM-DQ-007` | `FEAT-CRM-DQ-02` | Review duplicate groups in a shared queue | confirmed | observed | | `CAP-CRM-DQ-008` | `FEAT-CRM-DQ-02` | Dismiss false matches and reopen review | confirmed | observed | | `CAP-CRM-DQ-009` | `FEAT-CRM-DQ-02` | Record a duplicate that matching missed | confirmed | partially-observed | | `CAP-CRM-DQ-010` | `FEAT-CRM-DQ-02` | Review and merge from a record page | confirmed | observed | | `CAP-CRM-DQ-011` | `FEAT-CRM-DQ-02` | Understand the duplicate-review workload | confirmed | partially-observed | | `CAP-CRM-DQ-012` | `FEAT-CRM-DQ-03` | Prepare a merge and choose its survivor | confirmed | observed | | `CAP-CRM-DQ-013` | `FEAT-CRM-DQ-03` | Resolve differences in the merge preview | confirmed | observed | | `CAP-CRM-DQ-014` | `FEAT-CRM-DQ-03` | Preserve the right information on the surviving contact | confirmed | observed | | `CAP-CRM-DQ-015` | `FEAT-CRM-DQ-03` | Keep related records consistent through a merge | confirmed | partially-observed | | `CAP-CRM-DQ-016` | `FEAT-CRM-DQ-04` | Execute the approved merge atomically | confirmed | observed | | `CAP-CRM-DQ-017` | `FEAT-CRM-DQ-04` | Combine organizations and their connections safely | confirmed | observed | | `CAP-CRM-DQ-018` | `FEAT-CRM-DQ-05` | Resolve identities absorbed by a merge | confirmed | partially-observed | | `CAP-CRM-DQ-019` | `FEAT-CRM-DQ-05` | Inspect merge outcomes and attribution | confirmed | partially-observed | ## Journeys | Journey | Feature | Staff Outcome | Intent | Implementation | Browser Policy | |---|---|---|---|---|---| | `JNY-CRM-DQ-001` | `FEAT-CRM-DQ-02` | A reviewer settles duplicate work and revisits a dismissal | confirmed | observed | required | | `JNY-CRM-DQ-002` | `FEAT-CRM-DQ-04` | An administrator combines a sparse contact with a member record | confirmed | observed | required | | `JNY-CRM-DQ-003` | `FEAT-CRM-DQ-03` | An administrator renews approval after records change | confirmed | observed | required | | `JNY-CRM-DQ-004` | `FEAT-CRM-DQ-04` | An administrator unifies duplicate organizations | confirmed | observed | required | | `JNY-CRM-DQ-005` | `FEAT-CRM-DQ-01` | An administrator configures blocking and an accountable exception | confirmed | observed | required | | `JNY-CRM-DQ-006` | `FEAT-CRM-DQ-05` | A colleague follows an absorbed identity and inspects its history | confirmed | observed | required |