--- id: CAP-CRM-DQ-001 title: Detect duplicate records after changes feature: FEAT-CRM-DQ-01 product_area: crm module: data-quality revision: 1 status: draft product_intent: confirmed implementation_evidence: observed supersedes: - CAP-CRM-MRG-001 - CAP-CRM-MRG-002 source_sha: services-crm: a8d1364f1dd44b723bb5f9acba12f983e9ec8d69 react-console: baf6dae382d6f41dce976c7f848667ccc37e076f intent_sources: - 'services-crm:docs/merging/README.md#Status (architecture and plan approved by Andrew 2026-10-11)' - >- services-crm:docs/merging/decisions.md#Settled by inspection or engineering practice (LLM in detection: none) purpose: >- Staff receive duplicate suggestions as records change, while saving remains independent of the detection work. outcomes: - Background changes produce review suggestions without another staff action. - 'Detection failures do not block saves, imports or integration writes.' - Overly common values do not create an unbounded queue of comparisons. concepts: - CON-CRM-DQ-001 actors: - ACT-CRM-005 - ACT-CRM-001 - ACT-CRM-004 surfaces: - job preconditions: - 'A workspace contact or organization has a committed create, edit or delete.' postconditions: - 'Detection has refreshed the applicable keys and inserted, updated or removed the affected candidates.' permissions: [] rules: - id: BR-CAP-CRM-DQ-001-01 proved_by: [ATS-CRM-DQ-001-01, ATS-CRM-DQ-001-11, ATS-CRM-DQ-001-20, TST-CRM-DQ-176] statement: >- Committed change events trigger background detection; saving neither waits for that work nor fails because detection fails. kind: policy enforced_at: - service must_prove: true provenance: observed source: >- services-crm:modules/duplicates/services/detection/duplicate-change-events.ts:DUPLICATE_DETECTION_HANDLER_NAME origin: 'services-crm:docs/merging/README.md#Status (architecture and plan approved by Andrew 2026-10-11)' - id: BR-CAP-CRM-DQ-001-02 proved_by: [ATS-CRM-DQ-001-02, ATS-CRM-DQ-001-11, ATS-CRM-DQ-001-20, TST-CRM-DQ-176, TST-CRM-DQ-180] statement: >- Detection refreshes keys and scores only for changes to matching names, birth date, spouse, affiliations, organization identity and hierarchy, origin identifiers or contact details, excluding job title, notes, status and region. kind: policy enforced_at: - service must_prove: true provenance: test source: >- services-crm:modules/duplicates/tests/integration/services/detection/duplicate-change-events.integration.spec.js#wrote no key, a job title changing no key and no evidence origin: none examples: positive: 'For example, a coordinator corrects a member''s last name; the member''s pairs are re-scored.' negative: 'By comparison, a coordinator changes the member''s job title; nothing is re-scored.' - id: BR-CAP-CRM-DQ-001-03 proved_by: [ATS-CRM-DQ-001-03, ATS-CRM-DQ-001-20] statement: >- Comparison requires a shared key: contacts use e-mail, phone, phonetic name, phonetic surname with postal code, surname stem with birth date or origin; organizations use normalized name, phonetic name with postal code, non-free-mail domain, phone or origin. kind: calculation enforced_at: - service must_prove: true provenance: test source: >- services-crm:modules/duplicates/tests/unit/services/matching/match-keys.spec.js#names the six contact key types of §3 origin: 'services-crm:docs/merging/README.md#Status (architecture and plan approved by Andrew 2026-10-11)' - id: BR-CAP-CRM-DQ-001-04 proved_by: [ATS-CRM-DQ-001-04, ATS-CRM-DQ-001-20, TST-CRM-DQ-180] statement: >- Lookup skips a workspace key value with more than 200 holders, still uses rarer keys and adds a zero-point reason identifying the skipped key type. kind: calculation enforced_at: - service must_prove: true provenance: test source: >- services-crm:modules/duplicates/tests/integration/services/detection/duplicate-detection.integration.spec.js#still found the two records a rarer key joins, which is the telephone number only they hold origin: none - id: BR-CAP-CRM-DQ-001-05 proved_by: [ATS-CRM-DQ-001-05, ATS-CRM-DQ-001-20, TST-CRM-DQ-180] statement: >- When a key becomes too common, detection removes open detected pairs supported only by it, refreshes pairs with independent evidence and preserves dismissed, confirmed and manual pairs. kind: lifecycle enforced_at: - service must_prove: true provenance: test source: >- services-crm:modules/duplicates/tests/integration/services/detection/duplicate-detection.integration.spec.js#retired the pair the number alone joined, its only evidence being disqualified origin: >- D-47 (cited in services-crm:docs/merging/duplicate-detection.md#3. Match keys; no milestone record holds it, FND-CRM-DQ-018) - id: BR-CAP-CRM-DQ-001-06 proved_by: [ATS-CRM-DQ-001-06, ATS-CRM-DQ-001-12, ATS-CRM-DQ-001-20, TST-CRM-DQ-180] statement: >- Within a detection batch, all records receive their keys before comparison, allowing records created together to match each other. kind: calculation enforced_at: - service must_prove: true provenance: test source: >- services-crm:modules/duplicates/tests/integration/services/detection/duplicate-detection.integration.spec.js#found the two records of the batch in each other, which no lookup outside the batch could have done origin: none - id: BR-CAP-CRM-DQ-001-07 proved_by: [ATS-CRM-DQ-001-07, ATS-CRM-DQ-001-11, ATS-CRM-DQ-001-14, ATS-CRM-DQ-001-20, TST-CRM-DQ-176, TST-CRM-DQ-180] statement: >- An open detected pair disappears when it loses every shared key or scores below 35; dismissed, confirmed, manual and portal-sign-up pairs remain stored. kind: lifecycle enforced_at: - service must_prove: true provenance: test source: >- services-crm:modules/duplicates/tests/integration/services/persistence/duplicate-candidates.integration.spec.js#removes exactly the pairs whose other record no longer scores origin: none - id: BR-CAP-CRM-DQ-001-08 proved_by: [ATS-CRM-DQ-001-08, ATS-CRM-DQ-001-13, ATS-CRM-DQ-001-20, TST-CRM-DQ-180] statement: >- Rescoring updates an existing pair's score, band, evidence and evaluation time without replacing its identity, status, origin, dismissal or first detection time. kind: invariant enforced_at: - service - database must_prove: true provenance: test source: >- services-crm:modules/duplicates/tests/integration/services/persistence/duplicate-candidates.integration.spec.js#keeps the moment the pair was first detected, which no re-score owns origin: none - id: BR-CAP-CRM-DQ-001-09 proved_by: [ATS-CRM-DQ-001-09, ATS-CRM-DQ-001-20, TST-CRM-DQ-180] statement: >- Contact matching excludes addresses and phones linked from an organization, so shared employer details do not establish a duplicate pair. kind: calculation enforced_at: - service must_prove: true provenance: test source: >- services-crm:modules/duplicates/tests/integration/services/detection/duplicate-detection.integration.spec.js#pairs the two people on nothing their organization supplies origin: none - id: BR-CAP-CRM-DQ-001-10 proved_by: [ATS-CRM-DQ-001-01, ATS-CRM-DQ-001-20] statement: >- A failed detection pass returns failures for every event in its batch, allowing the entire batch to be delivered again. kind: lifecycle enforced_at: - service must_prove: true provenance: test source: >- services-crm:modules/duplicates/tests/integration/services/detection/duplicate-change-events.integration.spec.js#fails every event of the group, so the queue delivers the whole group again origin: none - id: BR-CAP-CRM-DQ-001-11 proved_by: [] statement: >- Import-marked changes are reserved for a later completion pass, but current captured events lack that marker and receive ordinary detection. kind: policy enforced_at: - service must_prove: false provenance: test source: >- services-crm:modules/duplicates/tests/unit/services/detection/duplicate-change-events.spec.js#answers no job id for the events the capture writes today, so every one of them is processed origin: none - id: BR-CAP-CRM-DQ-001-12 proved_by: [ATS-CRM-DQ-001-10, ATS-CRM-DQ-001-20, TST-CRM-DQ-180] statement: A detection pass reads and creates pairs only within the workspace of its execution. kind: invariant enforced_at: - service - database must_prove: true provenance: test source: >- services-crm:modules/duplicates/tests/integration/services/detection/duplicate-detection.integration.spec.js#answers a pass over a record of ANOTHER tenant with nothing, the tenant being the execution’s origin: none scenarios: - id: SC-CAP-CRM-DQ-001-01 given: Two records for the same person arrive together. when: The background handler processes their change events. then: 'One pair records their shared names, birth date and address.' provenance: test source: >- services-crm:modules/duplicates/tests/integration/services/detection/duplicate-change-events.integration.spec.js#found the two records in each other, which only a batch whose keys are written first can do - id: SC-CAP-CRM-DQ-001-02 given: 'A previously rare number supports two pairs, one of which also shares an e-mail address.' when: Another holder pushes the number above the frequency ceiling. then: The phone-only pair disappears and the e-mail-supported pair remains. provenance: test source: >- services-crm:modules/duplicates/tests/integration/services/detection/duplicate-detection.integration.spec.js#kept the pair that also shares an e-mail address, by its id edge_cases: - id: SC-CAP-CRM-DQ-001-03 given: Two employees link their work phone and address from their employer. when: Detection evaluates both contacts. then: Those linked details supply neither keys nor duplicate evidence. provenance: test source: >- services-crm:modules/duplicates/tests/integration/services/detection/duplicate-detection.integration.spec.js#keys neither the linked number nor the linked address’s postal code, which are the organization’s constraints: [] depends_on: [] related: - CAP-CRM-DQ-002 - CAP-CRM-DQ-003 - CAP-CRM-DQ-005 - CAP-CRM-DQ-008 journeys: - JNY-CRM-DQ-001 open_questions: [] evidence: - repository: services-crm source_sha: a8d1364f1dd44b723bb5f9acba12f983e9ec8d69 paths: - docs/merging/README.md - docs/merging/decisions.md - modules/duplicates/db/match-keys.table.ts - modules/duplicates/services/detection/duplicate-change-events.ts - modules/duplicates/services/detection/duplicate-detection.ts - modules/duplicates/tests/integration/services/detection/duplicate-change-events.integration.spec.js - modules/duplicates/tests/integration/services/detection/duplicate-detection.integration.spec.js - modules/duplicates/tests/integration/services/persistence/duplicate-candidates.integration.spec.js - modules/duplicates/tests/unit/services/detection/duplicate-change-events.spec.js - modules/duplicates/tests/unit/services/matching/match-keys.spec.js note: 'Static examination connects rule claims to implementation, declared checks and the cited tests.' controls: [TC-CRY-004, TC-DAT-001, TC-DAT-002, TC-DAT-003, TC-IAM-001, TC-IAM-005, TC-IAM-006, TC-LOG-005, TC-MON-001, TC-MON-002, TC-PI-001, TC-PI-002, TC-PI-004, TC-PRV-001, TC-PRV-002, TC-TEN-001, TC-TEN-002, TC-TEN-003] exercised_by: [FAS-CRM-DQ-01-01, FAS-CRM-DQ-01-02, FAS-CRM-DQ-01-03, FAS-CRM-DQ-01-05, FAS-CRM-DQ-01-06] suites: [TST-CRM-DQ-176, TST-CRM-DQ-180] --- # Detect duplicate records after changes ## Purpose Staff receive duplicate suggestions as records change, while saving remains independent of the detection work. ## How It Works Rhythm processes committed record changes in the background. It selects changes to the names, identity information and contact details used for matching. Other edits do not trigger another comparison. Records are found through shared, normalized values rather than a comparison with the entire workspace. A common switchboard number can be disregarded while a rarer value still connects two records. The pass writes all keys for a batch before looking for matches within it. It refreshes evidence on existing pairs and removes detected suggestions that no longer qualify. Human decisions remain separate from the changing resemblance. Linked employer details do not count as a person's own matching information. A failed pass returns errors for the batch so processing can be attempted again. The product should complete the import-completion connection before import changes begin carrying a job marker. ## Relationships Scoring supplies the reasons for each suggestion. Duplicate settings determine the action associated with a match. Re-checks use the same detection process, and review decisions constrain which suggestions may disappear.