--- id: ARCH-CRM-03 title: 'Rhythm 2.0: Data Enrichment Architecture' product_area: crm revision: 1 status: proposed source: artifact_url: https://claude.ai/artifact/KHdGrMkqzRaNWJxXCLrTqo artifact_title: 'Rhythm 2.0: Data Enrichment Architecture' library_group: 'CRM and people' library_status: 'Decided in part' artifact_version: 3 markdown_source: 'services-platform:docs/proposals/enrichment@ee02d1a554fa9543feff4cba3805c0c99c6e50fc' converted_at: 2026-10-23 converted_by: 'Forge 2.0 artifact strategy session' contents: - n: 1 title: 'Rhythm 2.0: Data Enrichment Architecture' anchor: '1-rhythm-2-0-data-enrichment-architecture' - n: 2 title: '01. Executive summary and recommendation' anchor: '2-01-executive-summary-and-recommendation' - n: 3 title: '02. Market and competitive analysis' anchor: '3-02-market-and-competitive-analysis' - n: 4 title: '03. Enrichment provider comparison' anchor: '4-03-enrichment-provider-comparison' - n: 5 title: '04. Architectural principles' anchor: '5-04-architectural-principles' - n: 6 title: '05. Service ownership and Integration Platform alignment' anchor: '6-05-service-ownership-and-integration-platform-alignment' - n: 7 title: '06. Core workflows and use cases' anchor: '7-06-core-workflows-and-use-cases' - n: 8 title: '07. Provider contracts and orchestration' anchor: '8-07-provider-contracts-and-orchestration' - n: 9 title: '08. Data model, provenance and policies' anchor: '9-08-data-model-provenance-and-policies' - n: 10 title: '09. Security, privacy and licensing' anchor: '10-09-security-privacy-and-licensing' - n: 11 title: '10. API and event architecture' anchor: '11-10-api-and-event-architecture' - n: 12 title: '11. Console and portal experience' anchor: '12-11-console-and-portal-experience' - n: 13 title: '12. Commercial model and economics' anchor: '13-12-commercial-model-and-economics' - n: 14 title: '13. Phased implementation milestones' anchor: '14-13-phased-implementation-milestones' - n: 15 title: '14. Alternatives and tradeoffs' anchor: '15-14-alternatives-and-tradeoffs' - n: 16 title: '15. Open architectural decisions' anchor: '16-15-open-architectural-decisions' - n: 17 title: '16. Sources, research dates and how this document was produced' anchor: '17-16-sources-research-dates-and-how-this-document-was-produced' - n: 18 title: 'R1. The Integration Platform as built: evidence for a Data Enrichment design' anchor: '18-r1-the-integration-platform-as-built-evidence-for-a-data-enrichment-design' - n: 19 title: 'Evidence r2-crm: Rhythm 2.0 CRM as the target of Data Enrichment' anchor: '19-evidence-r2-crm-rhythm-2-0-crm-as-the-target-of-data-enrichment' - n: 20 title: 'R3: Platform core, evidence for the Data Enrichment architecture' anchor: '20-r3-platform-core-evidence-for-the-data-enrichment-architecture' - n: 21 title: 'R4. Data operations (Data Job Engine) and the query layer: evidence for Rhythm 2.0 Data Enrichment' anchor: '21-r4-data-operations-data-job-engine-and-the-query-layer-evidence-for-rhythm-2-0-data-enrichment' - n: 22 title: 'R5. Rhythm 1.0 FullContact and Google Civic enrichment, and the customer-360 precedent' anchor: '22-r5-rhythm-1-0-fullcontact-and-google-civic-enrichment-and-the-customer-360-precedent' - n: 23 title: 'W1 Evidence: How association management systems (and the nearest nonprofit CRMs) handle data enrichment and data quality' anchor: '23-w1-evidence-how-association-management-systems-and-the-nearest-nonprofit-crms-handle-data-enrichment-and-data-quality' - n: 24 title: 'W2 evidence: how enterprise CRM/ERP platforms do data enrichment' anchor: '24-w2-evidence-how-enterprise-crm-erp-platforms-do-data-enrichment' - n: 25 title: 'W3 Evidence: Enrichment Provider Landscape for an Association Management System' anchor: '25-w3-evidence-enrichment-provider-landscape-for-an-association-management-system' - n: 26 title: 'W4 Evidence Report: Privacy Law and Data-Licensing Constraints on Rhythm 2.0 Data Enrichment' anchor: '26-w4-evidence-report-privacy-law-and-data-licensing-constraints-on-rhythm-2-0-data-enrichment' - n: 27 title: 'W5. Supplementary provider checks (read directly, 2026-10-22, 11:00 to 11:30 ET)' anchor: '27-w5-supplementary-provider-checks-read-directly-2026-10-22-11-00-to-11-30-et' decisions: [] modules: [] superseded_by: --- # Rhythm 2.0: Data Enrichment Architecture ## What This Record Is Proposes how Rhythm 2.0 enriches contact and organization records from external data providers for association staff: provider answers are stored as claims rather than facts, staff review suggestions before anything changes, CRM (customer relationship management) keeps every write, and spend is bounded per association. Proposed on 2026-10-22 as RHYTHM-ENR-ARCH-001; the same day Andrew ruled the six decisions that gate the first milestone (enrichment lives inside CRM, tier 3 personal-identity data offered from ENR-M1 behind attestation, the other four as recommended), six decisions remain open and nothing is built. (Summary from the Rhythm 2.0 Architecture Library card; the parts below are the proposal's own text, converted verbatim.) ## Contents 1. [Rhythm 2.0: Data Enrichment Architecture](#1-rhythm-2-0-data-enrichment-architecture) 2. [01. Executive summary and recommendation](#2-01-executive-summary-and-recommendation) 3. [02. Market and competitive analysis](#3-02-market-and-competitive-analysis) 4. [03. Enrichment provider comparison](#4-03-enrichment-provider-comparison) 5. [04. Architectural principles](#5-04-architectural-principles) 6. [05. Service ownership and Integration Platform alignment](#6-05-service-ownership-and-integration-platform-alignment) 7. [06. Core workflows and use cases](#7-06-core-workflows-and-use-cases) 8. [07. Provider contracts and orchestration](#8-07-provider-contracts-and-orchestration) 9. [08. Data model, provenance and policies](#9-08-data-model-provenance-and-policies) 10. [09. Security, privacy and licensing](#10-09-security-privacy-and-licensing) 11. [10. API and event architecture](#11-10-api-and-event-architecture) 12. [11. Console and portal experience](#12-11-console-and-portal-experience) 13. [12. Commercial model and economics](#13-12-commercial-model-and-economics) 14. [13. Phased implementation milestones](#14-13-phased-implementation-milestones) 15. [14. Alternatives and tradeoffs](#15-14-alternatives-and-tradeoffs) 16. [15. Open architectural decisions](#16-15-open-architectural-decisions) 17. [16. Sources, research dates and how this document was produced](#17-16-sources-research-dates-and-how-this-document-was-produced) 18. [R1. The Integration Platform as built: evidence for a Data Enrichment design](#18-r1-the-integration-platform-as-built-evidence-for-a-data-enrichment-design) 19. [Evidence r2-crm: Rhythm 2.0 CRM as the target of Data Enrichment](#19-evidence-r2-crm-rhythm-2-0-crm-as-the-target-of-data-enrichment) 20. [R3: Platform core, evidence for the Data Enrichment architecture](#20-r3-platform-core-evidence-for-the-data-enrichment-architecture) 21. [R4. Data operations (Data Job Engine) and the query layer: evidence for Rhythm 2.0 Data Enrichment](#21-r4-data-operations-data-job-engine-and-the-query-layer-evidence-for-rhythm-2-0-data-enrichment) 22. [R5. Rhythm 1.0 FullContact and Google Civic enrichment, and the customer-360 precedent](#22-r5-rhythm-1-0-fullcontact-and-google-civic-enrichment-and-the-customer-360-precedent) 23. [W1 Evidence: How association management systems (and the nearest nonprofit CRMs) handle data enrichment and data quality](#23-w1-evidence-how-association-management-systems-and-the-nearest-nonprofit-crms-handle-data-enrichment-and-data-quality) 24. [W2 evidence: how enterprise CRM/ERP platforms do data enrichment](#24-w2-evidence-how-enterprise-crm-erp-platforms-do-data-enrichment) 25. [W3 Evidence: Enrichment Provider Landscape for an Association Management System](#25-w3-evidence-enrichment-provider-landscape-for-an-association-management-system) 26. [W4 Evidence Report: Privacy Law and Data-Licensing Constraints on Rhythm 2.0 Data Enrichment](#26-w4-evidence-report-privacy-law-and-data-licensing-constraints-on-rhythm-2-0-data-enrichment) 27. [W5. Supplementary provider checks (read directly, 2026-10-22, 11:00 to 11:30 ET)](#27-w5-supplementary-provider-checks-read-directly-2026-10-22-11-00-to-11-30-et) ## 1. Rhythm 2.0: Data Enrichment Architecture ## Rhythm 2.0: Data Enrichment Architecture **Recommendation: make enrichment a Platform module, treat provider answers as claims rather than facts, and keep all record writes in CRM.** Start with one provider and a small, reviewable workflow. The recommendations below remain open to Andrew's approval or rejection. Status: **proposed 2026-10-22; the six decisions that gate ENR-M1 were ruled by Andrew the same day (2026-10-22 about 11:50 ET): ENR-D1 overturned (enrichment lives inside CRM), ENR-D2, D3, D4 and D8 as recommended, ENR-D6 overturned (tier 3 offered from ENR-M1 behind attestation); six decisions remain open; nothing built: no code, migration, console or portal screen, legacy repository, other proposal or milestone record was changed.** No code, migration, console or portal screen, legacy repository, other proposal or milestone record was changed. Nothing in this proposal is decided or built. The package is on branch `enrichment-architecture` of services-platform, in `docs/proposals/enrichment/`. It follows the precedent of the Integration Platform, CRM, Households and Intelligence packages. Andrew requested it in his brief of 2026-10-22, "Rhythm 2.0 Enrichment Architecture". Cite the package as **RHYTHM-ENR-ARCH-001**, with chapter and section, for example: "RHYTHM-ENR-ARCH-001 §08 2.5". **Labels used throughout.** These labels distinguish evidence, existing decisions and proposed design. Chapter 16 defines them: - **Verified**: read in code, DDL (database schema definitions), a test, a repository document or a vendor's primary page at the cited place on 2026-10-22. - **Planned**: recorded in a proposal or milestone record, not built. - **Decided** / **Ruled**: an Andrew decision, dated. - **Market**: an external fact with a dated URL in an evidence report. - **Published** / **Derived** / **Reported** / **Assumed**: qualifications on economics figures. - **Inferred**: a conclusion drawn from verified facts. - **Recommended**: this package's design, open to overturn. - **Required**: a requirement of the brief or a standing Rhythm 2.0 rule. - **Estimate**: an estimated size or effort. - **Absent** / **Unavailable**: distinctions defined in chapter 16. ### The recommendation in four lines 1. Enrichment is a **Platform module** with its own application key. Providers are ordinary Integration Platform adapters behind a Rhythm-owned **claim vocabulary**, a common language for what providers report. CRM keeps every write (ENR-D1, ENR-D2). 2. A provider's answer is a **claim**, never a fact. Claims, suggestions and the record are three separate layers. Trusted values are never overwritten silently; an employer change is always a proposal a person approves (ENR-D3, ENR-D10). 3. Every call belongs to one association. Keys, caches, suppressions and budgets are scoped to that association. Every value carries a licence class, and access tiers require attestations. Start with bring-your-own-key access; offer Rhythm-managed credits only after signed terms (ENR-D4, ENR-D6, ENR-D8). 4. The first milestone, **ENR-M1**, covers one provider, contacts first (organizations read-only until ENR-M2), manual and on-create triggers, a review panel and a budget. It is estimated at about 18 to 24 agent-days and needs nothing from CRM-M15. ### The documents The package separates the recommendation, supporting evidence, design and open decisions. The evidence reports provide the underlying code and external-source citations. | # | Document | Brief section | | --- | --------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | 01 | [Executive summary and recommendation](#2-01-executive-summary-and-recommendation) | 11.1 | | 02 | [Market and competitive analysis](#3-02-market-and-competitive-analysis) | 2, 11.2 | | 03 | [Enrichment provider comparison](#4-03-enrichment-provider-comparison) | 3, 11.3 | | 04 | [Architectural principles](#5-04-architectural-principles) | 11.4 | | 05 | [Service ownership and Integration Platform alignment](#6-05-service-ownership-and-integration-platform-alignment) | 1, 5, 11.5 | | 06 | [Core workflows and use cases](#7-06-core-workflows-and-use-cases) | 4 (triggers, affiliations, identity), 6 (five sequence diagrams), 11.6 | | 07 | [Provider contracts and orchestration](#8-07-provider-contracts-and-orchestration) | 4 (provider abstraction, orchestration), 6, 11.7 | | 08 | [Data model, provenance and policies](#9-08-data-model-provenance-and-policies) | 4 (field-level, audit, tenant administration), 6, 11.8 | | 09 | [Security, privacy and licensing](#10-09-security-privacy-and-licensing) | 7, 11.9 | | 10 | [API and event architecture](#11-10-api-and-event-architecture) | 6, 11.10 | | 11 | [Console and portal experience](#12-11-console-and-portal-experience) | 9, 11.11 | | 12 | [Commercial model and economics](#13-12-commercial-model-and-economics) | 8, 11.12 | | 13 | [Phased implementation milestones](#14-13-phased-implementation-milestones) | 10, 11.13 | | 14 | [Alternatives and tradeoffs](#15-14-alternatives-and-tradeoffs) | 11.14 | | 15 | [Open architectural decisions](#16-15-open-architectural-decisions) | 10, 11.15 | | 16 | [Sources, research dates and method](#17-16-sources-research-dates-and-how-this-document-was-produced) | 11.16 | | - | [evidence/](evidence/) | r1 to r5 code investigations with `path:line` citations; w1 to w5 market, provider and legal research with URLs and access dates | ### Decisions for Andrew All twelve decisions remain open. No approval is requested on this page. Each decision is written out in full in [15](#16-15-open-architectural-decisions), including what it governs, why it matters now, the options and tradeoffs, and the recommendation with its reasoning. The gates column identifies the milestones that depend on the decision. | # | Decision | Recommendation | Gates | | ------- | ------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | ENR-D1 | Where enrichment lives | Recommended a Platform module; **ruled: inside CRM** (services-crm `modules/enrichment`, schema `crm`); see [05 §8](#6-05-service-ownership-and-integration-platform-alignment) | ENR-M1 | | ENR-D2 | The provider contract vocabulary | A new `enrichment` category with Rhythm-owned claims; **ruled as recommended** | ENR-M1 | | ENR-D3 | The default apply policy | Suggest for people; auto-fill organization brand and identifiers; never overwrite; employer changes always reviewed ; **ruled as recommended** | ENR-M1 | | ENR-D4 | Commercial model and first provider | Bring your own key; PDL first; Rhythm-managed credits after signed OEM terms; baseline included ; **ruled as recommended** | ENR-M1 | | ENR-D5 | Photos | Preview on receipt; copy through the asset pipeline on acceptance where licensed; no biometrics | ENR-M2 | | ENR-D6 | Personal-identity data (tier 3) in year one | Recommended declared, not offered; **ruled: offered from ENR-M1 behind attestation**; see [13 §2](#14-13-phased-implementation-milestones) | ENR-M1 | | ENR-D7 | Member transparency and control | A sentence and a "keep my record as I enter it" switch in the portal; staff suppression from ENR-M1 | ENR-M4 | | ENR-D8 | Licensing and packaging | Application key; baseline granted to every workspace; premium licensed ; **ruled as recommended** (CRM registers the application key) | ENR-M1 | | ENR-D9 | AI-assisted research | Keep the plan step kind; build ENR-M6 as opt-in, one record at a time, always reviewed | ENR-M6 | | ENR-D10 | Employment changes | Reviewed by the data-quality responsibility; never automatic | ENR-M1 | | ENR-D11 | Refresh cadence and retention | Claims 24 months; suggestions 90 days; refresh off by default | ENR-M3 | | ENR-D12 | First adapters | PDL, Logo.dev, Gravatar, one e-mail verifier | ENR-M1, M2 | ### How this document was produced The work began with nine parallel read-only investigations on 2026-10-22: five over the Rhythm 2.0 and 1.0 repositories and four over the market and the law, plus the architect's own provider checks (w5). The repository revisions were services-platform `43478ffda`, services-crm `e818ee48`, macroservice-libraries `49728e6`, and react-console `45a47ab6b`. The architect also read provider pages the researchers could not reach. The chapters were written against those reports, with labels on individual claims. The prose was humanized and checked fact by fact, and this single page was built from the result. Chapter 16 contains the method, its limits and every source. ### Standing decisions this proposal treats as constraints These are existing constraints, not new decisions made by this proposal: - Custom fields are opt-in per entity (R2-ARCH-001). - Milestones follow the milestone standard R2-DEV-001. - Operations must be submission-compatible (R2-ARCH-002). - Record writes follow the record write contracts; record merges follow the record merge participant protocol. - Queued work, persistence, indexing and RLS (row-level security) follow the standing standards. - Configuration ids are strings; record ids are UUIDv7. - Data uses snake_case; code uses camelCase. - CRM owns the consent ledger (Marketing D-M2). - Dataset queries use `.query` and `.read` (CRM-ACC-0064). - Every console screen starts in v0. The portal comes last, using Andrew's components. - No Asana; tickets go in Jira RX. - Humanize is used for everything presented. ## 2. 01. Executive summary and recommendation ## 01. Executive summary and recommendation Status: proposed 2026-10-22; six gating decisions ruled by Andrew the same day (ENR-D1 overturned: enrichment lives inside CRM; ENR-D6 overturned: tier 3 offered from ENR-M1 behind attestation; the rest as recommended; see [15](#16-15-open-architectural-decisions) "Rulings" and [05 §8](#6-05-service-ownership-and-integration-platform-alignment)). Labels as in the [README](#1-rhythm-2-0-data-enrichment-architecture). This package is research and architecture only. Nothing in the enrichment proposal is decided or built. ### The recommendation Make enrichment a **Platform capability**, with one module in services-platform responsible for lookup policy, requests, provider orchestration, claims, suggestions, provenance, suppressions and budget. It should call every provider through the Integration Platform's gateway, using ordinary adapters rather than a separate integration system. **CRM keeps every write.** Enrichment proposes changes; CRM applies accepted suggestions to contacts and organizations through its own update path, preserving CRM history and events. A person's employer change is always a proposal that a person approves. Keep providers interchangeable by defining a small, Rhythm-owned vocabulary of claims rather than exposing each provider's schema. A claim records what a provider said; a suggestion records what Rhythm proposes to change; the CRM record remains the authoritative record. Provenance records where a value came from. The proposed first milestone covers one provider, contacts first (organizations read-only until ENR-M2), manual and on-create triggers, a review panel on the Contact 360, and a monthly budget. ### What the research found - **The association market has not solved this.** No association management system (AMS) documents native, administrator-controlled enrichment of employer, title or organization data. Customers already pay for postal hygiene and duplicate handling: Blackbaud every 90 days, Bloomerang nightly, and Tessitura at about $1 per 1,000 records per service. Duplicate tooling is mature and converges on similar approaches, but provenance and approval are thin everywhere. No AMS addresses the member known by a personal e-mail ([02](#3-02-market-and-competitive-analysis)). - **Enterprise CRMs show what to copy and what to avoid.** Keep provider output in separate tables and apply it by policy, as Microsoft and Twenty do. Set per-field rules of fill-empty, suggest or never, and let manual edits always win, as HubSpot and Attio do. Charge for successful matches and show an estimate before a run, as Twenty, Clay and folk do. Never pool customers' data: HubSpot reversed its cross-customer sharing plan within four days. Provider volatility also argues against tight coupling: Clearbit's APIs closed, Data.com and DataFox retired, and Proxycurl shut down under a LinkedIn settlement ([02 §2](#3-02-market-and-competitive-analysis)). - **Providers were built for business-to-business sales, not membership.** Only People Data Labs (PDL) and FullContact document personal-e-mail matching, and no provider publishes a match rate for it. Certifications and licences are not available from any of them. PDL is API-accessible, scores matches 1 to 10, charges nothing for a miss and publishes an entry price of $98 a month for 350 credits, about $0.28 per match. Its own licence terms could not be read ([03](#4-03-enrichment-provider-comparison)). - **Law and licences impose three non-negotiable constraints.** A lookup discloses the member to an independent controller, so the association must have a legal basis and give notice. A response obtained for one association may never be reused for another. Each provider's terms also differ on member display, export, retention and AI use; FullContact bans any machine-learning use outright ([09](#10-09-security-privacy-and-licensing)). - **Rhythm 2.0 already has most of the supporting machinery.** The Integration Platform's catalogue, connections, credentials, gateway, executions, rate limits and software development kit (SDK) are built, and INTG-M1 is accepted. A `crm.contact.enrich@1` contract exists, but it is address-only and unimplemented. The AI usage ledger provides the metering pattern. CRM's duplicate normalisation, data jobs with receipts and rollback, revisions, Attention, business events, Durable Jobs and workflow actions are also present. The gaps are per-field provenance, a review queue, published `contact.created` events, a service write path into CRM, enforcement of the `personal_data` flag, and metering outside the AI module ([05](#6-05-service-ownership-and-integration-platform-alignment), evidence r1 to r4). - **Rhythm 1.0 shows what not to repeat.** It used one global key, had no budget, and kept a cache with no association boundary or expiry. Raw vendor responses were mapped in the browser, photos were stored as vendor pointers, and member records were populated silently. A public portal operation also let any signed-in user look up any e-mail ([r5](#22-r5-rhythm-1-0-fullcontact-and-google-civic-enrichment-and-the-customer-360-precedent)). ### The design in ten lines 1. **One capability, many sources.** Providers map their output onto a Rhythm-owned claim vocabulary, so changing providers does not change the rest of the design ([07](#8-07-provider-contracts-and-orchestration)). 2. **Keep three layers separate.** The claim is what a source said; the suggestion is what Rhythm proposes; the record is what is true in CRM. These layers never collapse into one ([08](#9-08-data-model-provenance-and-policies)). 3. **Protect trusted values.** Never overwrite them silently. Auto-fill only empty fields where policy allows, and never auto-fill a person's employer. 4. **The owner writes.** Enrichment proposes changes through CRM's data jobs now and through CRM's service function after CRM-M15. 5. **Do not make members wait for providers.** Every trigger runs after the CRM change has committed, so no member-facing operation depends on a provider response. 6. **Establish identity before enriching.** Combine the provider's match likelihood with Rhythm's own corroboration. Share CRM's normalisation rules, but never merge records through enrichment. 7. **Keep every call within one association.** Scope keys, caches, suppressions and budgets to that association. Never pool data across customers. 8. **Bound spend before calling.** Use an allowance ledger copied from the AI module, show estimates before bulk runs, and settle usage in the provider's own billing units. 9. **Treat licence and privacy restrictions as data.** Put a licence class on every claim and use one policy function to control display, export and AI use. Keep a suppression ledger that survives erasure, with tiers that require attestations and enforce region rules. 10. **Build a small foundation with clear extension points.** Use an ordered plan of at most three providers, not a Clay-style graph. Employment watch, AI research and member controls should be additions, not redesigns. ### Economics At the candidate provider's published price, and with an assumed 40 percent match rate, initial enrichment costs about $1,300 for 10,000 contacts, $13,000 for 100,000 and $120,000 to $132,000 for a million. A steady year after that is roughly a tenth of the initial cost. No vendor publishes a match rate, so these are planning assumptions, not measured outcomes. A miss costs nothing. The allowance therefore bounds spending exposure rather than leaving it dependent on the match rate. The baseline, logos, avatars, registry identifiers and e-mail verification, costs pennies per contact and can be included for every workspace. Bring-your-own-key is the default. A Rhythm-managed credit tier waits for signed original equipment manufacturer (OEM) terms that permit that commercial arrangement ([12](#13-12-commercial-model-and-economics)). ### The first milestone **ENR-M1 is estimated at about 18 to 24 agent-days.** It establishes the foundation and gives staff a visible review-and-apply workflow: - The contracts category and claim vocabulary, plus the Dummy and PDL adapters. - The module's tables and allowance ledger. - Manual and on-create triggers for contacts, with identity acceptance before enrichment is used. - Suggestions for empty and differing scalar fields, meaning fields that hold a single value, plus an assisted employer-change path. - Application of accepted changes through a CRM data job, acting as the reviewer. - The Contact 360 panel, field badges, review queue and Enrichment configuration screens, with v0 first. - The `personal_data` enforcement fix, plus merge and lifecycle participants so enrichment takes part in those record operations. It needs INTG-M2 verified and a PDL developer account. It needs nothing from CRM-M15. Five more milestones follow in order ([13](#14-13-phased-implementation-milestones)). ### The decisions Andrew needs to make The package's twelve decisions, ENR-D1 to ENR-D12, are written out in full in [15](#16-15-open-architectural-decisions) for Andrew to approve or overturn. This document summarizes the proposals; nothing is decided or built. Six decisions gate the first milestone: - **Ownership:** a Platform module. - **Contract vocabulary:** a new `enrichment` category. - **Default apply policy:** suggest for people, auto-fill organization brand, never overwrite. - **Commercial model and first provider:** bring your own key, PDL first, Rhythm-managed later. - **Personal-identity data in year one:** declared, not offered. - **Licensing:** an application key with a baseline for everyone. The remaining decisions concern photos, member transparency, AI research, employment-change review, retention and the first adapters. ## 3. 02. Market and competitive analysis ## 02. Market and competitive analysis Rhythm should use external providers for enrichment, keep their results separate from member records, and apply changes through a shared policy and review process. The opportunity is not to build another data source. It is to give association staff control over what changes, why it changes, and what happens when the evidence is weak. Status: proposed 2026-10-22. Nothing in this document is decided or built. Labels follow the [README](#1-rhythm-2-0-data-enrichment-architecture). Every vendor statement comes from the two competitive evidence reports: [w1 (association management systems)](#23-w1-evidence-how-association-management-systems-and-the-nearest-nonprofit-crms-handle-data-enrichment-and-data-quality) and [w2 (enterprise CRM and ERP)](#24-w2-evidence-how-enterprise-crm-erp-platforms-do-data-enrichment). Those reports contain URLs, access dates and per-claim labels: **Verified** from primary documentation, **Reported** from a third party, **Marketing** without operational documentation, and **Unavailable**. Research date 2026-10-22. Both researchers exhausted their web-search budgets before completing the review. §5 explains the remaining gaps. ### 1. What the association management market actually does **No association management system (AMS) reviewed here documents a native, administrator-controlled feature for enriching employer, title or organization data.** Momentive Software makes one claim for MomentiveIQ: “cleansing and enrichment built into Agentic Workers,” with review-and-approve and confidence thresholds. That remains a marketing claim without operational documentation (**Marketing**: [w1 §2.16](#23-w1-evidence-how-association-management-systems-and-the-nearest-nonprofit-crms-handle-data-enrichment-and-data-quality)). What the market does offer, and customers already pay for, is postal-address cleanup and duplicate management. Fundraising and arts systems also offer address, deceased and demographic enrichment. These are useful precedents, but they do not solve the professional-member employment problem. | Platform (owner, late 2026) | Native enrichment | How it reaches the product | Duplicates | Employment change | Personal e-mail | Provenance and approval | | --------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------- | | iMIS (ASI) | None | partner add-ons: CSI iValidate (address, e-mail, phone, deceased, NCOA; "coming soon", shipping unconfirmed) | licensed Duplicate Merge: formula-based pairs, scheduled search, manual merge, no undo | none | not addressed | formulas shown per pair; merge logs; no scores | | GrowthZone / ChamberMaster / MemberSuite | None found | none | account-number uniqueness; manual merge | none | not addressed | not found | | netFORUM (Momentive, ex-Community Brands) | None; address-correction plug-in | Melissa, CDYNE, AccuMail and others, customer-licensed | separate duplicate report; hide-instead-of-delete merge | none | not addressed | log of hidden merges | | Fonteva and Nimble AMS (Salesforce-native) | None documented | whatever the Salesforce org installs (ZoomInfo, D&B) | Salesforce matching and duplicate rules; Nimble "alert and report" | ZoomInfo's "Person Has Moved" flag exists on the platform; neither vendor documents using it | not addressed | Salesforce duplicate reports | | Personify360, MemberClicks, YourMembership, Wild Apricot (Momentive since January 2026) | None | none | batch and online merge; manual "find a duplicate"; Wild Apricot has no finder (wishlist open since 2012) | none | Wild Apricot: e-mail is the unique id, one per contact | archive-and-restore of a merged contact (Wild Apricot) | | Aptify | None; address verification connector | ranked list of verification providers (Melissa first) | sample duplicate-check objects, off by default | none | not addressed | not found | | Impexium, Novi, Glue Up, Raklet, Higher Logic | not assessed (budget) | | manual merges | | | | | Tessitura (arts, constituent) | NCOA utility plus bureaus | Loqate, Experian; about $1 per 1,000 records per service | potential-duplicate queue | none | | Do Not Mail service | | Blackbaud Raiser's Edge NXT | **Yes**: Address Finder and Deceased Finder in product, every 90 days; e-mail, phone, age and gender finders as a bureau | USPS NCOA, SSA death data, LiveRamp (Identity Thread) | scored fuzzy pairs with a confidence and a minimum-likelihood filter; strict exact for automatic form matching | none | | per-pair confidence; previous and next address visible; a data-processing acknowledgement at enablement | | Bloomerang | **Yes**: NCOA, deceased and demographics via TrueGivers, nightly | TrueGivers; DonorSearch and Dataro as partners | real-time scan on change; no automatic merge | none | shared e-mail is a duplicate signal | an "NCOA" note and "Last Modified By: TrueGivers" on each write; no approval step | Three patterns matter for Rhythm: 1. **The successful pattern is an external provider connected through a replaceable integration.** Examples include Aptify's ranked verification providers, netFORUM's pluggable address correction, Tessitura's Loqate and Experian integrations, Bloomerang's TrueGivers integration, and Blackbaud's bureaus. These products rely on providers rather than owning the underlying data asset. Salesforce's retirement of Data.com is a caution against building around an owned dataset. 2. **Duplicate handling is mature and converges on a common workflow.** Staff see candidate pairs side by side, record a lasting “not a duplicate” decision, and retain a merge log. Blackbaud's confidence scores and Bloomerang's real-time scanning are differentiators. Almost no merge can be undone. 3. **Evidence and approval controls are thin.** Provenance means the record of where a value came from and how it was applied. Bloomerang's per-write note and Blackbaud's per-pair confidence are the strongest examples here, but neither has an approval step for enrichment writes. The reviewed documentation does not describe per-person enrichment opt-out, region-aware rules, or a record of which provider holds which data. The important gap is **the professional member known by a personal e-mail address**. AMS documentation is silent on this case, while business-to-business (B2B) providers mostly identify people through a work e-mail address ([03](#4-03-enrichment-provider-comparison)). ### 2. What the enterprise CRM and ERP platforms do Customer relationship management (CRM) and enterprise resource planning (ERP) platforms offer more developed patterns for controlling enrichment. The useful comparison is not just which fields they can fill. It is how they protect staff-entered values, retain evidence, handle consent and charge for results. In the table, “survivorship” means the rule used to choose a value when sources disagree. “Provenance” means the retained evidence about that value's source and history. | Platform | Native enrichment | Providers behind it | When and how | Conflicts | Provenance | Consent and privacy | Pricing | | ------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | Salesforce (Sales Cloud, Data 360, Agentforce) | no dataset of its own; Data 360 unifies and reconciles (match rules; survivorship: most recent, most frequent, source priority, with field overrides) | AppExchange: ZoomInfo, D&B, Demandbase, LinkedIn; Clearbit no longer standalone | integration rules periodic; Data 360 scheduled or manual; duplicate rules on save | blank-only for integration rules; reconciliation rules for unified profiles; source values left untouched behind link objects | link objects; field history on a limited number of fields; no provider-per-field stamp | Individual, Data Use Purpose and Legal Basis objects; enforcement against enrichment undocumented | Data 360 credits (about $500 per 100k flex credits, Reported); Agentforce about $0.10 per action | | HubSpot (Data Hub, ex-Breeze Intelligence, ex-Clearbit) | contact and company enrichment, form shortening, intent, duplicate management, AI "smart properties" | HubSpot's purchased and assembled dataset plus unnamed third parties and the public web | on create (off by default), monthly continuous, on engagement, on import, in workflows, on demand, bulk of 100 | **per property: fill empty, overwrite, never**; automatic fills blank only; continuous enrichment stops on a property a person edited; some writers bypass the rules | property history with a source column; no enrichment-specific per-value stamp | Super Admin configuration; a contact-level "enrichment opt out" marker; an individual opt-out form; the July 2026 cross-customer data-sharing plan was reversed within four days and is now promised as opt-in only | standard enrichment credit-free on the tier; smart properties 10 credits per prompt per record; overage about $10 per 1,000 (Reported) | | Oracle NetSuite | none found; duplicate detection and merge only | third-party connectors; DataFox sunset 2023 | duplicate alerts on save | merge: blanks copied, populated kept, source deleted, irreversible | a system note | merge is a permission | none native | | Microsoft Dynamics 365 (Customer Insights - Data) | first-party enrichments (brand and interest affinities, enhanced addresses) and partners | Experian, LiveRamp, HERE, Azure Maps, SFTP; Leadspace B2B closed to new customers | on demand or scheduled, segment-scoped | **enrichment kept in separate tables joined by the unified id**; match normalisation with exception and override lists; survivorship by field group; suggested updates with accept, reject, revert | per-field coverage, run status, previous ids; citations for e-mail-derived suggestions | admin-only connections with a consent acknowledgement; the customer owns the lawful basis | Microsoft enrichments included; partners separate; capacity packs | | Zoho CRM | Zia Data Enrichment per module (organization and personal) | unnamed web crawl and databases | on create or edit by click; automatic trigger fields Reported but conflicting | the user ticks fields; overwrite semantics undocumented | an Enrichment History that distinguishes Zia from manual, not exportable, erased when a field is hidden | permissions only | edition limits of 500 to 2,000 per user up to 50,000 a month | | Attio | automatic enrichment plus AI attributes | unnamed providers; Apollo and others via apps | on record add, seconds; refresh when new data | manual values never overwritten | a coloured cell; AI confidence, reasoning and sources | enriched data not exportable | plan-tiered; AI 10 credits per web-agent attribute; no charge when empty | | Clay | the product _is_ enrichment orchestration | 100+ providers; waterfall per column | table-driven, per row | n/a (a staging workspace) | per-cell results and a validation step | not researched | actions plus data credits; hit-only billing; bring your own key | | Pipedrive | Data Enrichment | Surfe | manual and bulk of 100 | empty fields only | not described | the three-party GDPR split documented: Surfe an independent controller with an opt-out | 1, 2 or 5 credits; reimbursed on fill or fail in bulk | | folk | people enrichment waterfall | Apollo, PDL, Clearbit, Datagma, Prospeo, Dropcontact, Wiza | on demand; groups of 500 | additive; never replaces | a sparkle icon | DPA (Marketing) | 1 credit e-mail, 4 phone; refund on fail | | Twenty (open source) | a PDL app plus server-side company enrichment | People Data Labs | command menu, workflow step, AI chat; throttled | tri-state overwrite per field; empty checks | `pdl*` sidecar fields, the raw payload, a likelihood, a timestamp and a status | not documented | $0.15 per person match, $0.075 per company; misses free | ### 3. What Rhythm should adopt The proposal combines these patterns to protect member records without making staff review every low-risk update. Provider results remain separate until a field policy permits them to be applied. Staff can inspect the evidence, reject a suggestion or reverse an accepted change. These are proposed design choices, not implemented capabilities. A provider “waterfall” tries providers in a defined order. “Bring your own key” lets a customer use its own provider credentials. A domain-keyed organization lookup identifies the organization through its internet domain rather than trying to identify an individual member first. | Adopt | From | Where it lands in this proposal | | ----------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------- | | Keep provider output in its own tables, apply by policy | Microsoft, Twenty, Salesforce link objects | claims and suggestions as separate layers ([P2](#5-04-architectural-principles), [08](#9-08-data-model-provenance-and-policies)) | | Per-field policy: fill empty, suggest, never; manual edits always win | HubSpot, Twenty, Pipedrive, Attio | field rules and the `locked` provenance flag ([08 §2.1, §2.6](#9-08-data-model-provenance-and-policies)) | | Suggest or auto-apply per field class, with accept, reject, revert and an evidence link | Microsoft Customer Insights, Attio | the suggestion states and the 360 panel ([06](#7-06-core-workflows-and-use-cases), [11](#12-11-console-and-portal-experience)) | | Stored, thresholded confidence; a stricter rule for weak identifiers | Twenty (`weakIdentifierMinLikelihood`), Clay | the identity acceptance rule ([06 §7](#7-06-core-workflows-and-use-cases)) | | Bill on the hit, show the estimate first, cap per tenant | Twenty, Clay, folk, Pipedrive, ZoomInfo's dry run | the allowance and estimates ([12 §4](#13-12-commercial-model-and-economics)) | | Provider adapter boundary; ordered waterfall later; bring your own key | Clay, Aptify's ranked providers, every AMS bureau pattern | contracts and the plan ([07](#8-07-provider-contracts-and-orchestration)) | | Store licence terms with the data; export knows what is licensed | Attio's non-exportable enriched data | licence classes and the policy function ([09 §4](#10-09-security-privacy-and-licensing)) | | Enrichment as a separate processing activity with a lawful basis, a controller split, an opt-out and a suppression list | Pipedrive, HubSpot, Microsoft | tiers, attestations, suppression ([09](#10-09-security-privacy-and-licensing)) | | Organization (domain-keyed) enrichment first; person second | HubSpot, Attio, Twenty | tier 1 on by default; tier 2 and 3 by choice | | A visible data-health status with honest "paused" states | Blackbaud's scorecard, Bloomerang's bell | the 360 status line and the Budget tab ([11](#12-11-console-and-portal-experience)) | | Employment change as a proposed task with provenance, never an automatic re-association | ZoomInfo's own caveat ("does not associate contacts with their new companies") | [06 §6](#7-06-core-workflows-and-use-cases) | ### 4. What Rhythm should avoid The same review exposes failure modes worth excluding from the design: - **Three engines for one question.** Salesforce separates duplicate rules, Data 360 identity resolution and integration rules. Rhythm should use one identity vocabulary for enrichment and duplicate detection, so both make consistent judgments about whether records refer to the same person or organization. - **Write paths that bypass the policy.** HubSpot's conversational enrichment can write regardless of mapping rules. Every Rhythm path, including workflow actions and future agents, should pass through the same suggestion gate. - **Fragile provenance.** Zoho's history disappears when a field is hidden, and some field-history systems cover only a handful of fields. Rhythm should retain provenance in a dedicated table keyed by field, so staff do not lose the evidence when the interface changes. - **Pooling customers' data.** HubSpot reversed its cross-customer data-sharing plan. The proposal prohibits a cache or dataset shared across customers, ever. - **Charging on the attempt.** HubSpot smart properties are billed “filled or not”; Salesforce charges per-row unify credits. Rhythm should settle charges on `cost_units`, which is zero for a free miss. - **Irreversible merges and silent re-association.** Enrichment should never merge records. Employment changes should be proposed for review, not automatically move a member to another organization. - **Form-time or page-time provider calls.** Rhythm 1.0 used field-blur enrichment, triggering work when someone left a field. The proposal moves all enrichment work until after the record has been committed, rather than making a form or page wait for a provider. ### 5. Limits of this research Several vendor help centres blocked fetching: GrowthZone, Aptify, Blackbaud and Neon community pages; Salesforce Help returned a loading shell, and developer.salesforce.com returned a 403. Claims affected by those blocks rely on search summaries and are labelled Reported. The web-search budget ran out before the researchers could check MemberSuite, YourMembership, Impexium, Glue Up, Raklet, Higher Logic, Neon One, HubSpot's privacy FAQ and opt-out page, Salesforce consent enforcement, Dataverse-native enrichment, or the NetSuite SuiteApp marketplace. Zoho's automatic trigger-field enrichment is documented only by a third party and conflicts with Zoho's own help. Lightning Data's 2026 status is unconfirmed. These gaps limit claims about what the wider market does not offer. They do not change the proposal's conclusion: the differentiation is real, and the design patterns worth copying are consistent across the vendors reviewed. The recommendation remains to combine external providers with stronger field-level control, evidence and review, not to treat missing documentation as proof that a capability cannot exist. ## 4. 03. Enrichment provider comparison ## 03. Enrichment provider comparison **The proposed shortlist is PDL for person enrichment, FullContact as the alternative to evaluate, and a low-cost baseline of Gravatar, Logo.dev, GLEIF, Wikidata and one e-mail verifier.** Start with organization data and contact verification. Do not choose a person provider until a consented association-member sample establishes match quality and the contract permits Rhythm's intended use. Status: proposed 2026-10-22. Nothing in this document is decided or built. ENR-D4 is the provider-selection decision; this document supplies the comparison and the tests needed before that decision. Labels follow the [README](#1-rhythm-2-0-data-enrichment-architecture). Sources are [w3](#25-w3-evidence-enrichment-provider-landscape-for-an-association-management-system) (provider research, per-claim labels and the full URL list), [w4 §5](#26-w4-evidence-report-privacy-law-and-data-licensing-constraints-on-rhythm-2-0-data-enrichment) (licence terms) and [w5](#27-w5-supplementary-provider-checks-read-directly-2026-10-22-11-00-to-11-30-et) (the architect's direct checks). Research date: 2026-10-22. Prices are vendor-published figures where marked **Published**, arithmetic on those figures where **Derived**, and third-party figures where **Reported**. Nothing here is a quote. ### 1. The distinction that matters: B2B sales enrichment versus association membership Most modern enrichment providers serve business-to-business (B2B) sales: they help a salesperson identify who works at a target company. They usually identify a person by **work e-mail** or **LinkedIn profile**, with a data model of "one person, one current employer". Association members do not fit that model. They may join with a personal e-mail and keep it for decades while changing employers. They can belong to several organizations at once, including an employer, a practice, a university and a chapter. They hold certifications and licences that most providers do not carry, and they are often outside the United States. The evidence (**Market**: [w3 §11.A](#25-w3-evidence-enrichment-provider-landscape-for-an-association-management-system)): - **No provider publishes a match rate for personal-e-mail-only input.** PDL's own guidance says an e-mail-only match rarely scores above 6 of 10 and name plus location rarely above 4. - Only **PDL** and **FullContact** document accepting personal identifiers. Cognism says it holds no personal e-mails. Apollo withholds them in GDPR regions and scopes its terms to business contact information. Hunter cannot infer an employer from a Gmail address and gives webmail a fixed score of 50. - **Certifications and licences** appear only in PDL's thin `certifications` field, with no coverage figure. Credentialing bodies are the authoritative sources and require a different kind of integration. - **Consumer-data appenders** (AtData, now Experian; Versium; Melissa; FullContact Resolve) answer "is this a real, current person and where do they live" and "did the address change". They do not document employer data. They support address hygiene and duplicate detection, not affiliations. The proposed design therefore sends all known identifiers, including e-mail, name, employer, city and school, rather than e-mail alone. It accepts a result only when the provider's likelihood score and Rhythm's own corroboration support the match, and pays only on a match. Organization enrichment is the reliable first tier: it uses a domain as its lookup key and has low personal-data exposure. ### 2. Person and identity providers PDL has the strongest documented combination of capability and price, but its contract remains unresolved. FullContact has the closest documented contract structure for supplying data through a customer-facing application, but pricing and association-member coverage need testing. The other providers mostly serve business-contact lookup, sales workflows or LinkedIn-derived data. | Provider (owner) | Operations | Inputs | Personal e-mail | Returns | Confidence | Pricing (per record) | Terms on re-supply to Rhythm's customers | Notes | | --------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | | **People Data Labs** (independent; California-registered data broker) | person enrich, company enrich, Identify (up to 20 candidates), Search, Person Changelog (monthly or quarterly diffs), bulk (100 per call), enterprise webhooks | any e-mail, name plus company or location, LinkedIn URL, phone; one identifier or name plus one supporting field | **yes**, documented; `personal_emails` populated on 64 % of its e-mail slice | experience, education, certifications, skills, interests, locations, profiles, phones (premium) | likelihood 1 to 10 | $98 per month for 350 credits (**Published**) → $0.28 per match (**Derived**); misses free; enterprise custom | **own terms unreadable** (script-rendered page); a reseller's reproduction bars resale and sublicensing (**Reported**); embedding claims are marketing | the strongest documented fit on capability and price; the contract question is open | | **FullContact** (Ziff Davis) | Enrich (person, company), Resolve (identity), Verify, batch by sales | e-mail (clear or hashed), phone, social profiles, name **only with a postal address**, MAIDs | **yes**, consumer identity graph, 240M US profiles, rebuilt biweekly | name, age range, gender, location, title, organization, socials, bio, avatar, website, plus bundles (employment history, affinities) | none documented | not published | the Services Agreement defines how a **Customer Application** may pass data to **Third Party End Users** under flow-down terms (the closest ready-made OEM shape); delete within 30 days of termination; **no LLM or machine-learning use of its data at all** | US-centred (Reported); the 1.0 provider; sales-led | | **Apollo.io** (independent) | people and organization enrich, bulk (10), e-mail status, search | e-mail, name plus domain or organization, LinkedIn URL | withheld in GDPR regions | employment history, business e-mail, phone (8 extra credits) | `match_confidence` high to none | credits; dollars not fetched | **no**: no sublicensing, no API access for third parties, internal use only | unsuitable for embedding | | **ZoomInfo** (public; "GTM" platform) | enrich, search, intent; 25 per call; OEM and partner programme exists | undocumented in pages read | no evidence | B2B profiles | accuracy score and "no match reason" in its Salesforce product | about $0.38 per record at the $1,000 tier (**Reported**) | contract-defined; bars putting its API credentials into third-party applications | enterprise sales; B2B only | | **Clearbit → HubSpot Breeze Intelligence** | inside HubSpot only | | | | | about $45 per month for 100 credits plus a HubSpot subscription (**Reported**) | no API for non-HubSpot customers; standalone subscriptions closed; free Logo API ended 1 December 2025 | not a candidate | | **Clay** (independent) | orchestration over 100+ providers, Claygent AI research, signals, webhooks | whatever the chosen provider takes | per provider | per provider | per provider; a validation step | data credits from $0.05; actions metered separately; hit-only; bring your own key | a staff tool; the underlying provider's rights apply | the pattern, not a supplier | | **Cognism** | search, enrich | work identifiers | **no personal e-mails** by its own statement | B2B contact data, do-not-call screening | | not fetched | not found | Europe-focused B2B | | **Lusha, RocketReach, Hunter, Dropcontact, Datagma, Prospeo, Kaspr** | reveal and finder tools | work e-mail, domain, LinkedIn | no (Hunter: webmail fixed at 50) | business contact data | | Lusha $0.094 to $0.125 per e-mail credit; Hunter about $0.012 to $0.025 per credit (**Derived**) | not researched or not permitted | B2B reveal tools; Kaspr fined by the CNIL (EUR 200,000 to 240,000) in December 2024 | | **Crustdata, Coresignal, Proxycurl (closed July 2025)** | LinkedIn-derived person and company data; Crustdata Person Watcher (job changes) | LinkedIn URL | no | profile data | | Crustdata 5 credits per changed person at 30-day freshness | sourcing risk: Proxycurl shut down under a LinkedIn settlement | job-change feeds exist here, at the cost of supply-side legal exposure | | **UserGems, Champify** | job-change tracking for sales teams | CRM contacts | | "champion moved" alerts | | $15,000 to $100,000 a year (**Reported**) | sales tooling | not built for membership | | **LinkedIn official APIs** | consenting member's own profile only | OAuth | | the member's own data for the member's own experience | | | no storage beyond necessity; no re-supply | not an enrichment source | ### 3. Organization, brand, photo and verification providers These services cover narrower needs: identifying organizations, displaying logos or member-published photos, and checking contact details. The baseline remains subject to licence and retention checks. Richer company data, address updates and AI research are later capabilities. | Provider | What it gives | Inputs | Pricing | Terms | Role in this design | | -------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ | | **Logo.dev** | logos and brand assets; search; webhooks on paid plans | domain | free up to 500,000 CDN requests a month (attribution on the free plan for commercial use); $33 and $150 plans; $0.02 per API logo (**Published**) | "store anything the Logo.dev API returns, logo files included", on every plan; full terms unread | the baseline logo source (`open`, pending full terms) | | **Brandfetch** | brand data, logos, search | domain | $99 to $129 for 2,500 to 10,000 fetches; $0.10 overage (**Derived**) | internal use for the licensee's own business; 30-day cache then re-call; third-party use needs a Custom agreement | only under a Custom agreement | | **Gravatar** (Automattic) | the member's own published avatar and profile by SHA-256 of the e-mail; `d=404` probe | e-mail | free; profile lookups about 1,000 an hour | responsible-use guidelines: refresh stored data at reasonable intervals; disclose in a privacy policy; no excessive calls | the baseline photo source (`self_published`) | | **GLEIF (LEI), Wikidata, Companies House, SEC EDGAR, OpenCorporates** | registry identifiers and legal names | name, identifier | free (CC0) except OpenCorporates' share-alike free tier | open | baseline identifiers (narrow coverage for small associations' employers) | | **Crunchbase, Dun & Bradstreet Direct+, Diffbot** | firmographics, hierarchies, knowledge graph | domain, name, DUNS | D&B quote-only; Diffbot about $0.025 to $0.03 per entity (**Derived**) | D&B no redistribution; Crunchbase 10-day deletion | later, for associations that need verified hierarchies | | **E-mail verification: Bouncer, Emailable, ZeroBounce, Mailgun, Kickbox, NeverBounce** | deliverable, undeliverable, risky, catch-all, disposable, role | e-mail | $0.002 to $0.0195 per check (**Published**) | retention terms not researched | the baseline `email.verify` source | | **Consumer appenders: AtData (Experian), Versium, Melissa** | address, phone, e-mail append; NCOA and move update; demographics | name plus address, e-mail, phone | Melissa about $0.012 to $0.014 per record at one million a year; Versium $0.04 per match (**Reported**) | opt-out obligations | a future address-hygiene capability on the same framework, not wave one | | **AI research: Exa, Parallel, Perplexity, Anthropic or OpenAI web search** | cited public-web findings | name, employer, city | $0.005 to $0.10 per request plus tokens (**Published**) | terms on personal data unread | ENR-M6; Anthropic's search tool is not on Bedrock | ### 4. Where the capabilities come from, by kind No single provider covers the association use case. Identity resolution helps staff determine whether a result refers to the same person; it must not merge records. Person results need corroboration, and AI research requires opt-in and human approval. | Kind of enrichment | Realistic sources for associations | Comment | | -------------------------------------------------- | ------------------------------------------------------------------------ | ----------------------------------------------------------------------- | | Person enrichment (title, employer, profiles, bio) | PDL; FullContact; AI research as a cited fallback | personal-e-mail match rate unmeasured; corroboration essential | | Organization enrichment (firmographics) | PDL company, Apollo organization (internal use only), Diffbot, D&B | domain-keyed, low privacy exposure, the reliable first tier | | Identity resolution | PDL Identify, FullContact Resolve, consumer appenders | used only to decide "same person", never to merge | | Contact verification | e-mail verifiers; Melissa for addresses and phones | cheap; baseline | | Photos and social profiles | Gravatar (self-published); person providers (licence-dependent) | provider photos as suggestions, copied only on acceptance where allowed | | Employment and job-change detection | PDL changelog (monthly, enterprise); Crustdata watchers (LinkedIn-keyed) | later; per-provider decision on terms and freshness | | Company firmographics | as organization enrichment | | | Continuous data monitoring | PDL webhooks and changelog; Exa and Parallel monitors | later | | AI-driven research | Exa, Parallel, search-tool-equipped LLMs | opt-in, human-approved, cited | ### 5. Legal and commercial use of returned data, in one table A successful lookup does not give Rhythm permission to use the result everywhere. Staff access, member-facing display, storage, AI use and reuse across customers are separate rights. The proposed design checks those rights for each returned claim, meaning each individual piece of information. The default is bring your own key (BYOK): an association supplies its own provider credentials. A shared Rhythm key requires a provider-specific original equipment manufacturer (OEM) agreement that permits Rhythm to embed and supply the service. Neither arrangement overrides restrictions on the returned data. | Question | Answer from the terms read | Consequence | | --------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ | | May an association's staff see the results in Rhythm? | yes for PDL (Reported), Apollo, ZoomInfo (internal use), FullContact (as end users of a named application), Logo.dev, Gravatar, registries | the `internal_staff` floor | | May results be shown to members in a portal or directory? | not under self-serve terms (Apollo, ZoomInfo, PDL Reported, Brandfetch); FullContact only through a Customer Application with flow-downs; Gravatar and open sources yes | the `customer_app` and `open` classes; default no | | May results be kept indefinitely? | Logo.dev yes; Gravatar with periodic refresh; Brandfetch 30 days; FullContact until termination plus 30 days; Apollo tied to the subscription; PDL unread | `expires_at` per claim | | May a photo be displayed? | Gravatar yes (member-published); LinkedIn no; providers' photos are Profile Data with storage limits | copy only on acceptance where allowed; Gravatar re-probed | | May results feed AI features? | FullContact forbids any LLM or ML use; Apollo and Brandfetch forbid training; others silent | the `no_ai` modifier; AI prompts consult the policy function | | May Rhythm hold one key for many customers? | Apollo and ZoomInfo bar on-behalf and third-party-app use; FullContact and PDL need a contract; Logo.dev silent | BYOK default; OEM addendum per provider for a Rhythm key | | May a response be reused across customers? | no licence read permits it; California's regulations give it as the forbidden example | never | ### 6. The shortlist and what must be tested before any provider is chosen The proposed shortlist separates baseline services from capabilities that need more evidence or a negotiated contract: - **Baseline, free or near-free where terms permit:** Gravatar probe, Logo.dev, GLEIF and Wikidata, and one e-mail verifier. - **Premium person enrichment:** PDL first. Its service is API-accessible, it documents personal-e-mail matching with a score, and it publishes an entry price. Obtain a FullContact quote as the alternative, specifically addressing US member coverage and OEM rights. - **Organization enrichment:** Logo.dev plus PDL company. - **Employment change:** No provider is priced for associations. PDL's monthly changelog is the only structural fit. - **AI research:** A search API plus the LLM module, later. LLM means large language model. Before the provider for ENR-M1 is fixed through ENR-D4: 1. Run the same 1,000-record sample of consented association members through PDL and FullContact. Supply personal e-mail, name, city and employer. Compare match rate, likelihood distribution and wrong-person rate. 2. Obtain PDL's actual terms in writing. 3. Read Logo.dev's full terms and one verifier's retention terms. **The match-rate number is the single most decisive missing datum in this proposal** (**Market**: [w3 §12](#25-w3-evidence-enrichment-provider-landscape-for-an-association-management-system)). ## 5. 04. Architectural principles ## 04. Architectural principles Status: proposed 2026-10-22. Labels as in the [README](#1-rhythm-2-0-data-enrichment-architecture). Everything here is **Recommended** unless labelled otherwise. The recommendation is to treat enrichment as a source of evidence, not as an authority over an association's records. Providers supply claims; Rhythm decides what to suggest; the service that owns the record applies approved changes. Member-facing work must not wait for a provider, and every lookup must respect the association's privacy rules and budget. These ten principles define the proposed architecture. They do not approve the design or authorise implementation. The package's twelve decisions, ENR-D1 to ENR-D12, are set out in document 15 for Andrew to approve or overturn. Each principle below states the rule and its supporting evidence. Where it restates a standing Rhythm 2.0 rule, that rule is named. ### P1. Enrichment is one capability with many sources Staff and members should see one enrichment capability: "tell me what the outside world knows about this record, and let me decide what to keep." Providers are interchangeable sources behind that capability. No application, screen, workflow or policy names a provider's field, endpoint or quirk. Instead, every provider works through a Rhythm-defined vocabulary of **claims** ([07 §2](#8-07-provider-contracts-and-orchestration)), rather than exposing its own schema. This keeps provider-specific details out of the rest of the product. The Integration Platform already requires that "the contract is the only thing they share" (**Verified**: `docs/proposals/integration-platform/10-capability-contracts.md:9`). Today's enrichment contract breaks that rule: `fields_requested` are "the provider's field names" (**Verified**: [r1 §A.2](#18-r1-the-integration-platform-as-built-evidence-for-a-data-enrichment-design)). Rhythm 1.0 went further and mapped the raw FullContact response in the browser (**Verified**: [r5 §1](#22-r5-rhythm-1-0-fullcontact-and-google-civic-enrichment-and-the-customer-360-precedent)). Both approaches make the provider impossible to swap without changing the consumers that depend on its schema. ### P2. A claim is not a fact A provider's answer is a **claim**, not an automatic change to the record. It contains a value for one field of one record, from one source, at one time, with one likelihood. Claims are stored as received and never edited. A policy or a person separately decides what Rhythm's record should say. That decision is recorded with the claim it rests on. Enrichment therefore keeps three distinct layers: - **Claim:** what the provider said. - **Suggestion:** what Rhythm proposes to do about it. - **Record:** what is true in Rhythm. This separation lets staff inspect the evidence without treating it as an accepted value. Microsoft keeps enrichment in separate tables joined to the profile. Twenty stores the raw payload and a match likelihood beside the applied fields. HubSpot and Attio stop enriching a field once a person has edited it (**Market**: [w2 §7](#24-w2-evidence-how-enterprise-crm-erp-platforms-do-data-enrichment)). Rhythm 1.0 collapsed these layers and silently populated member records (**Verified**: [r5 §1, §9](#22-r5-rhythm-1-0-fullcontact-and-google-civic-enrichment-and-the-customer-360-precedent)). ### P3. Trusted values are never overwritten silently Values entered by members in the portal, typed by staff, or supplied by a verified import outrank any provider claim. If a claim disagrees with one of those values, it becomes a suggestion for review, never an automatic change. An empty field may be filled automatically only when the association's policy allows it for that field group and tier. Every automatic fill must be visibly marked as provider-sourced and reversible from the record's history. This protects staff and members from losing their own work. It follows the brief's requirement, the "no silent loss of user edits" guardrail in the record write contracts (**Verified**: `docs/standards/record-write-contracts.md:24`), and the market pattern that "manual edits win" (**Market**: [w2 §7](#24-w2-evidence-how-enterprise-crm-erp-platforms-do-data-enrichment)). ### P4. The owner writes; enrichment proposes An accepted enrichment suggestion must receive the same validation, history and change handling as any other record update. Enrichment therefore never writes a contact, an organization, an e-mail, an address or an affiliation directly. It stores claims and suggestions in its own tables, then asks the service that owns the record to apply an accepted suggestion through that service's write path. That path provides the owner's validation, version checks, asset pipeline, history capture and change events. This restates the shared-table ownership rule: "Every macroservice reads any table; only its owner writes it" (**Required**: `AGENTS.md`, Platform shared tables section; **Verified**: [r4 §I-4](#21-r4-data-operations-data-job-engine-and-the-query-layer-evidence-for-rhythm-2-0-data-enrichment)). CRM's revision capture fails closed: it refuses a write that bypasses revision capture (**Verified**: [r2 §4](#19-evidence-r2-crm-rhythm-2-0-crm-as-the-target-of-data-enrichment)). Giving enrichment direct write access to CRM tables would require it to duplicate CRM's rules and would break the merge, history and event guarantees that CRM provides to every other application. ### P5. Nothing a member is waiting for depends on a provider Registration, checkout and other member-facing work must succeed even when an enrichment provider is slow or unavailable. Enrichment is therefore asynchronous: the record is created, registration is completed, or the profile is saved before an enrichment request is queued, after the commit. The queued request runs within the provider's time budget and retries under the gateway's rules. If the provider is down, the record remains unenriched until the next attempt. No route in Rhythm awaits a provider except an explicit, staff-initiated "Enrich now" action whose screen shows a progress state. This follows the brief's requirement that registration and checkout never depend on a provider. The gateway's synchronous budget is 25 seconds, and a `provider_unavailable` circuit stays open for a minute or more, temporarily blocking calls to an unavailable provider (**Verified**: [r1 §4.4, §4.6](#18-r1-the-integration-platform-as-built-evidence-for-a-data-enrichment-design)). Rhythm 1.0 instead triggered enrichment when a user left a form field and held the browser while it ran (**Verified**: [r5 §1](#22-r5-rhythm-1-0-fullcontact-and-google-civic-enrichment-and-the-customer-360-precedent)). ### P6. Identity before enrichment Before attaching a claim to a record, Rhythm must be satisfied that the provider matched the same person or organization. A plausible answer about the wrong person is not useful enrichment. Rhythm makes its own acceptance decision using both the provider's likelihood and corroboration against the existing record: agreement on name, employer or domain, and location. It uses the same normalisation as CRM's duplicate detection, so both processes compare values consistently. A claim that fails corroboration is retained in the claims table with its request marked not accepted, labelled low confidence, and shown only on request. Enrichment never merges records. If a provider's answer suggests that two Rhythm records represent the same person, enrichment passes that information to CRM's duplicate detection as a candidate. PDL's own guidance says that an e-mail-only match rarely scores above 6 of 10, and name plus location rarely scores above 4 (**Market**: [w3 §2.1](#25-w3-evidence-enrichment-provider-landscape-for-an-association-management-system)). The 1.0 engine had no corroboration and collapsed every failure into "not found" (**Verified**: [r5 §6](#22-r5-rhythm-1-0-fullcontact-and-google-civic-enrichment-and-the-customer-360-precedent)). CRM already has match keys, a free-mail list and scoring, and these are decided to be rules-based, not machine learning (**Verified**: [r2 §6](#19-evidence-r2-crm-rhythm-2-0-crm-as-the-target-of-data-enrichment)). ### P7. Every call is a tenant's call A lookup discloses a member to a provider on one association's instructions. Credentials, caches, suppressions, budgets and results must therefore stay scoped to that association, never pooled across customers. A response obtained for association A is never reused for association B, even for the same e-mail address. A Rhythm-held provider key may serve many associations only under a signed agreement that permits it. Even then, each call, cached response and usage measurement remains association-specific. An e-mail-keyed cache shared across associations mirrors the example forbidden by the California regulations and breaks the no-resale clauses of every licence reviewed (**Market**: [w4 §11](#26-w4-evidence-report-privacy-law-and-data-licensing-constraints-on-rhythm-2-0-data-enrichment)). Rhythm 1.0's cache key did not include the association (**Verified**: [r5 §3](#22-r5-rhythm-1-0-fullcontact-and-google-civic-enrichment-and-the-customer-360-precedent)). HubSpot withdrew its cross-customer sharing plan within four days of announcing it (**Market**: [w2 §2](#24-w2-evidence-how-enterprise-crm-erp-platforms-do-data-enrichment)). ### P8. Spend is bounded before the call, not discovered after An association must be able to limit enrichment spending before it happens and see what it has spent and on what. Before every provider call, Rhythm checks the association's monthly allowance and the provider connection's limits, then reserves the authorised cost. After the call, it settles that reservation using the adapter's reported `cost_units`. A bulk run estimates its cost before starting and refuses to start if the estimate exceeds the remaining allowance. This follows the brief's requirement. The integration framework already records `cost_units`, but nothing reads it (**Verified**: [r1 §10](#18-r1-the-integration-platform-as-built-evidence-for-a-data-enrichment-design)). Rhythm 1.0 had one global key, no budget, and a test disabled "once usage limits have been addressed" (**Verified**: [r5 §2](#22-r5-rhythm-1-0-fullcontact-and-google-civic-enrichment-and-the-customer-360-precedent)). The AI usage ledger is an existing implementation pattern to copy (**Verified**: [r3 §4.3](#20-r3-platform-core-evidence-for-the-data-enrichment-architecture)). ### P9. Licence and privacy are data, not documentation Staff should not have to interpret a provider's licence each time they display, export or use an enriched value. Rhythm must store the applicable permissions with the data and enforce them consistently. Each provider connection records the terms accepted for that connection: - Whether results may be shown to members. - Whether results may be exported. - Whether results may be kept after termination. - Whether results may be used in AI features. - How long results may be cached. Each stored claim carries the resulting licence class and expiry. Export, portal display, mail merge, webhooks and AI prompts all consult one policy function. Its default answer is no. Suppression, meaning "do not enrich this person", is kept in a separate ledger keyed by a salted hash, a one-way identifier rather than the original identifying value. This allows suppression to survive erasure of the record. The licences reviewed differ on every one of these permissions and change over time (**Market**: [w4 §5, §13](#26-w4-evidence-report-privacy-law-and-data-licensing-constraints-on-rhythm-2-0-data-enrichment); [w5 §2](#27-w5-supplementary-provider-checks-read-directly-2026-10-22-11-00-to-11-30-et)). Platform currently has no data classification for third-party-licensed fields in exports or queries (**Verified**: [r4 §12](#21-r4-data-operations-data-job-engine-and-the-query-layer-evidence-for-rhythm-2-0-data-enrichment)). ### P10. Small foundation, visible seams The proposed first implementation is deliberately limited: one provider per operation, the person and organization operations with contacts first, a review screen and a budget. The contract, policy and data model must leave clear extension points. A second provider, an ordered waterfall that tries providers in sequence, an employment-change watch, a member-facing control or an AI research step should be an addition, not a redesign. Those extension points do not authorise building the later features. Nothing beyond the first implementation is built until its milestone plan is approved (**Required**: R2-DEV-001). This follows the brief's request for a foundation that can grow without implementing everything. The market evidence also shows that Clay-style routing is a staff tool for sales operations, not a feature an association administrator configures (**Market**: [w2 §6](#24-w2-evidence-how-enterprise-crm-erp-platforms-do-data-enrichment)). The milestone standard forbids unapproved build-up. The first implementation remains proposed, not approved or built under this proposal. ### Standing rules this proposal treats as constraints The proposed architecture must also follow the existing Rhythm 2.0 constraints: - Custom fields are opt-in per entity (R2-ARCH-001; contacts and organizations are approved). - Operations must be submission-compatible (R2-ARCH-002: enrichment publishes no intake operation, recorded in [05 §6](#6-05-service-ownership-and-integration-platform-alignment)). - Work follows the milestone standard (R2-DEV-001). - Storage uses PostgreSQL on Aurora behind RDS Proxy, with transaction-local settings only. - Record ids use UUIDv7; configuration ids are strings. - Data and API names use snake_case; code uses camelCase. - Forced row-level security applies to every association-scoped table, enforcing access boundaries in the database. - Implementation follows the persistence layout and queued-work standards, the record write contracts, and the record merge participant protocol. - CRM owns the consent ledger (Marketing D-M2). - Every console screen comes from v0 first. The portal is built last on Andrew's components. - No Asana; Jira RX for tickets; Humanize for everything presented. ## 6. 05. Service ownership and Integration Platform alignment ## 05. Service ownership and Integration Platform alignment Status: proposed 2026-10-22. Labels as in the [README](#1-rhythm-2-0-data-enrichment-architecture). This document recommends an ownership boundary; it does not approve or implement it. The recommendation is decision **ENR-D1** in [15](#16-15-open-architectural-decisions), where Andrew can approve or overturn it. Existing infrastructure is distinguished from proposed work by the evidence labels below. ### 1. The recommendation in one paragraph The recommendation is to make enrichment a **Platform module**, `services-platform/modules/enrichment`, registered under its own application key, `enrichment`. It would own enrichment policy, requests, provider orchestration, claims, suggestions, provenance, the suppression ledger and the budget ledger. A claim records what a provider said; a suggestion proposes a record change; provenance records where a field's value came from. Enrichment would call providers only through the Integration Platform's gateway, using ordinary catalogue adapters, installations, connections, credentials and executions. It would never write a CRM table directly: CRM would apply accepted suggestions through its own write path. CRM would publish the `crm.contact.created` and `crm.organization.created` events that trigger enrichment, expose the write operation for applying suggestions, and retain ownership of affiliation, duplicate and consent rules. Other applications, including Events, Membership and Committees, would use the same enrichment capability by addressing a record as `(object_id, record_id)`, without needing to know CRM's internals. ### 2. The four candidate homes, assessed The ownership choice is about reuse and responsibility, not just where the code sits. Enrichment needs one place to assess provider answers and manage review, while each application must retain control of its records. The proposed Platform module gives it that boundary without adding a service. | Home | What it would mean | For | Against | Verdict | | ------------------------------------------ | --------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | | **Platform module** (recommended) | A module beside `integrations`, `llm`, `data-jobs`, `attention`; schema `platform`, tables `enrichment_*` | Sits next to everything it reuses: the gateway (same service, no cross-service hop for a provider call), the AI usage ledger pattern, Attention, business events, Durable Jobs, workflow actions, the asset rules; no new service principal, no library release for a new identity; the "record address" `(object_id, record_id)` is already how audit, attention and revisions address any record (**Verified**: [r3 §12](#20-r3-platform-core-evidence-for-the-data-enrichment-architecture)); Platform is the agreed exception for shared infrastructure tables (**Required**: `AGENTS.md`) | Platform has no bulk data-job lane (500-item cap in the request; **Verified**: [r4 §G3](#21-r4-data-operations-data-job-engine-and-the-query-layer-evidence-for-rhythm-2-0-data-enrichment)); keys under `platform.*` are never licence-gated (**Verified**: [r3 §2.2](#20-r3-platform-core-evidence-for-the-data-enrichment-architecture)); Platform's stack count and deploy time grow | Both objections are answered below: the fetch side does not use the data-job engine, and the module registers its own application key | | **Inside CRM** | Tables in `crm`, orchestration in services-crm | Closest to the records; the apply step is a local transaction; CRM already has the duplicate normalisation | Organizations are enriched for Events exhibitors, Membership organizational units and Committees too; a second record type (a custom object, an exhibitor) would force CRM to learn about it or force a copy; every provider call would cross from CRM to Platform's gateway and back; CRM is mid-stabilisation with twenty-three milestones queued (**Verified**: [r2 §2](#19-evidence-r2-crm-rhythm-2-0-crm-as-the-target-of-data-enrichment)); RHYTHM-CRM-ARCH-001 itself recommends "Integration Platform connectors writing through CRM's service functions; no CRM-internal vendor calls" (**Planned**: `crm-arch:05-functional-architecture.md:349`) | Rejected: it ties a cross-domain capability to one domain and adds to the most loaded roadmap | | **Only the integration infrastructure** | A capability contract and adapters, with each application calling `invoke()` itself and keeping its own results | Smallest Platform change; the contract already exists in outline | Every consumer re-implements identity acceptance, suggestions, provenance, budget and review; the brief's "one coherent capability" becomes N partial ones; the metering gap (`cost_units` unread) stays open; the Integration Platform's job is to connect, not to decide what a claim is worth (**Verified**: [r1 §Gaps](#18-r1-the-integration-platform-as-built-evidence-for-a-data-enrichment-design)) | Rejected as the whole answer; adopted as the provider layer | | **New macroservice `services-enrichment`** | Its own repository, stacks, principal, tables, bulk lane | Clean boundary; its own deploy cadence; could host a bulk lane | A new `svc_enrichment` principal needs an authorization-library release, a registry seed and IAM grants on every callee (**Verified**: [r3 §2.5](#20-r3-platform-core-evidence-for-the-data-enrichment-architecture)); it would still call Platform for the gateway, Attention, events, the LLM module and metering, so every operation is two hops; it adds a fourteenth macroservice for a capability whose first milestone is a few tables and one screen; the brief explicitly warns against a service "because enrichment sounds like an independent capability" | Rejected now; the module is shaped so extraction later is mechanical ([14 §2](#15-14-alternatives-and-tradeoffs)) | The verdicts above are recommendations within this proposal, not approved decisions. Provider fetching would use the gateway rather than the data-job engine. Licensing would use enrichment's own application key, as explained in section 7. The increase in Platform's stack count and deploy time remains a tradeoff. ### 3. Dependency direction and ownership The proposed boundary keeps provider evidence separate from the association's records. Enrichment would decide what to propose; staff or policy would decide whether to accept it; the application that owns the record would perform the write. CRM would remain responsible for deciding how affiliations, duplicates and consent affect that write. ```mermaid flowchart LR subgraph CRM[services-crm] C1[contacts, organizations,\naffiliations, custom field values] C2[duplicate detection,\nmatch keys, normalisation] C3[record revisions,\nchange events] C4[consent ledger\n(ruled, not built)] end subgraph PLAT[services-platform] E[modules/enrichment\npolicy · requests · claims ·\nsuggestions · provenance ·\nsuppression · allowance] I[modules/integrations\ncatalogue · connections ·\ncredentials · gateway ·\nexecutions · rate limits] L[modules/llm\n(AI research, later)] A[attention · business events ·\ndurable jobs · workflow actions] end subgraph ADAPT[adapter repositories] P1[integrations-pdl] P2[integrations-logodev] P3[integrations-gravatar] P4[integrations-emailverify] end C3 -- contact.created / updated --> E E -- invoke enrichment.* --> I I -- envelope + lease --> P1 & P2 & P3 & P4 E -- apply accepted suggestion\n(CRM write path) --> C1 E -- "is this the same person?" --> C2 E -- items, events, runs, actions --> A E -. research operation (ENR-M6) .-> L C4 -. "do not enrich" consent .-> E ``` The table assigns each responsibility to one owner. BYOK means the customer supplies its own provider key; `platform_account` means it uses a Rhythm-held provider account. A suppression ledger would enforce “do not enrich” instructions, while the member-facing consent record would remain in CRM. | Concern | Owner | Where it lives | Note | | ------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | --------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | Which providers exist, their manifests, versions, deployments | Integration Platform (HQ publishes) | `platform.integration_definitions`, `_versions`, `_deployments` | unchanged | | A tenant's provider accounts and keys | Integration Platform (tenant installs, connects) | `integration_installations`, `_connections`, `_credentials` | unchanged; BYOK and `platform_account` both exist (**Verified**: [r1 §3](#18-r1-the-integration-platform-as-built-evidence-for-a-data-enrichment-design)) | | Which provider answers which operation for this tenant | **Enrichment** (policy and plan), using explicit-connection invocation | `enrichment_policies`, `enrichment_provider_settings` | the gateway's one-connection-per-scope bindings are too narrow for an ordered plan ([07 §5](#8-07-provider-contracts-and-orchestration)) | | Making the call: validation, lease, retries, circuits, rate limits, execution history | Integration Platform (gateway) | `integration_executions`, `_attempts`, `_keys` | unchanged; `subject_ref` links each execution to the record | | What the provider said | **Enrichment** | `enrichment_claims` (append-only) | never edited, expires per licence | | Whether Rhythm believes it is the same person | **Enrichment**, using CRM's normalisation library | `enrichment_requests.match` | no merge decisions | | What to do about a claim | **Enrichment** (policy) and the reviewer | `enrichment_suggestions` | pending, accepted, rejected, applied | | Writing the record | **CRM** (or the owning application) | CRM's own persistence, history, events | enrichment submits; the owner writes | | Where each field's current value came from | **Enrichment** | `enrichment_field_provenance` | keyed by `(object_id, record_id, field_key)`; CRM's revisions remain the audit of the write | | Affiliation and employment rules | **CRM** | `docs/affiliations/README.md` | enrichment proposes an employer change; CRM's `set_primary_organization` performs it | | Duplicate candidates raised by a provider answer | **CRM** | `crm.contact_duplicate_candidates` | enrichment hands over a pair with `origin = 'enrichment'` (a CRM change) | | Member consent and "do not enrich" | **CRM** consent ledger (ruled) and the enrichment suppression ledger | `crm.*` (not built) and `platform.enrichment_suppressions` | the suppression ledger is the enforcement cache; the consent ledger is the member-facing record ([09 §3](#10-09-security-privacy-and-licensing)) | | Budget, allowance, usage | **Enrichment** | `enrichment_allowance_periods`, `enrichment_usage_daily` | the AI ledger pattern; migrates to licensing v2 when that exists | | AI research | Platform `llm` module (operation `enrichment.research`) | `modules/llm` | ENR-M6; not before cross-service LLM consumption is decided for the module's own use | | Screens | react-console (v0 first) and rhythm-web (last) | | [11](#12-11-console-and-portal-experience) | ### 4. How the module uses the Integration Platform, exactly Staff would install and connect providers through the existing Integration Library. Enrichment would then choose among those connections and use the gateway to make calls. The proposed additions are the enrichment contracts, adapters and policy, plus the specific Platform extensions identified below. 1. **Catalogue.** Each provider would have an adapter repository, `integrations-`, built with the software development kit (SDK) and published by `integration publish`. Registration requires adding its key to `REGISTERING_ADAPTER_KEYS`, a Platform change for each new key (**Verified**: [r1 §Gaps 9](#18-r1-the-integration-platform-as-built-evidence-for-a-data-enrichment-design)). The first wave would add `pdl`, `logodev`, `gravatar` and one e-mail verifier. The Dummy adapter would gain enrichment capabilities in stub mode for tests. 2. **Contracts.** The proposal adds a category, `enrichment`, in `@rhythmsoftware/integration-contracts` ([07](#8-07-provider-contracts-and-orchestration)). The release order is fixed: publish the contracts release, upgrade Platform to it, then compile adapters against it. This is the publication gate (**Verified**: [r1 §2.1](#18-r1-the-integration-platform-as-built-evidence-for-a-data-enrichment-design)). 3. **Installation and connection.** The association would install the provider in the Integration Library and connect an account using its own key or Rhythm's `platform_account` with an account qualifier. Enrichment would discover the workspace's eligible connections through `GET /integrations/{t}/connections?capability=enrichment.person.enrich` and record the chosen connections in its policy. 4. **Invocation.** Enrichment would select the connection explicitly by setting `connection_id`. This skips the gateway's binding resolution (**Verified**: [r1 §4.1](#18-r1-the-integration-platform-as-built-evidence-for-a-data-enrichment-design)). Each call would include `context.application = 'enrichment'`, `subject_ref = ':'`, and an idempotency key derived from the request and the identifier digest to prevent duplicate execution. Calls would use `mode: 'sync'` for single records or `async` for provider batches. 5. **Execution history.** Staff would see enrichment calls in the Library's Activity and Execution detail screens. No new console work is proposed beyond labels. Enrichment would store `execution_id` on every claim, linking provider evidence to the call that produced it. 6. **Rate limits and circuits.** Existing per-connection rate limits and circuit breakers would govern calls. A Rhythm-held key shared by many associations also needs a platform-wide bucket. That small extension to `rate-limits.ts` is recorded as a dependency of ENR-M5. 7. **Webhooks and events.** A provider that pushes changes, such as an employment-change feed, would deliver them through the webhook ingress and integration-events consumer registration. Enrichment would register `svc_platform` as a consumer of `enrichment.person.changed` in ENR-M4. 8. **Personal data.** Provider payloads must not be recorded when the capability handles personal data. The contracts registry would mark enrichment capabilities `personal_data: true`, and Platform must refuse `record_payloads` for those capabilities before the first adapter ships. The existing gap is in `payload-recording.ts:31-37` (**Verified**: [r1 §12](#18-r1-the-integration-platform-as-built-evidence-for-a-data-enrichment-design)). The fix is in ENR-M1's scope. ### 5. What CRM must add, and when CRM needs to expose events and controlled write operations, not take on provider orchestration. Those changes would let enrichment propose updates while preserving CRM's version checks, history and domain rules. | CRM change | Needed by | CRM milestone that carries it | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | | Publish `crm.contact.created` and `crm.organization.created` business events (declared in the catalogue, not published; **Verified**: [r2 §12](#19-evidence-r2-crm-rhythm-2-0-crm-as-the-target-of-data-enrichment)) | ENR-M1 enrich-on-create | CRM-M15 (Platform-facing contracts), already planned | | A service-principal write path: widen `crm.record_revisions` sources to accept `service` (**Verified**: [r2 §4](#19-evidence-r2-crm-rhythm-2-0-crm-as-the-target-of-data-enrichment)) and a `contacts` / `organizations` service function with an `apply_enrichment` operation that takes `{ record_id, expected_version, fields, custom_fields, provenance_ref }` | ENR-M2 automatic fill | CRM-M15 | | Explicit add-e-mail, add-phone, add-address and affiliation operations (ruled D-09, not built; **Verified**: [r2 §3.4](#19-evidence-r2-crm-rhythm-2-0-crm-as-the-target-of-data-enrichment)) | ENR-M2 collections, ENR-M4 employment | CRM-M14, CRM-M15 | | A duplicate-candidate origin `enrichment` | ENR-M4 | CRM-M16 | | Merge policy for enrichment tables keyed by contact id (Platform implements the participant; CRM's coverage test must know the tables) | ENR-M1 | none; a Platform participant function per the protocol (**Verified**: [r4 §9](#21-r4-data-operations-data-job-engine-and-the-query-layer-evidence-for-rhythm-2-0-data-enrichment)) | Until CRM-M15 lands, the proposed ENR-M1 apply path would require a reviewer. Until the business events are published, ENR-M1 takes the same "created" signal from CRM's internal change-event stream ([13 §2](#14-13-phased-implementation-milestones)); organizations use the matching `crm.organizations update` data job, which also declares a rollback. Accepted suggestions would be submitted as a CRM data job on the reviewer's behalf, using the `records[]` form of `crm.contacts update`. That operation already exists and is versioned and reversible for scalar fields (**Verified**: [r4 §I-1](#21-r4-data-operations-data-job-engine-and-the-query-layer-evidence-for-rhythm-2-0-data-enrichment)). It would deliberately be the first milestone's only apply path: a person would always be in the loop. ### 6. Submission-compatible operations (R2-ARCH-002) The proposal publishes **no** enrichment business operation to Submissions. Requesting enrichment, deciding a suggestion and running a refresh are staff data-quality actions on existing records, not intake operations. This is the deliberate “not published” entry that rule 1 requires. If a future process needs to “enrich the applicant's organization on submit”, it would call the workflow action `enrichment.enrich_record` after fulfilment. That is the sanctioned automation path ([10 §5](#11-10-api-and-event-architecture)). ### 7. Licensing The proposed licensing boundary separates baseline tools from premium services. Logos, avatars, verification and suggestions review with a customer-provided key would be available to every workspace through an HQ default licence. Rhythm-managed credits, scheduled refresh, employment watch and AI research would require both the relevant licence and allowance ([12](#13-12-commercial-model-and-economics), decision ENR-D8). To enforce that boundary, `enrichment` would be registered in `APPLICATIONS` with `enrichment.application.access`. Its own application key is necessary even though the module would live in Platform: `platform.*` keys are exempt from licence checks (**Verified**: [r3 §2.2](#20-r3-platform-core-evidence-for-the-data-enrichment-architecture)). This is the only way to gate a Platform-hosted capability. The registration and licence model remain proposed, not approved or built. ### 8. As ruled: enrichment inside CRM (Andrew, 2026-10-22 about 11:50 ET) Andrew ruled ENR-D1 for option (b): the module lives in **services-crm** as `modules/enrichment`, schema `crm`, tables `crm.enrichment_*`. Everything else in this package stands; what changes is where the same pieces sit and which seams disappear. The rulings on ENR-D2, D3, D4 and D8 were as recommended; ENR-D6 was ruled to offer tier 3 from ENR-M1. | Topic | As recommended (Platform) | As ruled (CRM) | | ----------------------------------------- | ------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Home and ownership | `services-platform/modules/enrichment` | `services-crm/modules/enrichment`; CRM owns policy, requests, claims, suggestions, provenance, suppressions and the allowance ledger | | Applying a suggestion | a CRM data job (ENR-M1) or a CRM service function (ENR-M2+) across a service boundary | a local write through CRM's own persistence in the same transaction as the suggestion's state change, with revision capture (`sys_source` from the context, operation `enrichment.apply`) and change events; no interim data-job path is needed, and the `service` revision source stays unused | | Triggers | Platform subscribes to CRM's business events (CRM-M15) | CRM's internal change-event handlers (the ones duplicate detection and Quick Search use today) raise enrichment directly; the published business events are no longer a dependency of ENR-M1 | | Calling providers | the gateway in the same service | the gateway's service function over IAM, exactly as Accounting calls `payments.*`; no change to the Integration Platform | | Metering | a copy of the AI ledger in Platform | the same ledger copied into CRM; when licensing v2 exists, `recurring_credits` is fed from it | | Bulk fetch | a Durable Job plus queued work in Platform, which has no bulk data-job lane | the same Durable Job plus queued work in CRM, which also has a bulk data-job lane for the apply side | | Other applications | call Platform's enrichment service function by `(object_id, record_id)` | call a CRM enrichment service function by the same address; organizations enriched for Events, Membership and Committees are CRM organizations, so the address still resolves without those applications knowing CRM's internals | | Merge, lifecycle, reset | Platform implements the participant protocol for its tables | CRM's own merge planner and lifecycle cover the new tables directly; no cross-service participant | | Licensing | `enrichment` application key registered by Platform | the same key registered by CRM, which publishes `enrichment.*` permission keys beside `crm.*` (Platform publishes `platform.*` and `hq.*` the same way); `crm.*` stays always available, `enrichment.*` is licence-gated | | Milestone records | services-platform `docs/milestones/ENR-M-…` | services-crm `docs/milestones/ENR-M-…`; the application for R2-DEV-001 rule 4 is `enrichment`, not CRM, so an enrichment milestone does not count against CRM's own one-in-progress limit | | A second enriched record type outside CRM | addressed by Platform generically | would need a CRM service function the owning application calls; recorded as a known limit of the ruling | What this costs: enrichment is tied to CRM's deploy cadence and roadmap, and a future non-CRM record type needs a cross-service path. What it buys: the apply step is a local transaction with no interim, ENR-M1 loses its CRM-M15 dependency entirely, and the merge, lifecycle and bulk-lane seams close. The chapters that follow are read with this table; their text keeps the Platform wording where it describes mechanisms that are identical in either home. ## 7. 06. Core workflows and use cases ## 06. Core workflows and use cases Status: proposed 2026-10-22. Nothing in this enrichment design is decided or built. Labels are as defined in the [README](#1-rhythm-2-0-data-enrichment-architecture). The proposed design keeps enrichment out of contact creation, registration, checkout and imports. Those business processes finish first. Enrichment then runs through a shared queue, checks policy and budget, and produces suggestions. CRM remains responsible for applying changes. Employment changes always require review. This chapter shows what staff and members would experience, how each workflow starts, which existing services it reuses, and what it writes. The five sequence diagrams requested by the brief are §1, §2, §3, §4 and §6. Milestone labels show when each capability is proposed to become available. ### 0. The trigger table | Trigger | Who starts it | Sync or async | Enters through | Wave | | ----------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | ---------------------------------- | | Contact or organization created | CRM's `crm.contact.created` / `crm.organization.created` business event (CRM-M15) | async, after CRM's commit | event intake → queued work | ENR-M1 | | Membership registration, event registration, checkout | the owning application's own "completed" event (`membership.membership.activated`, Events' registration events) | async, after the order is complete; **never** in the request | event intake → queued work | ENR-M2 (Membership), later per app | | Portal profile update | CRM change event with `sys_source = 'portal'` | async | change-event subscriber; marks the saved fields `member`, `locked`, then re-enriches only unlocked groups | ENR-M2 | | Import or migration | the Data Job Engine's job-completed notification for `crm.contacts insert`/`upsert` | async, batched as a run of kind `import_followup` | job event → run | ENR-M3 | | Bulk administrative action | a staff member selects records in the console | async run; the console shows progress | `POST /enrichment/{t}/runs` | ENR-M3 | | Workflow automation | a tenant rule invokes the workflow action `enrichment.enrich_record` | durable action; the rule continues when the request completes | workflow action executor | ENR-M3 | | Manual "Enrich now" | a staff member on a 360 screen | sync-looking: the screen waits up to the gateway budget and then shows "still working" | `POST /enrichment/{t}/requests` | ENR-M1 | | Scheduled refresh | an EventBridge rule, one Durable Job per tenant per day | async run of kind `scheduled_refresh` | sweep on `next_refresh_at` | ENR-M3 | | Detected change in external data | a provider's change feed (`enrichment.person.changed`) | async | integration-events consumer | ENR-M4 | Every trigger uses the same processing path: **one queued-work row per subject**, where a subject is the contact or organization being enriched. One Lambda consumes that work. It builds the provider plan, checks suppression and budget, calls the Integrations gateway, turns provider claims into suggestions, and publishes the result. A claim is a value supplied by a provider; a suggestion is a proposed CRM change evaluated against the association's policy. The trigger determines only the work's priority and which field groups to request. It does not create a separate enrichment implementation for each application. ### 1. New contact enrichment (on create, suggestions for review) Contact creation would remain as fast as it is today. Enrichment starts only after CRM has saved the contact, and staff review the resulting suggestions on Contact 360. ```mermaid sequenceDiagram autonumber participant Staff as Staff (console) participant CRM as services-crm participant Ev as Platform business events participant Enr as modules/enrichment participant Q as enrichment queue (SQS) participant GW as Integrations gateway participant Grav as integrations-gravatar participant PDL as integrations-pdl participant Att as Attention Staff->>CRM: POST /contacts (name, personal e-mail, employer) CRM-->>Staff: 201 (the contact exists; nothing waited) CRM->>Ev: outbox: crm.contact.created Ev->>Enr: platform_event_intake (envelope.id dedup) Enr->>Enr: policy on? trigger on_create? subject suppressed? identifier digest new? Enr->>Q: queue row (subject, trigger=on_create) Q->>Enr: consumer (reserved concurrency) Enr->>Enr: build plan: [gravatar d=404 probe, pdl person.enrich tier 2]; reserve units Enr->>GW: invoke enrichment.person.enrich@1 (connection=gravatar, sync) GW->>Grav: envelope Grav-->>GW: matched {person.photo_url self_published} cost 0 Enr->>GW: invoke enrichment.person.enrich@1 (connection=pdl, sync, idempotency=request:digest) GW->>PDL: envelope + lease PDL-->>GW: matched, likelihood 7, claims[job_title, employer, linkedin] cost 1 GW-->>Enr: succeeded, execution_id Enr->>Enr: corroborate (name agrees, employer domain agrees) → accepted Enr->>Enr: store claims; evaluate policy → suggestions pending (job_title fill, employer employment_change, photo) Enr->>Enr: settle 1 unit; subject status=enriched; request completed Enr->>Att: item "3 suggestions for Jane Doe" to crm.data_quality_review Enr->>Ev: enrichment.request.completed, enrichment.suggestion.created ×3 Staff->>Enr: GET /enrichment/{t}/subjects/crm.contacts/{id} (360 panel) Staff->>Enr: POST /suggestions/{id}/decisions {accept} Enr->>CRM: data job crm.contacts update records[{id, expected_version, fields}] (as the reviewer) CRM-->>Enr: receipt applied, revision id Enr->>Enr: suggestion applied; provenance job_title = provider (pdl, claim) ``` The member or staff member would see the contact created in the time CRM normally takes. Minutes later, Contact 360 would show a badge, "3 suggestions". In this example, the reviewer accepts the job title, rejects the photo, and sends the employer change to the affiliation flow (§6). The accepted job title is applied through CRM's own update mechanism. CRM's history therefore records the reviewer as the actor and the data job as the source. Enrichment's provenance record, the record of where a value came from, identifies the provider and its claim. ### 2. Registration-time enrichment (never in the critical path) A provider outage must not prevent someone from becoming a member. The proposed workflow listens for the completed business process rather than adding a provider call to registration or payment. ```mermaid sequenceDiagram autonumber participant Member as Member (portal) participant Mem as services-membership participant Com as Commerce / Accounting participant Ev as Platform business events participant Enr as modules/enrichment participant GW as Integrations gateway participant Ver as integrations-emailverify Member->>Mem: submit application, pay Mem->>Com: order, tender Com-->>Mem: paid Mem-->>Member: confirmation (seconds; no provider involved) Mem->>Ev: membership.membership.activated Ev->>Enr: intake Enr->>Enr: policy on_registration? tier for this member's region? suppressed? Enr->>Enr: queue two subjects: the contact (groups: contact_channels, professional) and the organization (brand, firmographics) Enr->>GW: enrichment.email.verify@1 (contact's primary e-mail) GW->>Ver: envelope Ver-->>GW: deliverable, free_provider=true, cost 1 Enr->>Enr: claim person.email.status; no suggestion needed (status stored in provenance; CRM e-mail row untouched until CRM-M14's explicit e-mail operations) Enr->>GW: enrichment.organization.enrich@1 (logodev; domain from the employer) GW-->>Enr: matched {organization.logo_url} licence open Enr->>Enr: rule brand=auto_fill, logo empty → suggestion accepted auto → apply via CRM service function (ENR-M2) through the asset pipeline Enr->>Ev: enrichment.request.completed ×2 ``` This separation is structural, not a timeout. Enrichment subscribes to the business process's _completed_ event. Even if every provider is down, the member is still a member and the organization still exists. E-mail verification would initially inform staff without changing the CRM e-mail row. CRM has no verification status column today (**Verified**: [r2 §2.1](#19-evidence-r2-crm-rhythm-2-0-crm-as-the-target-of-data-enrichment)). The result would be held in provenance and shown on the 360 screen until CRM-M14 adds explicit e-mail operations. At that point, an `undeliverable` result would become a suggestion to mark the e-mail inactive. ### 3. Bulk enrichment (staff-selected records, budget-gated, applied through a data job) Staff would see an estimate before starting a bulk run, then progress and results. Fetching provider data and applying accepted changes are separate operations: the first spends enrichment budget; the second uses CRM's safeguards for bulk updates. ```mermaid sequenceDiagram autonumber participant Staff as Staff (console) participant Enr as modules/enrichment participant Job as Durable Job (lib/jobs) participant Q as enrichment queue participant GW as Integrations gateway participant P as providers participant DJE as CRM data job (apply) Staff->>Enr: POST /enrichment/{t}/runs {object: crm.contacts, record_ids[≤10,000], operation: person.enrich, groups} Enr->>Enr: estimate = Σ per-subject plan units (cache hits and suppressed excluded); compare with allowance Enr-->>Staff: 200 {estimate_units, remaining_units, subjects: 8,214 eligible, 311 suppressed, 1,475 cached} (dry run) Staff->>Enr: POST /runs/{id}/start Enr->>Enr: reserve estimate; start Durable Job Job->>Q: one queued row per subject (batches of 25) loop per subject, under the connection's rate limit and the consumer's concurrency Q->>Enr: consume Enr->>GW: invoke (sync) or batch (async, 1,000 subjects) when the adapter declares `batch` GW->>P: envelope P-->>GW: result Enr->>Enr: claims → suggestions; settle units; counts++ end Job-->>Enr: run completed {matched 3,900, no_match 4,100, failed 214} Enr-->>Staff: Notification "Run finished: 3,900 matched, 6,120 suggestions" Staff->>Enr: POST /suggestions/decisions {filter: run_id, kind: fill, field_group: professional, decision: accept} Enr->>DJE: submit crm.contacts update, records[] (per-record values, expected_version each), as the reviewer DJE-->>Enr: job accepted; receipts stream back Enr->>Enr: suggestions applied / apply_failed(record_changed) per receipt ``` The separation reuses existing infrastructure without putting external calls inside CRM transactions: - **Fetch:** a Durable Job, a background job that tracks work through completion, feeds queued work to the enrichment consumer. Provider calls must never run inside a data-job batch transaction (**Verified**: [r4 §G1](#21-r4-data-operations-data-job-engine-and-the-query-layer-evidence-for-rhythm-2-0-data-enrichment)). - **Apply:** a CRM data job writes accepted changes. This provides record-version capture, named conflicts, per-row receipts, rollback for scalar fields, and the PLAT-M20 dialog (**Verified**: [r4 §I-1](#21-r4-data-operations-data-job-engine-and-the-query-layer-evidence-for-rhythm-2-0-data-enrichment)). Bulk acceptance remains subject to the data job's limits. It can apply scalars and custom fields now. E-mails, phones, addresses and affiliations must wait for CRM's explicit operations (CRM-M14, CRM-M15). ### 4. Scheduled continuous enrichment (refresh, bounded by cadence and budget) Scheduled refresh would keep verification dates current without producing suggestions for unchanged values. It is off by default and, when enabled, spreads work across the year within the association's budget. ```mermaid sequenceDiagram autonumber participant Cron as EventBridge rule (daily) participant Sweep as enrichment refresh scheduler (concurrency 1) participant Job as Durable Job per tenant participant Q as enrichment queue participant Enr as modules/enrichment participant GW as Integrations gateway Cron->>Sweep: 02:00 tenant local (macroservice-schedules cadence) Sweep->>Sweep: for each tenant with refresh.cadence != off and allowance remaining Sweep->>Job: start run kind=scheduled_refresh, selection = subjects where next_refresh_at <= today, limit max_per_day Job->>Q: queued rows Q->>Enr: consume; plan uses provider_person_id for a cheaper refresh where the adapter declares input.provider_id Enr->>GW: invoke GW-->>Enr: result (often "same values": provenance verified_at updated, no suggestion) Enr->>Enr: next_refresh_at = now + cadence; settle units Job-->>Sweep: completed Sweep->>Sweep: if allowance crossed 80 %: enrichment.budget.threshold_crossed → Notification to administrators ``` The default cadence is `off`. When enabled, the proposed cadence is `annual`, with `max_per_day` sized to cycle through the association's whole base in a year within budget. The example is 10,000 contacts at 40 a day. A refresh that finds the same values creates no suggestion. It updates `verified_at`, which is what "last verification date" means in the 360 panel. A different employer follows the employment-change path (§6). The scheduler would be one function with concurrency 1, following the pattern used by every other Platform sweeper (**Verified**: [r3 §7](#20-r3-platform-core-evidence-for-the-data-enrichment-architecture)). ### 5. Imports, portal updates and workflow actions These entry points use the same queue, but differ in which fields they protect and what work they request. - **Import follow-up (ENR-M3).** After a `crm.contacts insert` or `upsert` job completes, the module would start an `import_followup` run over the job's applied record ids. It requests only the groups the import left empty. Imported values are recorded as `source_kind = 'import'` in provenance, but are not locked. An import is not a person's own statement, so policy may still suggest changes to those values. - **Portal profile update (ENR-M2).** A member's saved values take precedence over provider claims. A contact revision with `sys_source = 'portal'` would lock every field it touched, mark it `source_kind = 'member'`, and expire pending suggestions for those fields as `superseded`. Unlocked groups would be re-enriched only if `on_portal_update` is enabled in policy. This option exists because adding a work e-mail can make tier-2 enrichment possible. - **Workflow action (ENR-M3).** An association's automation rule could request enrichment and branch on the result. `enrichment.enrich_record` is proposed as a `durable` action type with params `{ object_id, record_id, groups[], wait_for: 'request' | 'none' }`. A service-function executor would execute it idempotently on the run id, so a retry would not create duplicate work. It requires `enrichment.requests.create`, following the workflow-action contract (**Verified**: [r3 §8](#20-r3-platform-core-evidence-for-the-data-enrichment-architecture)). Its output is `{ request_id, outcome, suggestions_created }`, allowing a rule to branch on "matched". ### 6. Employment change detection and approval (affiliations are proposed, never overwritten) **An employer claim must always be reviewed.** A wrong automatic change could end a real affiliation, revoke roles and alter an organization's compliance. Rhythm 2.0 does not store an organization directly on a contact. The employer is the primary affiliation, which has dates, a type and a job title, with automatic roles that follow it (**Verified**: [r2 §3](#19-evidence-r2-crm-rhythm-2-0-crm-as-the-target-of-data-enrichment)). The proposed workflow therefore shows the reviewer both affiliations and the consequences before anything changes. ```mermaid sequenceDiagram autonumber participant Src as provider (refresh or watch feed) participant Enr as modules/enrichment participant CRMd as CRM duplicate check (organizations) participant Att as Attention participant Rev as Reviewer (console) participant Ev as Platform business events participant CRM as services-crm (contact aggregate) Src-->>Enr: claim person.employer {name: "Globex Corp", domain: globex.com, title: "Director", started_on: 2026-08} Enr->>Enr: compare with the contact's current primary affiliation (Acme, since 2021) Enr->>CRMd: POST /duplicates/{t}/organizations/checks {name, domain} (synchronous, read-only) CRMd-->>Enr: candidates [{Globex Corporation, band very_high, org id}] or none alt exactly one high-band organization Enr->>Enr: suggestion employment_change {organization_id, job_title, started_on, previous: 'end'} pending else none or several Enr->>Enr: suggestion employment_change {organization_candidate: {name, domain, claims}} pending, needs "create or pick organization" end Enr->>Att: item "Possible employer change: Jane Doe → Globex" to the reviewer responsibility Enr->>Ev: enrichment.employment_change.detected Rev->>Enr: GET suggestion (shows current affiliation, proposed one, provider evidence, dates, what will end) Rev->>Enr: accept {effective_date, previous: 'end' | 'keep', create_organization?: true} opt create_organization Enr->>CRM: organizations.match_or_create (CRM-M15) with the organization claims as a new organization suggestion set end Enr->>CRM: contacts.apply_enrichment {set_primary_organization: {organization_id, effective_date, previous_affiliation}, job_title} CRM-->>Enr: revision id (with previous: 'end': old affiliation ended the day before; new primary created, title synced, automatic roles recalculated) Enr->>Enr: suggestion applied; provenance person.employer = provider; subject provider_ids updated ``` This path follows CRM's rules (**Verified**: [r2 §3.3](#19-evidence-r2-crm-rhythm-2-0-crm-as-the-target-of-data-enrichment)): - The employer must be an existing organization or one created through CRM's match-or-create. It cannot be free text. - CRM refuses a future-dated primary affiliation. A claimed start date after today therefore becomes "today", with the provider's claimed date retained in the note. - Ending the previous affiliation must be explicit, using `previous: 'end'`, and audited. - The old affiliation keeps its title. Enrichment would never end an affiliation or create an organization on its own. It would never auto-apply an employment change under any policy (ENR-D10). Until CRM-M15's `apply_enrichment` exists, accepting a suggestion would open CRM's own edit screen with the proposed values pre-filled. The suggestion would be marked applied when the contact's change event shows the new primary affiliation. This is ENR-M1's "assisted apply" for affiliations ([13 §2](#14-13-phased-implementation-milestones)). **Watch feeds (ENR-M4).** A watch feed lets a provider notify Rhythm when its data about a subject changes. `enrichment.person.watch@1` would subscribe the subject with the provider. The adapter's feed emits `enrichment.person.changed`, which the module would treat as a claim set entering this same review path. PDL's changelog is monthly and enterprise-only; LinkedIn-sourced watchers carry supply-side risk (**Market**: [w3 §11.C, §6](#25-w3-evidence-enrichment-provider-landscape-for-an-association-management-system)). Watching is therefore proposed for a later milestone and would be a per-provider decision. ### 7. Identity resolution without merging Enrichment must establish that a provider is describing the right contact before treating its claims as credible. That is a different question from whether two CRM records are duplicates: - Enrichment asks, "Is the provider talking about _this_ record?" - Duplicate detection asks, "Are _these two_ records one person?" They share only the normalisation library: names, nicknames, metaphone for comparing names by sound, E.164 phone formatting, and free-mail domains (**Verified**: [r2 §6.1](#19-evidence-r2-crm-rhythm-2-0-crm-as-the-target-of-data-enrichment)). They do not share responsibility for merging records. The proposed person-match acceptance rules apply in this order: 1. If the provider supplies `likelihood`, it must meet or exceed the step's minimum, default 6. 2. There must be at least one corroboration against the CRM record: family-name sound agreement **and** given-name agreement using a canonical nickname; employer domain agreement, where the contact's work e-mail domain or primary affiliation's organization domain equals a claimed employer domain; or location agreement on city and region. 3. A provider without a score, such as FullContact, requires two corroborations. 4. A family-name disagreement with an e-mail match and given-name agreement is accepted as a `change` suggestion on `person.name.family`, never rejected for the surname difference. People change their names, and their e-mail addresses can outlive those names. 5. A subject with only a personal e-mail and no name agreement receives `match.accepted = false`. Its claims are still stored because they cost money, but are shown under "low confidence" only when a reviewer asks. If a provider returns `ambiguous` candidates, or a claim makes two Rhythm records look like one person, the module would create a `duplicate_hint` suggestion. One example is a claimed work e-mail that already belongs to another contact. In ENR-M4, the pair would also be passed to CRM's duplicate candidates with origin `enrichment`. The module would never merge records and would never ask CRM to merge them. ### 8. Use cases the design must keep serving These cases test whether the proposed workflows preserve member control, association policy, provider licence obligations and CRM ownership of records. | Use case | What happens | | ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | A chapter administrator imports 2,000 members from a spreadsheet with personal e-mails and no employers | Import completes; import follow-up run requests `professional` for the 2,000 (tier 2 needs a work e-mail or name plus location; the plan skips subjects with nothing to corroborate); suggestions appear by group; the administrator accepts job titles in bulk and reviews employers one by one | | A member changes jobs and updates their portal profile | The portal save locks the fields the member touched and ends enrichment's interest in them; the old pending employer suggestion is superseded | | A member asks not to be enriched | Staff (or the member in the portal, later) record "do not enrich"; a suppression row is written; the next trigger skips the subject; existing claims and suggestions for the subject are deleted and the provenance rows keep `source_kind` only | | An association in the EU turns enrichment on | Region rules cap EU/UK members at tier 2 and refuse tier 2 until a notice date is recorded; tier 3 is never offered for them | | A provider contract ends | HQ retires the connection; the retention job deletes claims of that connection within the licence's deletion window; provenance keeps `source_kind = provider` with no claim link; the subjects become `scheduled` for the next provider | | Two enriched contacts are merged | CRM's merge calls Platform's participant: subjects collapse, pending suggestions re-point, provenance keeps the survivor's rows | ## 8. 07. Provider contracts and orchestration ## 07. Provider contracts and orchestration Status: proposed 2026-10-22. Nothing in this proposed design is decided or built. The package's decisions are in [15](#16-15-open-architectural-decisions) for Andrew to approve or overturn. Labels follow the [README](#1-rhythm-2-0-data-enrichment-architecture). The recommendation is to give person and organization enrichment their own contracts, with a small shared vocabulary of fields. Each provider would translate its results into that vocabulary. Rhythm would then run an ordered plan of at most three provider steps, rather than a general-purpose workflow graph. This keeps provider differences out of the CRM and gives association staff predictable results, costs and usage restrictions. The contract shapes below are illustrative TypeScript, written to the conventions of `@rhythmsoftware/integration-contracts` 0.7.1 (**Verified**: [r1 §2.1](#18-r1-the-integration-platform-as-built-evidence-for-a-data-enrichment-design)). The proposal generates JSON Schema documents from them at build time. Decision **ENR-D2** in [15](#16-15-open-architectural-decisions) covers the vocabulary. ### 1. What exists and why it is not enough The existing `crm.contact.enrich@1` would remain unchanged for a future district adapter. Person and organization enrichment need a separate category because the existing contract was designed for address-based district lookup, not identity matching. `crm.contact.enrich@1` has request `{ address, fields_requested[] }` and response `{ matched, fields }`. It is synchronous, has no idempotency, no lookup counterpart and no features, and declares `personal_data: true`. It was written for the 1.0 Google Civic district lookup; nothing implements or calls it (**Verified**: [r1 §A.2](#18-r1-the-integration-platform-as-built-evidence-for-a-data-enrichment-design)). It cannot express a person or organization lookup by e-mail, name or domain. It also passes through the provider's field names and has no members for confidence, licence or batch processing. Those limits make it unsuitable as the foundation for the proposed enrichment service. ### 2. The claim vocabulary A **claim** is one assertion about one field of one record. For example, a provider might claim a contact's job title or an organization's domain. A claim is evidence, not an instruction to overwrite the CRM. Field keys belong to Rhythm and are fixed in the contracts package. Every adapter maps its provider's schema onto those keys. If a provider cannot produce a field, it makes no claim for that field. This gives staff and policy rules the same meaning regardless of which provider supplied the data. The first vocabulary: | Field key | Type | Record | Typical sources | | --------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | --------------- | ------------------------------------------------------------------------------------- | | `person.name.given`, `person.name.family`, `person.name.full` | string | contact | person providers | | `person.job_title` | string | contact | person providers, AI research | | `person.employer` | employer object: `{ name, domain?, linkedin_url?, identifiers?: { lei?, duns? }, title?, started_on?, ended_on?, current }` | contact | person providers (employment history), AI research | | `person.employment_history[]` | employer objects, past | contact | PDL `experience` | | `person.photo_url` | URL plus `source_kind` (`gravatar`, `provider`, `public_web`) | contact | Gravatar, person providers | | `person.profile.linkedin`, `.twitter`, `.facebook`, `.instagram`, `.github`, `.website` | URL | contact | person providers | | `person.location.city`, `.region`, `.country` | string | contact | person providers | | `person.bio` | string | contact | person providers, AI research | | `person.email.status` | `deliverable`, `undeliverable`, `risky`, `unknown` plus `reason` | contact e-mail | e-mail verifiers | | `person.phone` | E.164 plus `kind` | contact | person providers (tier 3) | | `person.education[]`, `person.certifications[]`, `person.skills[]` | objects | contact | PDL (thin; **Market**: [w3 §11.A](#25-w3-evidence-enrichment-provider-landscape-for-an-association-management-system)) | | `organization.name`, `.legal_name` | string | organization | company providers, registries | | `organization.domain`, `.alternate_domains[]` | string | organization | company providers | | `organization.logo_url` | URL plus `source_kind` | organization | Logo.dev, company providers | | `organization.description` | string | organization | company providers, AI research | | `organization.industry`, `.employee_range`, `.founded_year`, `.type` | string, range, int, enum | organization | company providers (custom fields in CRM, **Verified**: [r2 §1.2](#19-evidence-r2-crm-rhythm-2-0-crm-as-the-target-of-data-enrichment)) | | `organization.location.*` | address parts | organization | company providers | | `organization.profile.linkedin`, `.twitter`, `.facebook` | URL | organization | company providers | | `organization.identifiers.lei`, `.duns`, `.wikidata`, `.companies_house`, `.sec_cik` | string | organization | GLEIF, Wikidata, D&B | | `address.district.*` | strings | contact address | the existing `crm.contact.enrich` lineage (not in wave one) | Policy uses field groups to control related fields together ([08 §3](#9-08-data-model-provenance-and-policies)): - `identity`: names. - `professional`: job title, employer, history, education and certifications. - `social`: profiles. - `photo`: photos. - `location`: location fields. - `contact_channels`: e-mail status and phone. - `firmographics`: organization characteristics. - `brand`: logo and description. - `identifiers`: identifiers. Adding a field key requires a minor contracts release; renaming or removing one requires a major release. The vocabulary is shared by every provider, policy and screen. It should therefore stay deliberately small and grow through explicit decisions, not simply because a provider offers more fields. ### 3. The capabilities The proposed category is `enrichment`. Adding it requires a contracts change because the category list is closed in `naming.ts` (**Verified**: [r1 §2.1](#18-r1-the-integration-platform-as-built-evidence-for-a-data-enrichment-design)). The capabilities separate person lookup, organization lookup, e-mail verification and ongoing employment monitoring. Idempotency controls duplicate requests; a lookup counterpart checks the status of an uncertain operation. | Key | Mode | Idempotency | Sensitive | Personal data | Lookup | Timeout | Events | | ---------------------------------- | ------------------------ | ------------------------------------------------- | --------- | ----------------- | ----------------------------------------------------------------------------------------------------- | ------------------------------- | -------------------------------------------------------------- | | `enrichment.person.enrich@1` | `either`, default `sync` | `optional` (key = request id + identifier digest) | no | yes | none; an `unknown` outcome is treated as `no_match` and retried on the next trigger, never reconciled | 25 s sync; 15 min async attempt | `enrichment.person.enrich.completed` (async only) | | `enrichment.organization.enrich@1` | `either`, default `sync` | `optional` | no | no (domain-keyed) | none | 25 s | as above | | `enrichment.email.verify@1` | `sync` | `optional` | no | yes | none | 10 s | none | | `enrichment.person.watch@1` | `async` | `required` (key = subject) | no | yes | `enrichment.person.watch.status@1` | 15 min | `enrichment.person.changed`, `enrichment.person.watch.stopped` | The enrich capabilities do not need lookup counterparts because an enrichment read has no side effect to settle. The reconciler settles money and other side effects (**Verified**: [r1 §4.4](#18-r1-the-integration-platform-as-built-evidence-for-a-data-enrichment-design)); it is not needed to resolve a missing enrichment answer. The module would treat `unknown` as “no answer this time,” leaving the subject due for another attempt on the next trigger. This satisfies the contracts registry's rule that a capability without a lookup “cannot be sensitive.” Here, sensitive is the registry's side-effect classification, distinct from the personal-data declaration in the table. #### 3.1 `enrichment.person.enrich@1` The request supplies what Rhythm already knows about a contact. The adapter uses the identifiers its provider accepts. The requested field groups and policy tier limit what it may return; the response distinguishes a match, no match and an ambiguous result. ```ts // Request: what Rhythm already knows. Adapters use what they accept and ignore the rest; the gateway strips nothing. export interface PersonEnrichRequest { subject_ref: string; // 'crm.contacts:019a…' for logs and the provider's optional record id; never a name identifiers: { emails?: Array<{ value: string; kind: 'work' | 'personal' | 'unknown'; primary: boolean }>; phones?: Array<{ value: string; kind: 'mobile' | 'work' | 'home' | 'unknown' }>; name?: { given?: string; family?: string; full?: string }; location?: { city?: string; region?: string; postal_code?: string; country?: string }; employer?: { name?: string; domain?: string }; profiles?: Array<{ network: 'linkedin' | 'twitter' | 'facebook' | 'github' | 'other'; url: string }>; provider_person_id?: string; // the provider's own id from an earlier claim, for a refresh }; requested_groups: FieldGroup[]; // 'professional' | 'social' | 'photo' | 'location' | 'identity' | 'contact_channels' | 'education' minimum_likelihood?: number; // 0..10; the adapter may decline to answer below it (saves a billable call where the provider pre-scores) tier: 1 | 2 | 3; // the policy tier this call runs under; an adapter refuses a tier it is not certified for } export type PersonEnrichResponse = | { outcome: 'matched'; match: MatchEvidence; claims: Claim[]; provider_person_id?: string; cost_units: number } | { outcome: 'no_match'; cost_units: number } // 0 for providers that do not charge a miss | { outcome: 'ambiguous'; candidates: Array<{ match: MatchEvidence; summary: string }>; cost_units: number }; export interface MatchEvidence { likelihood: number | null; // the provider's own score normalised to 0..10, null when the provider has none (FullContact) matched_on: Array<'email' | 'phone' | 'name_location' | 'profile' | 'provider_id'>; provider_confidence_raw?: string; // as the provider reported it, for the audit trail only } export interface Claim { field_key: FieldKey; // the vocabulary of §2, validated by the schema's enum value: unknown; // typed per field key in the schema's oneOf observed_at?: string; // RFC 3339, when the provider last verified it, if the provider says confidence?: number; // 0..1 per claim where the provider scores fields (PDL does not; Clay does) source_kind: 'provider' | 'public_web' | 'registry' | 'self_published'; source_ref?: string; // a URL or dataset version the adapter can cite } ``` Each adapter declares features for capability discovery and policy eligibility. For example, a tier-3 policy may choose only a connection whose adapter declares `input.personal_email`. | Feature | Meaning | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `input.email`, `input.personal_email`, `input.phone`, `input.name_location`, `input.profile_url`, `input.provider_id` | identifiers the provider accepts; `input.personal_email` means the provider documents consumer e-mail matching (PDL, FullContact; not Cognism, Hunter, Apollo in GDPR regions, **Market**: [w3 §11.A](#25-w3-evidence-enrichment-provider-landscape-for-an-association-management-system)) | | `claims.identity`, `claims.professional`, `claims.social`, `claims.photo`, `claims.location`, `claims.contact_channels`, `claims.education`, `claims.employment_history` | field groups the provider can return | | `likelihood` | the response carries a usable match score | | `miss_is_free` | a `no_match` costs nothing (PDL; **Verified**: [w5 §1](#27-w5-supplementary-provider-checks-read-directly-2026-10-22-11-00-to-11-30-et)) | | `batch` | the adapter accepts the async list form (`PersonEnrichBatchRequest`, up to 1,000 subjects per execution) | | `tier_3_certified` | the adapter has passed the privacy conformance for personal-identity data ([09 §2](#10-09-security-privacy-and-licensing)) | The resolver evaluates a closed vocabulary of manifest constraints. It already supports `countries`; the proposal adds `licence_class` (§6). #### 3.2 `enrichment.organization.enrich@1` Organization enrichment follows the same response model, using domains, names or registry identifiers to find an organization. Staff would receive claims in the shared vocabulary rather than a provider-specific company record. ```ts export interface OrganizationEnrichRequest { subject_ref: string; identifiers: { domain?: string; alternate_domains?: string[]; name?: string; location?: { city?: string; region?: string; country?: string }; identifiers?: { lei?: string; duns?: string; wikidata?: string; linkedin_url?: string }; provider_org_id?: string; }; requested_groups: Array<'firmographics' | 'brand' | 'identifiers' | 'location' | 'social'>; } export type OrganizationEnrichResponse = | { outcome: 'matched'; match: MatchEvidence; claims: Claim[]; provider_org_id?: string; cost_units: number } | { outcome: 'no_match'; cost_units: number } | { outcome: 'ambiguous'; candidates: Array<{ match: MatchEvidence; summary: string }>; cost_units: number }; ``` Features are `input.domain`, `input.name`, `input.registry_id`, `claims.firmographics`, `claims.brand`, `claims.identifiers`, `claims.social`, `claims.location` and `logo_storable`. `logo_storable` means the licence allows Rhythm to copy the logo file. Logo.dev permits this on every plan; Brandfetch permits it only under a Custom agreement (**Verified**: [w5 §4](#27-w5-supplementary-provider-checks-read-directly-2026-10-22-11-00-to-11-30-et), **Market**: [w3 §11.B](#25-w3-evidence-enrichment-provider-landscape-for-an-association-management-system)). #### 3.3 `enrichment.email.verify@1` E-mail verification returns a delivery status, an optional reason and the cost of the call. It also identifies whether the address uses a free e-mail provider. ```ts export interface EmailVerifyRequest { subject_ref: string; email: string; } export interface EmailVerifyResponse { status: 'deliverable' | 'undeliverable' | 'risky' | 'unknown'; reason?: 'invalid_syntax' | 'invalid_domain' | 'mailbox_not_found' | 'catch_all' | 'disposable' | 'role_account' | 'provider_error'; free_provider: boolean; // gmail, outlook, icloud… cost_units: number; } ``` #### 3.4 `enrichment.person.watch@1` Employment monitoring is not in wave one. The proposed capability would let staff receive new employment claims when a provider reports a change, rather than relying only on another enrichment request. The asynchronous subscription is `{ subject_ref, provider_person_id, watch: ['employment'] }` → `accepted`. The adapter then emits `enrichment.person.changed` with `{ subject_ref, change: 'employment', claims: Claim[], observed_at }` whenever its feed reports a change. PDL supplies a monthly changelog diff that the adapter polls; Crustdata supplies a per-person watcher webhook (**Market**: [w3 §11.C](#25-w3-evidence-enrichment-provider-landscape-for-an-association-management-system)). `enrichment.person.watch.status@1` is the lookup for an uncertain subscription. ### 4. The adapter's duties Adapters keep provider-specific behavior out of the rest of Rhythm. An adapter is an ordinary SDK adapter (**Verified**: [r1 §8](#18-r1-the-integration-platform-as-built-evidence-for-a-data-enrichment-design)), with four extra duties that the proposed conformance suite would check: 1. **Map to the vocabulary, drop the rest.** Return only claims whose field key is in the vocabulary and whose group was requested. Never return an unrequested field. This limits data collection at the connector (**Market**: [w4 §14.9](#26-w4-evidence-report-privacy-law-and-data-licensing-constraints-on-rhythm-2-0-data-enrichment)). 2. **Normalise the score.** Map the provider's confidence to `likelihood` 0 to 10 and document the mapping in the manifest. A provider without a score returns `null`, never an invented number. 3. **Report cost honestly.** `cost_units` is the provider's own billing unit for the call: 1 PDL credit, 0 for a free miss, 1 Logo.dev API credit or 0 for a CDN hit. 4. **Use free endpoints for health.** `connection.verify` and `health` must not consume billable units. The health job runs every 15 minutes per connection, so a paid health check would create recurring charges unrelated to enrichment (**Verified**: [r1 §6](#18-r1-the-integration-platform-as-built-evidence-for-a-data-enrichment-design)). The manifest's `auth_methods` determine the commercial mode: `api_key` for a customer-provided key, or `platform_account` with an `account_qualifier` for a Rhythm-managed account ([12](#13-12-commercial-model-and-economics)). ### 5. Orchestration: one plan, not a graph The recommendation is a short, ordered plan: try the baseline sources, then the primary provider, then a fallback if needed. The association sets the provider order, the stopping rule and the budget. This offers more control than a single provider lookup without turning enrichment into a workflow-building product. The Integration Platform's resolver selects exactly one connection per scope and merges nothing (**Verified**: [r1 §7](#18-r1-the-integration-platform-as-built-evidence-for-a-data-enrichment-design)). Enrichment needs more than that, but less than Clay: an ordered provider list per operation, a stop rule and a budget check. The proposed module would compute a **plan** from the association's policy and execute it in order through explicit-connection invocations. Each invocation names the connection to use rather than asking the resolver to choose one. ```ts export interface EnrichmentPlan { operation: 'person.enrich' | 'organization.enrich' | 'email.verify'; tier: 1 | 2 | 3; steps: Array<{ connection_id: string; // an eligible connection the policy named capability: CapabilityKey; major: number; requested_groups: FieldGroup[]; // the groups still unsatisfied when this step runs, intersected with the provider's claims.* features minimum_likelihood: number; estimated_units: number; // from the provider's price list in the policy (for the budget check) }>; stop_when: 'first_match' | 'required_groups_met'; required_groups: FieldGroup[]; // for 'required_groups_met' budget: { reserve_units: number; allowance_period_id: string }; } ``` The proposed plan rules are: - **Baseline first.** Free sources, Gravatar `d=404` probe, Logo.dev and GLEIF, run before any billable step. A billable provider is asked only for what the free sources could not answer. - **At most three steps.** This allows “free, then primary, then fallback.” More steps would move toward a graph, a product for sales operations teams rather than association administrators ([14 §4](#15-14-alternatives-and-tradeoffs)). - **No merging of disagreeing providers.** If step 2 claims one job title and step 3 claims another, both claims are recorded. The suggestion engine prefers the higher-likelihood match and shows the other as an alternative. The module never averages, votes or invents an answer. - **Cost awareness is a gate, not a search.** Refuse the plan before step 1 if its total `estimated_units` exceeds the period's remaining allowance. Skip a step if the provider's circuit is open, meaning calls are temporarily blocked because of failures. Wave one has no “cheapest provider that can answer field X” routing; the policy order expresses the association's cost preference. - **Field-specific routing is a later addition.** It would use per-group provider order in the policy. The plan already carries `requested_groups` per step, so this would need no contract change. - **AI research is a step kind, not a provider.** In ENR-M6, a step may be `{ kind: 'research', operation_key: 'enrichment.research' }`. The LLM module would execute it with an allow-listed search tool. Its claims carry `source_kind: 'public_web'` and `source_ref` citations, and are never auto-applied ([09 §5](#10-09-security-privacy-and-licensing)). ### 6. Licence class on the contract A provider's licence determines where an association may use a claim: staff screens, member-facing pages, exports or AI. The proposed contract makes those restrictions explicit so every surface can enforce the same policy. Each adapter manifest would declare a `licence_class` constraint per capability from a closed list. The connection would record the terms version accepted at connect time. The examples below are readings of the available terms, not confirmed contractual permissions. | Licence class | Meaning for Rhythm | Examples (as read; to be confirmed in contract) | | --------------------------- | ------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | | `open` | store, display to members, export, use in AI; no expiry | GLEIF, Wikidata (CC0), Logo.dev ("store anything returned") | | `internal_staff` | store; show to staff; no member display; no export; no AI training | PDL self-serve (reseller reproduction), Apollo, ZoomInfo (**Market**: [w3 §11.B](#25-w3-evidence-enrichment-provider-landscape-for-an-association-management-system)) | | `customer_app` | display to the tenant's own end users permitted under a named application agreement; no re-export | FullContact Exhibit A model (**Verified**: [w5 §2](#27-w5-supplementary-provider-checks-read-directly-2026-10-22-11-00-to-11-30-et)) | | `cache_ttl:` modifier | the value must be re-fetched or dropped after the period | Brandfetch 30 days; FullContact delete within 30 days of termination | | `no_ai` modifier | the value may not enter any LLM prompt or model | FullContact §2(d)(1)(vi) | | `self_published` | the subject published it for display (Gravatar); display allowed, refresh at reasonable intervals | Gravatar guidelines (**Verified**: [w5 §5](#27-w5-supplementary-provider-checks-read-directly-2026-10-22-11-00-to-11-30-et)) | The class is not part of the adapter's `Claim`: the module stamps it on each claim at receipt from the connection, and it would travel onto every claim ([08 §2](#9-08-data-model-provenance-and-policies)). It would feed the single policy function that every surface consults before using the value ([09 §4](#10-09-security-privacy-and-licensing)). ### 7. Capability discovery, end to end Under this proposal, adding a provider would add an adapter and a policy choice, not require changes throughout the product. The sequence would be: 1. HQ publishes an adapter. Its manifest declares capabilities, features and licence class. 2. The association installs it and connects an account. Verification records the features the account actually has as `verified_features` (**Verified**: [r1 §2.3](#18-r1-the-integration-platform-as-built-evidence-for-a-data-enrichment-design)). 3. The enrichment policy screen asks the discovery route for connections offering `enrichment.person.enrich@1`. It shows each connection's verified features, tier certification and licence class, and lets the administrator order the connections per operation. 4. At run time, the module builds the plan from the policy and the subject's known identifiers. It skips a provider without `input.personal_email` when the subject has only a personal e-mail. It skips a provider without `claims.photo` when only the photo group remains wanted. 5. Nothing in CRM, the console or the workflow action changes when a fourth provider arrives. ## 9. 08. Data model, provenance and policies ## 08. Data model, provenance and policies **Recommendation: keep enrichment separate from CRM records, and make every proposed field change traceable to its source.** The model separates what a provider said from what Rhythm proposes and what ultimately becomes a record's current value. Policy controls which fields can be enriched; suppression and allowance checks control whether a provider call can happen at all. Status: proposed 2026-10-22. As ruled (ENR-D1, Andrew 2026-10-22 about 11:50 ET), the schema is `crm` and the tables are `crm.enrichment_*` in services-crm; the DDL below keeps the `platform.` prefix as written and is read with that substitution. Nothing in this enrichment design is decided or built. The package's twelve decisions, ENR-D1 to ENR-D12, are presented in document 15 for Andrew to approve or overturn. This document explains the proposed model; it asks for no separate decision. Labels are as defined in the [README](#1-rhythm-2-0-data-enrichment-architecture). All DDL, the SQL that defines the tables, is illustrative. It follows Platform conventions: schema `platform`, snake_case, `tenant_id` first in every composite key, forced row-level security on workspace tables, UUIDv7 for records, immutable lowercase string ids for configuration, `sys_` for system-filled columns, and the `system-columns` set from `@rhythmsoftware/macroservice-records` (**Verified**: [r3 §14](#20-r3-platform-core-evidence-for-the-data-enrichment-architecture), [r4 §I-4](#21-r4-data-operations-data-job-engine-and-the-query-layer-evidence-for-rhythm-2-0-data-enrichment)). Forced row-level security ensures that database access remains scoped to the workspace. Index decisions follow `docs/indexing.md` and are listed in §8. ### 1. The shape in one picture The model keeps provider evidence, review decisions and current field sources separate. This lets staff see both what was proposed and why a value is on the record, without adding enrichment columns to CRM records. ```mermaid erDiagram enrichment_policies ||--o{ enrichment_provider_settings : "orders providers" enrichment_policies ||--o{ enrichment_field_rules : "per field group" enrichment_subjects ||--o{ enrichment_requests : "has" enrichment_requests ||--o{ enrichment_claims : "produced" enrichment_claims ||--o{ enrichment_suggestions : "backs" enrichment_subjects ||--o{ enrichment_suggestions : "about" enrichment_subjects ||--o{ enrichment_field_provenance : "current source per field" enrichment_suggestions ||--o| enrichment_field_provenance : "applied becomes" enrichment_runs ||--o{ enrichment_requests : "fans out" enrichment_allowance_periods ||--o{ enrichment_usage_daily : "rolls up" enrichment_suppressions }o--|| enrichment_subjects : "blocks" ``` There are seven core concepts: - **Policy:** what a workspace allows. - **Subject:** one record's enrichment state. - **Request:** one attempt to enrich one subject under one plan. - **Claim:** what a provider said. - **Suggestion:** what Rhythm proposes doing with a claim. - **Provenance:** where each field's current value came from. - **Allowance:** what the workspace may spend. Two supporting concepts handle scale and member choice: a **run** distributes bulk or scheduled work across subjects, and a **suppression** prevents enrichment of a person. ### 2. Tables #### 2.1 Policy (configuration; tenant-administered; string ids) Policy gives association administrators control over when enrichment runs, which providers and field groups it uses, what it may spend, and whether empty fields can be filled automatically. Existing values can be suggested for review, never overwritten automatically (P3). ```sql CREATE TABLE platform.enrichment_policies ( tenant_id text NOT NULL, id text NOT NULL DEFAULT 'default', -- one row per tenant in wave one status text NOT NULL CHECK (status IN ('off','on')), tiers_enabled smallint[] NOT NULL DEFAULT '{}', -- subset of {1,2,3}; see 09 §2 triggers jsonb NOT NULL, -- { on_create: bool, on_import: bool, on_portal_update: bool, on_registration: bool, refresh: { cadence: 'off'|'annual'|'semiannual', max_per_day: int } } budget jsonb NOT NULL, -- { monthly_units: int, hard_stop: bool, notify_at: [80,100] } retention jsonb NOT NULL, -- { claims_days: 730, suggestions_days: 90 } attestations jsonb NOT NULL, -- { lawful_basis: 'legitimate_interest'|'consent', lia_ref, notice_date, dpia_ref, tenant_type: 'general'|'special_category' } region_rules jsonb NOT NULL, -- { eu_uk_ca_au: { max_tier: 2, require_notice_date: true } } sys_version integer NOT NULL DEFAULT 1, -- system columns: sys_created_at, sys_created_by_id, sys_updated_at, sys_updated_by_id PRIMARY KEY (tenant_id, id) ); CREATE TABLE platform.enrichment_provider_settings ( tenant_id text NOT NULL, id text NOT NULL, -- '_' e.g. 'person_enrich_1' policy_id text NOT NULL, operation text NOT NULL CHECK (operation IN ('person.enrich','organization.enrich','email.verify','person.watch')), position smallint NOT NULL, -- plan order; 1..3 connection_id text NOT NULL, -- platform.integration_connections.id (composite FK with tenant) requested_groups text[] NOT NULL, minimum_likelihood smallint NOT NULL DEFAULT 6, unit_price_credits integer NOT NULL, -- credits per cost_unit, from the provider's price list (HQ-set for Rhythm-managed, tenant-set for BYOK) licence_terms_version text NOT NULL, -- hash of the terms accepted on this connection is_active boolean NOT NULL DEFAULT true, sys_version integer NOT NULL DEFAULT 1, PRIMARY KEY (tenant_id, id), UNIQUE (tenant_id, operation, position) ); CREATE TABLE platform.enrichment_field_rules ( tenant_id text NOT NULL, id text NOT NULL, -- '_' e.g. 'crm_contacts_professional' policy_id text NOT NULL, object_id text NOT NULL, -- 'crm.contacts' | 'crm.organizations' field_group text NOT NULL, -- 07 §2 empty_value text NOT NULL CHECK (empty_value IN ('auto_fill','suggest','ignore')), existing_value text NOT NULL CHECK (existing_value IN ('suggest','ignore')), -- never 'overwrite' (P3) protected_field_keys text[] NOT NULL DEFAULT '{}', -- never touched, never suggested custom_field_map jsonb NOT NULL DEFAULT '{}', -- { "organization.industry": "cf_industry" }: provider field key -> custom field definition id reviewer_responsibility_id text, -- 'crm.data_quality_review' by default PRIMARY KEY (tenant_id, id) ); ``` The proposed workspace defaults are seeded and reconciled on deploy, following the existing pattern (**Verified**: [r3 §3](#20-r3-platform-core-evidence-for-the-data-enrichment-architecture)): - Policy starts `off`; only tier 1 is enabled when it is switched on. - Triggers start off except `on_create` for organizations. - `empty_value = 'suggest'` for every person group, and `'auto_fill'` for organization `brand` and `identifiers`. - `existing_value = 'suggest'` everywhere. - Budget is 0 until set. ENR-D3 asks whether these are the right defaults. #### 2.2 Subject (one row per enriched record; UUIDv7-keyed records) The subject is the module's summary of a record: its enrichment status, when it was last matched, when it should be refreshed, and how many suggestions await review. The identifier digest detects whether there is anything new to ask a provider. ```sql CREATE TABLE platform.enrichment_subjects ( tenant_id text NOT NULL, id uuid NOT NULL, -- UUIDv7 object_id text NOT NULL, -- 'crm.contacts' record_id uuid NOT NULL, status text NOT NULL CHECK (status IN ('never','scheduled','enriched','no_match','suppressed','failed')), identifier_digest text NOT NULL, -- sha256 of the normalised identifiers last sent, to detect "nothing new to ask" last_requested_at timestamptz, last_matched_at timestamptz, next_refresh_at timestamptz, provider_ids jsonb NOT NULL DEFAULT '{}', -- { "": "" } for refreshes and watches open_suggestions integer NOT NULL DEFAULT 0, -- denormalised for the 360 badge sys_version integer NOT NULL DEFAULT 1, PRIMARY KEY (tenant_id, id), UNIQUE (tenant_id, object_id, record_id) ); ``` #### 2.3 Request (one attempt; monthly partitions like `integration_executions`) A request records what actually ran: the plan, match decision, provider executions, reserved and settled units, and outcome. Monthly partitions keep the growing request history manageable. ```sql CREATE TABLE platform.enrichment_requests ( tenant_id text NOT NULL, id uuid NOT NULL, started_at timestamptz NOT NULL, -- partition key subject_id uuid NOT NULL, run_id uuid, -- null for a single request trigger text NOT NULL CHECK (trigger IN ('manual','on_create','on_import','on_portal_update','on_registration','bulk','scheduled','workflow','watch')), tier smallint NOT NULL, plan jsonb NOT NULL, -- the EnrichmentPlan as executed (07 §5) status text NOT NULL CHECK (status IN ('queued','running','completed','no_match','skipped','failed','budget_refused','suppressed')), match jsonb, -- { accepted: bool, likelihood, matched_on[], corroboration: { name, employer, location }, reason } execution_ids text[] NOT NULL DEFAULT '{}', -- integration_executions.id per step units_reserved integer NOT NULL DEFAULT 0, units_settled integer NOT NULL DEFAULT 0, requested_by_id text, -- a person for manual/bulk; null for system triggers correlation_id text NOT NULL, completed_at timestamptz, error jsonb, -- { class, code, message_safe } from the gateway's taxonomy PRIMARY KEY (tenant_id, id, started_at) ) PARTITION BY RANGE (started_at); ``` #### 2.4 Claim (append-only; the provider's words) A claim preserves what a provider said, not a decision to trust or apply it. Each claim carries its source, confidence where available, licence class and expiry. Claims cannot be edited by the application; retention removes them when they expire. ```sql CREATE TABLE platform.enrichment_claims ( tenant_id text NOT NULL, id uuid NOT NULL, subject_id uuid NOT NULL, request_id uuid NOT NULL, connection_id text NOT NULL, execution_id uuid NOT NULL, field_key text NOT NULL, -- 07 §2 vocabulary value jsonb NOT NULL, -- typed per field key value_digest text NOT NULL, -- sha256 of canonical value, for "same claim again" likelihood smallint, -- the request's match likelihood, copied for queries confidence numeric(4,3), -- per-claim, where the provider scores fields source_kind text NOT NULL CHECK (source_kind IN ('provider','public_web','registry','self_published','ai_research')), source_ref text, observed_at timestamptz, received_at timestamptz NOT NULL, licence_class text NOT NULL, -- 07 §6, copied from the connection at receipt expires_at timestamptz, -- from the licence's cache TTL or the policy's retention, whichever is sooner PRIMARY KEY (tenant_id, id) ); -- No UPDATE grant on this table for the application role; expiry is a DELETE by the retention job. ``` #### 2.5 Suggestion (what Rhythm proposes; the review queue) A suggestion turns provider evidence into a proposed CRM change. It records the value staff would apply, the claim supporting it, any disagreeing claims, and the record's state when the proposal was made. Those checks prevent an old suggestion from replacing a newer edit. ```sql CREATE TABLE platform.enrichment_suggestions ( tenant_id text NOT NULL, id uuid NOT NULL, subject_id uuid NOT NULL, object_id text NOT NULL, record_id uuid NOT NULL, field_key text NOT NULL, kind text NOT NULL CHECK (kind IN ('fill','change','add_collection_item','employment_change','photo','duplicate_hint')), claim_id uuid NOT NULL, alternative_claim_ids uuid[] NOT NULL DEFAULT '{}', -- other providers' disagreeing claims proposed_value jsonb NOT NULL, -- normalised to the target field's shape (e.g. an organization_id for an employer after matching) current_value_digest text, -- what the record held when proposed; apply refuses if it changed record_version_at_proposal integer NOT NULL, -- sys_version of the record then (record write contracts W1/W4) state text NOT NULL CHECK (state IN ('pending','accepted','rejected','applied','apply_failed','expired','superseded')), auto boolean NOT NULL DEFAULT false, -- true when policy applied it without a person decided_by_id text, decided_at timestamptz, decision_reason text, applied_job_id uuid, -- the CRM data job (ENR-M1) or service-function correlation (ENR-M2+) applied_revision_ref text, -- 'crm.record_revisions:' when CRM returns it expires_at timestamptz NOT NULL, sys_version integer NOT NULL DEFAULT 1, PRIMARY KEY (tenant_id, id) ); -- One pending suggestion per (subject, field_key); a newer claim supersedes the older pending one. CREATE UNIQUE INDEX enrichment_suggestions_pending_uidx ON platform.enrichment_suggestions (tenant_id, subject_id, field_key) WHERE state = 'pending'; ``` An employment change is a suggestion of kind `employment_change`. Its `proposed_value` is `{ organization_id | organization_candidate, job_title, started_on, previous: 'end' | 'keep' }`, so review includes what happens to the previous employment ([06 §6](#7-06-core-workflows-and-use-cases)). #### 2.6 Provenance (the current source of each field's value) Provenance answers a staff member's question: “Where did this value come from?” It describes the current value, rather than every past claim. A digest mismatch makes the row stale; the source becomes unknown until it can be derived again. Once a member or staff sets a field, it is locked against future automatic filling (P3). ```sql CREATE TABLE platform.enrichment_field_provenance ( tenant_id text NOT NULL, subject_id uuid NOT NULL, field_key text NOT NULL, source_kind text NOT NULL CHECK (source_kind IN ('member','staff','import','provider','ai_research','system','unknown')), claim_id uuid, -- when source_kind is provider or ai_research suggestion_id uuid, connection_id text, value_digest text NOT NULL, -- the value this row describes; if the record's value digest differs, the row is stale and the source is 'unknown' until re-derived set_at timestamptz NOT NULL, verified_at timestamptz, -- last time a source confirmed the current value (a provider re-claiming it, a member saving it) locked boolean NOT NULL DEFAULT false, -- true once a member or staff sets the field: never auto-filled again (P3) licence_class text, expires_at timestamptz, PRIMARY KEY (tenant_id, subject_id, field_key) ); ``` CRM still owns writes to contacts and organizations. To keep provenance accurate, the module would subscribe to CRM change events. Contacts and organizations publish field-level revisions with `sys_source` (**Verified**: [r2 §7](#19-evidence-r2-crm-rhythm-2-0-crm-as-the-target-of-data-enrichment)). The proposed mapping is: - `portal` sets `source_kind = 'member', locked = true`. - `console` sets `source_kind = 'staff', locked = true`. - `job` with an import job sets `import`. - An operation named `enrichment.apply` (ENR-M2), or a job id matching a suggestion's `applied_job_id` (ENR-M1), sets `provider` with the claim. This uses the same subscription mechanism that Quick Search and duplicate detection use today. #### 2.7 Run (bulk and scheduled fan-out; a Durable Job root) A run tracks a bulk or scheduled selection and distributes the provider work into individual requests. Fetching evidence and applying accepted changes use different execution paths, so provider calls do not run inside a batch transaction. ```sql CREATE TABLE platform.enrichment_runs ( tenant_id text NOT NULL, id uuid NOT NULL, kind text NOT NULL CHECK (kind IN ('bulk','scheduled_refresh','import_followup','watch_sync')), object_id text NOT NULL, selection jsonb NOT NULL, -- { record_ids: [...] } (≤ 10,000, the console's selection cap) or { query_ref, frozen_at } when PLAT-M21 frozen selections exist operation text NOT NULL, estimate_units integer NOT NULL, reserved_units integer NOT NULL, counts jsonb NOT NULL DEFAULT '{}', -- { queued, completed, matched, no_match, suppressed, budget_refused, failed } status text NOT NULL, -- the Durable Job states job_id uuid NOT NULL, -- lib/jobs root requested_by_id text, started_at timestamptz, completed_at timestamptz, PRIMARY KEY (tenant_id, id) ); ``` The fetch side would be a Durable Job that creates one queued-work row per subject. It follows the queued-work standard: a queue table, dispatcher, SQS and consumer with reserved concurrency (**Verified**: [r3 §7](#20-r3-platform-core-evidence-for-the-data-enrichment-architecture)). The **apply** side of a bulk accept would be a CRM data job ([06 §3](#7-06-core-workflows-and-use-cases)). Enrichment runs are therefore not data jobs. This addresses the “Platform has no bulk lane” gap (**Verified**: [r4 §G3](#21-r4-data-operations-data-job-engine-and-the-query-layer-evidence-for-rhythm-2-0-data-enrichment)) without stretching the data-job engine beyond its purpose. The engine applies values with receipts and rollback; it must not call providers inside a batch transaction. #### 2.8 Suppression (survives the record) A member's request not to be enriched must still hold after their contact is erased and later re-imported. Suppression stores a workspace-specific hash of an identifier, rather than relying on the continued existence of the contact record. ```sql CREATE TABLE platform.enrichment_suppressions ( tenant_id text NOT NULL, id uuid NOT NULL, identifier_hash text NOT NULL, -- HMAC-SHA256(tenant salt, normalised e-mail | phone | record id) identifier_kind text NOT NULL CHECK (identifier_kind IN ('email','phone','record')), reason text NOT NULL CHECK (reason IN ('member_request','staff','provider_optout','erasure','policy')), source_ref text, -- the consent-ledger entry (CRM) or the provider notice created_at timestamptz NOT NULL, PRIMARY KEY (tenant_id, id), UNIQUE (tenant_id, identifier_kind, identifier_hash) ); ``` Suppression would be checked before every provider call and every cache read. CRM's consent ledger, when it exists, is the member-facing record of the choice (ruled D-M2, **Verified**: [r2 §13](#19-evidence-r2-crm-rhythm-2-0-crm-as-the-target-of-data-enrichment)). The suppression row is the enforcement copy and outlives erasure of the contact. #### 2.9 Provider cache (tenant-scoped, short-lived) The cache avoids paying twice for the same question within a month, for example after a re-import or a second manual click. It is not a shared store of provider data. ```sql CREATE TABLE platform.enrichment_provider_cache ( tenant_id text NOT NULL, connection_id text NOT NULL, operation text NOT NULL, identifier_digest text NOT NULL, response jsonb NOT NULL, -- the normalised PersonEnrichResponse, never the raw payload received_at timestamptz NOT NULL, expires_at timestamptz NOT NULL, -- min(licence cache TTL, policy retention, 30 days) PRIMARY KEY (tenant_id, connection_id, operation, identifier_digest) ); ``` The key includes the workspace and connection. Nothing is ever read across workspaces (P7). Only the normalised response is stored, never the raw payload, and expiry is bounded by the licence, policy and 30 days. Misses would also be cached, with a shorter expiry. Asking again next week is not expected to change the answer and may incur another charge from providers that charge for a miss. #### 2.10 Allowance and usage (the AI ledger pattern) The allowance ledger prevents concurrent requests from spending the same remaining budget. It reserves credits before a call, settles the actual cost afterwards, and records daily usage and threshold crossings. ```sql CREATE TABLE platform.enrichment_allowance_periods ( tenant_id text NOT NULL, period date NOT NULL, -- first day of the month recurring_credits integer NOT NULL, -- the tenant's monthly units (policy budget, or the licence's included credits) override_credits integer NOT NULL DEFAULT 0, -- HQ windowed top-ups reserved_credits integer NOT NULL DEFAULT 0, settled_credits integer NOT NULL DEFAULT 0, in_flight_count integer NOT NULL DEFAULT 0, PRIMARY KEY (tenant_id, period) ); CREATE TABLE platform.enrichment_usage_daily ( tenant_id text NOT NULL, day date NOT NULL, connection_id text NOT NULL, operation text NOT NULL, trigger text NOT NULL, requests integer NOT NULL, matched integer NOT NULL, units integer NOT NULL, credits integer NOT NULL, PRIMARY KEY (tenant_id, day, connection_id, operation, trigger) ); CREATE TABLE platform.enrichment_usage_threshold_crossings ( tenant_id text NOT NULL, period date NOT NULL, threshold smallint NOT NULL, crossed_at timestamptz NOT NULL, PRIMARY KEY (tenant_id, period, threshold) ); ``` Authorisation would use one conditional `UPDATE` against `recurring_credits + override_credits`: reserve before the call, settle from `cost_units × unit_price_credits` afterwards, and release on failure. This is the pattern used by `llm_allowance_periods` (**Verified**: [r3 §4.3](#20-r3-platform-core-evidence-for-the-data-enrichment-architecture)). When licensing v2's resource registry exists, it would feed `recurring_credits` instead of the policy. Nothing else in the ledger would change. ### 3. Policy evaluation: from claim to suggestion The proposed evaluation protects member and staff edits, limits stored data to what the association requested, and sends uncertain changes for review. For each claim in an accepted match, the checks run in this order: 1. **Suppressed?** Check the subject's identifiers against `enrichment_suppressions` before the provider call. A claim for a suppressed subject never reaches evaluation. 2. **Eligible field?** The field key's group must be in the policy's `requested_groups` for that step and must not be in `protected_field_keys`. Discard, rather than store, a claim for a group the workspace did not ask for. This minimises collected data. 3. **Match accepted?** The request's `match.accepted` must be true. Acceptance requires likelihood ≥ the step's minimum **and** at least one corroboration: name agreement with the record, employer or domain agreement, or location agreement. These checks use duplicate detection normalisation (**Verified**: [r2 §6.1](#19-evidence-r2-crm-rhythm-2-0-crm-as-the-target-of-data-enrichment)). An e-mail-only match without corroboration is accepted only for tier 1 organization data and `person.email.status`. Otherwise, store the claims with the request marked `match.accepted = false` and create no suggestion. 4. **Current value?** Read the current value and provenance. Read the record through its owner's read API, not its tables, except for `crm.*` datasets that Platform may read (**Verified**: [r4 §I-4](#21-r4-data-operations-data-job-engine-and-the-query-layer-evidence-for-rhythm-2-0-data-enrichment)). 5. **Decide the kind.** - Empty value, rule `auto_fill`, field not `locked`: create an `accepted` suggestion with `auto = true` and queue the apply. - Empty value, rule `suggest`: create `pending`, kind `fill`. - Different value, rule `suggest`: create `pending`, kind `change`, with `current_value_digest`. - Same value: create no suggestion; update `verified_at` in provenance. - Employer claim: always create `pending`, kind `employment_change`, after organization matching ([06 §6](#7-06-core-workflows-and-use-cases)). - Photo claim: always create `pending`, kind `photo`, with a proxied preview URL. The photo is never persisted until accepted ([09 §4](#10-09-security-privacy-and-licensing)). 6. **Supersede.** A new pending suggestion for the same field replaces the older pending one, which becomes `superseded`. 7. **Notify.** Publish an Attention item for the policy's reviewer responsibility. Batch pending suggestions per subject, for example “3 suggestions for Jane Doe”, using Attention's own transaction rule (**Verified**: [r3 §10](#20-r3-platform-core-evidence-for-the-data-enrichment-architecture)). Before applying an `accepted` suggestion, re-read the record's version. If the version has moved since proposal and the field's value has changed, refuse the apply with `apply_failed: record_changed` and return the suggestion to `pending` with a note. This follows record write contracts W4 (**Verified**: [r2 §8.3](#19-evidence-r2-crm-rhythm-2-0-crm-as-the-target-of-data-enrichment)) and prevents a reviewed proposal from replacing a subsequent edit. ### 4. Keeping the semantic layer simple Staff need to query enrichment status and review suggestions, but that should not expose provider evidence whose licence varies by row. The proposed query model therefore adds two datasets and leaves CRM datasets unchanged. - Contacts and organizations datasets are **not** changed. No enrichment column is added to `crm.contacts`. - Platform would register two new datasets: `platform.enrichment_subjects` for status, last matched, next refresh and open suggestions; and `platform.enrichment_suggestions`. Each has its own `.query` key and joins to contacts through `record_id` as a reference field. CRM-ACC-0064's rule applies: both `.query` and the contact `.read` key are required (**Decided**: Andrew 2026-10-21). - `enrichment_claims` and `enrichment_provider_cache` are **never** registered as datasets. Their licence classes vary per row, and the query layer has no field-level security (decision 11 of the semantic layer, **Verified**: [r4 §5.4](#21-r4-data-operations-data-job-engine-and-the-query-layer-evidence-for-rhythm-2-0-data-enrichment)). Claims are shown only on the record's enrichment panel, through the policy function. - The contact 360 badge, “2 suggestions”, reads `enrichment_subjects.open_suggestions` by primary key: one row. - The new registered tables attach `platform.attach_query_description_stamp`, as every registered table must. ### 5. Merge, lifecycle, export and reset participation Enrichment must participate in record and workspace lifecycle operations. Otherwise, it could block contact merges, leave data behind after erasure, or export claims that a provider's licence does not permit. - **Merge.** Every table keyed by `record_id` or `subject_id` declares a merge policy. Subjects collapse into the survivor's subject; open suggestions are re-pointed, and the merged-away subject remains as an alias row for history. Provenance rows move when the survivor's field was empty; otherwise, the merged-away row is dropped. Claims and requests are re-pointed. Platform would implement the participant functions `platform.merge_plan_contact` / `merge_apply_contact` under the existing protocol (**Verified**: [r4 §9](#21-r4-data-operations-data-job-engine-and-the-query-layer-evidence-for-rhythm-2-0-data-enrichment)). Without them, CRM-ADR-12 would block merges of enriched contacts. - **Record lifecycle.** Archive leaves enrichment rows in place. Hard delete and erasure delete the record's subjects, requests, claims, suggestions, provenance and cache. They also write a suppression row with reason `erasure`, so a re-import does not re-enrich the person, and an audit event without values ([09 §6](#10-09-security-privacy-and-licensing)). - **Tenant export.** Subjects and suggestions export under a new class `third_party_licensed`. Claims export only when the licence class allows it (`open`, `customer_app`). Otherwise, they are withheld with reason `licence`, using the `withheldFields` mechanism (**Verified**: [r4 §12](#21-r4-data-operations-data-job-engine-and-the-query-layer-evidence-for-rhythm-2-0-data-enrichment)). - **Tenant reset and purge.** The module would register a lifecycle participant that truncates its tables for the workspace. - **Snapshots.** None in wave one. ### 6. Retention Provider evidence is retained only as long as the licence and policy permit. Provenance lasts with the record, while suppression lasts with the workspace so that erasure does not undo an opt-out. Platform's audit retention decision remains open. | Table | Kept | Then | | ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | | `enrichment_requests` | 13 months online (monthly partitions, like executions) | archived as Parquet, values never present | | `enrichment_claims` | until `expires_at` (licence TTL, policy `claims_days`, default 730, or provider deletion notice) | deleted; the provenance row keeps `source_kind` and `connection_id` but loses the claim link | | `enrichment_suggestions` | pending until `expires_at` (policy `suggestions_days`, default 90); decided ones for the life of the tenant, keeping `proposed_value` after the claim they rested on expires | pending ones move to `expired`, then are deleted with their claims | | `enrichment_provider_cache` | ≤ 30 days | deleted | | `enrichment_field_provenance` | life of the record | deleted with the record | | `enrichment_suppressions` | life of the tenant | never deleted by retention | | `enrichment_usage_daily` | life of the tenant | | | audit events (`enrichment.*`) | per Platform's audit retention decision (open, **Verified**: [r3 §5](#20-r3-platform-core-evidence-for-the-data-enrichment-architecture)) | | ### 7. Illustrative domain code (code names camelCase, record properties snake_case, as the naming rule requires) The illustrative workflow reads policy, current record values and provenance, then passes them to a pure decision function. Persistence handles reads; the workflow does not build queries. ```ts export interface EnrichmentSubject { id: string; object_id: string; record_id: string; status: 'never' | 'scheduled' | 'enriched' | 'no_match' | 'suppressed' | 'failed'; identifier_digest: string; last_requested_at?: string; last_matched_at?: string; next_refresh_at?: string; provider_ids: Record; open_suggestions: number; sys_version: number; } export interface Suggestion { id: string; subject_id: string; object_id: string; record_id: string; field_key: FieldKey; kind: 'fill' | 'change' | 'add_collection_item' | 'employment_change' | 'photo' | 'duplicate_hint'; claim_id: string; alternative_claim_ids: string[]; proposed_value: unknown; current_value_digest?: string; record_version_at_proposal: number; state: 'pending' | 'accepted' | 'rejected' | 'applied' | 'apply_failed' | 'expired' | 'superseded'; auto: boolean; decided_by_id?: string; decided_at?: string; decision_reason?: string; applied_job_id?: string; applied_revision_ref?: string; expires_at: string; sys_version: number; } // services/policy/evaluate-claims.ts (workflow; calls persistence, never builds a query) export async function evaluateClaims(ctx: ExecutionContext, request: EnrichmentRequest, claims: Claim[]): Promise { const policy = await policies.readForTenant(ctx); if (!request.match.accepted) return []; const current = await recordReads.currentValues( ctx, request.subject, claims.map((c) => c.field_key), ); const provenance = await fieldProvenance.readMany( ctx, request.subject_id, claims.map((c) => c.field_key), ); return decideSuggestions({ policy, claims, current, provenance, now: ctx.now() }); // pure } ``` ### 8. Index decisions (docs/indexing.md) Indexes support record lookups, the review queue, refresh scheduling, history and retention sweeps. The choices below follow `docs/indexing.md`. | Table | Index | Why | | --------------------------- | ---------------------------------------------------------------------------------- | ------------------------------------------------- | | `enrichment_subjects` | `(tenant_id, object_id, record_id)` unique | the 360 badge and every trigger look up by record | | `enrichment_subjects` | `(tenant_id, next_refresh_at) WHERE status IN ('enriched','no_match')` partial | the scheduled refresh sweep (keyset) | | `enrichment_requests` | `(tenant_id, subject_id, started_at DESC)` per partition | the record's enrichment history | | `enrichment_requests` | `(tenant_id, run_id)` | run progress | | `enrichment_claims` | `(tenant_id, subject_id, field_key, received_at DESC)` | the panel's "what providers said" | | `enrichment_claims` | `(tenant_id, expires_at)` | retention sweep | | `enrichment_suggestions` | `(tenant_id, state, expires_at)`; `(tenant_id, record_id) WHERE state = 'pending'` | review queue, 360 panel | | `enrichment_suppressions` | `(tenant_id, identifier_kind, identifier_hash)` unique | the pre-call check | | `enrichment_provider_cache` | primary key only; `(tenant_id, expires_at)` | lookups by digest; sweep | All comparisons use plain equality or range on leakproof types, so they work under forced RLS, row-level security. Leakproof comparisons do not expose protected values through their evaluation. ## 10. 09. Security, privacy and licensing ## 09. Security, privacy and licensing Status: proposed 2026-10-22. Labels as in the [README](#1-rhythm-2-0-data-enrichment-architecture). Treat privacy, consent and provider licences as architectural requirements, not checks added after enrichment works. The proposed design checks whether a lookup is permitted **before sending any member data**, then controls where the returned data can appear, how long it can remain and whether AI may use it. Nothing proposed in this chapter is decided or built. Existing CRM capabilities and prior rulings are identified separately. This is engineering design informed by research ([w4](#26-w4-evidence-report-privacy-law-and-data-licensing-constraints-on-rhythm-2-0-data-enrichment), [w5](#27-w5-supplementary-provider-checks-read-directly-2026-10-22-11-00-to-11-30-et)), not legal advice. §7 separates shared controls from provider-specific obligations; §8 lists what counsel must confirm before launch. ### 1. The three facts that shape everything 1. **A lookup discloses the member.** Sending an e-mail address to a provider is itself a disclosure to an independent controller: an organization that decides how to use the data it receives. Several providers say they may use submitted data to improve their own data (HubSpot 6.5, ZoomInfo, FullContact §4(a); **Market**: [w4 §1](#26-w4-evidence-report-privacy-law-and-data-licensing-constraints-on-rhythm-2-0-data-enrichment), [w5 §2](#27-w5-supplementary-provider-checks-read-directly-2026-10-22-11-00-to-11-30-et)). The first question is therefore “may we ask?”, not just “may we store the answer?” 2. **The association is the controller; Rhythm is its processor; the provider is a third controller.** Rhythm processes data on the association's documented instructions (General Data Protection Regulation, GDPR Art. 28). If Rhythm reused lookups for its own purposes, or for another association, it would become a controller and very likely a data broker under California, Vermont, Texas and Oregon law (**Market**: [w4 §3, §11](#26-w4-evidence-report-privacy-law-and-data-licensing-constraints-on-rhythm-2-0-data-enrichment)). 3. **Provider licences differ on every point that matters.** They set different rules for member display, export, retention, AI use, caching and deletion windows. Those rules also change over time: ZoomInfo's retention wording changed between 2021 and 2025 (**Market**: [w4 §5](#26-w4-evidence-report-privacy-law-and-data-licensing-constraints-on-rhythm-2-0-data-enrichment)). FullContact bans any large language model (LLM) or machine-learning use of its data outright (**Verified**: [w5 §2](#27-w5-supplementary-provider-checks-read-directly-2026-10-22-11-00-to-11-30-et)). ### 2. Tiers: the risk gradient as a product switch The proposed product separates organization data from professional and personal data. Each tier controls both what staff may request and what the association must document before a lookup is allowed. | Tier | What is sent and received | Lawful-basis posture (association decides; Rhythm records) | Default | | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **1. Organization** | a domain or organization name out; firmographics, logo, identifiers back; no personal data | ordinary business information | on when enrichment is on | | **2. Professional, business context** | a work e-mail, name and employer out; job title, employer, business location, public professional profile URL back | legitimate interest with notice (Art. 13/14), easy objection; business-card data | off; on by the association's choice with attestation | | **3. Personal identity** | a personal e-mail, phone or name plus home location out; photo, social profiles, personal phone or address, demographics back | consent or a documented legitimate-interest assessment; not offered to EU, UK, Canadian or Australian members in wave one; never for religious, political or trade-union bodies without an Art. 9 basis | off; opt-in with attestation; adapters must be `tier_3_certified`; **ruled (ENR-D6, Andrew 2026-10-22 about 11:50 ET): offered from ENR-M1 behind attestation**, not deferred | | **X. Never** | face matching or any biometric processing, precise location, special-category inference, sources behind logins, any eligibility or credentialing decision driven by enrichment | prohibited in code and contract | n/a | For each tier, the policy records the association's attestations: its lawful basis, a reference to its legitimate-interest assessment or consent mechanism, the date its privacy notice named enrichment and its providers, and a data protection impact assessment (DPIA) reference where required (**Market**: [w4 §12.1](#26-w4-evidence-report-privacy-law-and-data-licensing-constraints-on-rhythm-2-0-data-enrichment)). Region rules refuse tier 2 for EU, UK, Canadian and Australian members until the notice date is recorded, and cap those members at tier 2. Decision ENR-D6 is whether tier 3 is offered at all in the first year. ### 3. Consent, objection and suppression A member's objection must stop enrichment before any data leaves Rhythm, including after the member's record has been erased and later re-imported. - **“Do not enrich me” is honoured before any call.** The suppression ledger ([08 §2.8](#9-08-data-model-provenance-and-policies)) stores a keyed hash, an HMAC, of the normalised e-mail, phone or record id, using a salt unique to each association. This lets the suppression survive erasure of the record and a later re-import of the same person. The worker checks it before every provider call and before every cache read. - **Three ways in.** Staff can record an objection on the 360 contact view (from ENR-M1); the member can do so in the portal (ENR-M4, on Andrew's components); or a provider's opt-out feed can supply it. PDL's changelog carries opt-outs; Hunter answers 451 for a claimed address (**Market**: [w3 §12](#25-w3-evidence-enrichment-provider-landscape-for-an-association-management-system)). - **The member-facing record is CRM's consent ledger** (ruled D-M2, not built; **Verified**: [r2 §13](#19-evidence-r2-crm-rhythm-2-0-crm-as-the-target-of-data-enrichment)). When that ledger exists, “do not enrich” is a consent-ledger entry whose change event writes the suppression row. Until then, the suppression row is the record, with `source_ref` available for the ledger id later. - **Objection deletes.** Recording a suppression deletes the subject's claims, suggestions and cache. It keeps provenance rows with `source_kind` only and writes an audit event without values. Provenance is the record of where a field value came from. ### 4. The licence policy function A provider's data must not become unrestricted simply because staff accept it into a contact record. Every claim carries a `licence_class`, copied from the connection at receipt ([07 §6](#8-07-provider-contracts-and-orchestration)). One function, `mayUse(claim | provenance, purpose)`, checks permission for every place the data can be used. Its default answer is **no**. | Purpose | `open` | `self_published` | `customer_app` | `internal_staff` | with `no_ai` | after `cache_ttl` | | ------------------------------------------------------------------------- | ------ | -------------------------------- | -------------------------------------------------- | -------------------------------------------------------------------------------------- | ------------ | ----------------- | | show to staff on the 360 | yes | yes | yes | yes | yes | no (re-fetch) | | apply to the record (becomes the record's own value) | yes | yes, photo by copy on acceptance | yes | yes, **but** the value is still labelled provider-sourced and expires with the licence | yes | no | | show in the member portal or a directory | yes | yes | yes, under the named application agreement | **no** | | | | include in a tenant data export, mail merge, webhook, Zapier subscription | yes | yes | no | **no** | | | | include in an LLM prompt (Describe with AI, Intelligence, research) | yes | yes | per agreement | no | **no** | | | keep after the provider connection ends | yes | yes | per agreement (FullContact: delete within 30 days) | per agreement (default: delete) | | | **Accepting a value does not remove its licence restrictions.** If a reviewer accepts an `internal_staff` claim as the contact's job title, the record holds a value whose origin is licensed. Its provenance row keeps the licence class and expiry. Export and portal display must consult the policy function **for that field** while its provenance says `provider`. Once a member or staff member edits the field, its provenance becomes `member` or `staff` and the licence lapses. This makes a licensed value useful to staff without allowing it to leak into an unlicensed use. **Photos require a separate acceptance step.** Provider photos are never persisted on receipt. The suggestion shows a proxied preview with a short-lived URL. On acceptance, and only when the licence class permits a copy (`open`, `self_published`, `customer_app` with photo rights), the image is imported through CRM's asset pipeline as the contact's own photo, with provenance `provider`. That pipeline checks for server-side request forgery (SSRF), caps image size and inspects the bytes to identify the file type (**Verified**: [r2 §9](#19-evidence-r2-crm-rhythm-2-0-crm-as-the-target-of-data-enrichment)). Gravatar is the one source where the member published the image for exactly this purpose. Its guideline is to refresh stored data at reasonable intervals, so a Gravatar-sourced photo carries `expires_at` of 180 days and a re-probe. No face detection or matching code path exists anywhere in the proposed design (decision ENR-D5). ### 5. AI research as a separate, human-approved pipeline (ENR-M6) AI research is proposed as a separate pipeline, not another source that can silently update a member record. A person must review every result. - **Controlled browsing.** It runs as an LLM-module operation `enrichment.research` inside Platform, using an allow-listed web search tool. It respects robots rules and never uses sources behind a login or a social network's walls. - **Suggestions only.** Every claim carries `source_kind: 'ai_research'` and a `source_ref` URL. It is **always** a pending suggestion; no policy can auto-apply it. - **Minimal inputs.** The model receives name, employer and city for a professional context, never a personal e-mail or phone. - **Filtered outputs.** Filters drop special-category inferences, including health, religion, politics, union membership and sexual orientation, and any claim without a citation. - **Licence restrictions still apply.** Licensed claims with `no_ai` never enter the prompt as context. - **Notice before research.** The European Data Protection Board's (EDPB's) 2026 draft web-scraping guidelines cover training rather than inference-time browsing. They nevertheless flag models connected to search tools for profiling and require individual notice where contact details exist (**Market**: [w4 §9](#26-w4-evidence-report-privacy-law-and-data-licensing-constraints-on-rhythm-2-0-data-enrichment)). The module therefore records the notice date before any research step runs for a subject in a notice-requiring region. - **Search service remains open.** The Anthropic web-search tool is not available on Bedrock, so a search API (Exa, Parallel, Tavily) carries the browsing (**Market**: [w3 §11.C](#25-w3-evidence-enrichment-provider-landscape-for-an-association-management-system)). Which one is a later decision. ### 6. Retention, erasure and audit The proposed controls remove data when permission expires, propagate a member's erasure request and retain enough history to explain what happened without logging provider payloads. - **Retention by rule, not by habit.** Every claim and cache row has `expires_at`, computed at receipt as the minimum of the licence time to live (TTL), the policy's retention and 30 days for cache. A daily retention function deletes expired data. The same function applies provider deletion notices, including ZoomInfo's monthly list and PDL changelog deletions. - **Erasure propagates.** A CRM hard delete or erasure of a contact deletes the subject's rows and writes a suppression row of reason `erasure`. In ENR-M4, it also calls each provider's own deletion or suppression endpoint where one exists and records completion evidence (**Market**: [w4 §14.6](#26-w4-evidence-report-privacy-law-and-data-licensing-constraints-on-rhythm-2-0-data-enrichment)). - **Access requests.** The subject's enrichment history is exportable per record for an Art. 15 response: which providers were asked, when, what field groups were requested and what was applied. The export excludes claims the licence forbids exporting. - **Audit without values.** `security_audit_events` record policy, suppression and allowance changes, including before, after and reason. The module's tables hold request and decision history with identifiers and digests, never claim values. CRM's revisions record every applied value, as today. A provider call's request and response payloads are never recorded (the `personal_data` enforcement fix, ENR-M1). - **Residency.** The gateway records the region of each execution. The provider registry records each provider's processing locations and transfer mechanism. A region rule can forbid a provider for EU members. ### 7. What is generic and what is provider-specific Build the enforcement mechanisms once, but configure them from each provider's contract. A shared framework cannot make different licences interchangeable. In the table, BYOK means “bring your own key”: the association supplies its provider credentials. An OEM addendum is the agreement needed for Rhythm to offer a provider's service through its own product and account. | Requirement | Handled once, in the framework | Handled per provider, in the manifest or contract | | --------------------------------------- | ----------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | | Lawful basis, notice, DPIA attestations | policy attestations per tier; region rules | extra provider duties (FullContact §4(b) consents; ZoomInfo UK/EU lawful-basis clause) recorded on the connection | | Minimisation | request only policy groups; adapters drop unrequested fields; conformance checks it | which fields the provider returns at all | | Objection and suppression | the ledger, the pre-call check, member self-service, erasure propagation | the provider's own opt-out feed or endpoint | | Retention and deletion | `expires_at`, the retention function, deletion on termination | the licence's TTL and deletion window; the provider's deletion notice format | | Display, export, AI | the policy function and licence classes | the class the contract assigns; the named application agreement for `customer_app` | | Keys and quotas | tenant-scoped connections; per-connection rate limits; per-tenant allowance | BYOK versus `platform_account`; the OEM addendum for a Rhythm-held key; a platform-wide bucket | | Photos | preview proxy; copy on acceptance only; no biometrics | whether the licence permits a copy | | Prohibited uses | code: no eligibility, credentialing or scoring input from enrichment; contract: flow-down | provider-specific prohibited-use lists | | Audit | security audit events; request history; CRM revisions | provider audit rights (FullContact SQL logs and three-year records) kept satisfiable by the request history | ### 8. What counsel must confirm before launch (not engineering decisions) These are legal confirmations required before launch, not decisions the architecture can settle. 1. Rhythm's customer data processing agreement (DPA) names enrichment as a processing purpose, lists provider recipients in two categories (processors and independent controllers), promises no secondary use by Rhythm, and carries each provider's flow-down restrictions ([w4 §12.1](#26-w4-evidence-report-privacy-law-and-data-licensing-constraints-on-rhythm-2-0-data-enrichment)). 2. A member-facing notice block the association can adopt ([w4 §12.2](#26-w4-evidence-report-privacy-law-and-data-licensing-constraints-on-rhythm-2-0-data-enrichment)). 3. Whether an association that is a religious, political or trade-union body may use tier 2 at all without consent. The Art. 9 inference in w4 is the researcher's, not settled. 4. The OEM or data-licence terms for any Rhythm-held key. PDL's own terms could not be read; FullContact's Exhibit A is the model; Logo.dev's full terms were not read (**Verified**: [w5](#27-w5-supplementary-provider-checks-read-directly-2026-10-22-11-00-to-11-30-et)). 5. Whether storing a provider photo on acceptance is permitted under each provider's terms, or whether Rhythm must stay at “link and proxy” for all but Gravatar. ## 11. 10. API and event architecture ## 10. API and event architecture Status: proposed 2026-10-22. Nothing in this proposed enrichment design is decided or built. **Verified** statements describe existing Platform or CRM capabilities, not completed enrichment work. Labels follow the [README](#1-rhythm-2-0-data-enrichment-architecture). The recommendation is to use Platform's existing API, service-call and business-event patterns. Staff actions enter through the API, other services call a dedicated service function, and background workers handle provider calls and accepted suggestions. This separates the decision to enrich a record from the work of fetching and applying data. It also means an accepted suggestion may be queued rather than applied immediately. Routes follow Platform's per-module REST API: `r2-platform-enrichment-api` behind the service hostname, with one API Gateway resource and method per route and no `{proxy+}` catch-all. Wire names use snake_case, and the association is identified by the `{tenantId}` path variable. Every field-changing write requires a record version precondition (record write contracts W1), so an update cannot silently overwrite a newer change. Creates use client-assigned UUIDv7 identifiers (W5). ### 1. Permission keys Permissions separate access to the application from authority to change policy, request enrichment, review suggestions or operate across associations. They are declared with `definePermissions` in the module and published at deploy (**Verified**: [r3 §2.1](#20-r3-platform-core-evidence-for-the-data-enrichment-architecture)). | Key | Tier | Grants | | -------------------------------- | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | `enrichment.application.access` | application | the licence and access gate for the application | | `enrichment.policy.manage` | admin, sensitive | switch enrichment on, choose tiers, order providers, set budgets and field rules, record attestations | | `enrichment.subjects.read` | read | the 360 panel, a subject's history, claims through the policy function | | `enrichment.subjects.query` | read | the two datasets | | `enrichment.requests.create` | user | "Enrich now" on a record; the workflow action | | `enrichment.suggestions.review` | user | accept, reject, defer a suggestion; bulk decisions (the apply still needs the target object's `.update` and `.bulk` keys, re-checked by CRM) | | `enrichment.runs.manage` | admin | start, pause, cancel bulk and scheduled runs | | `enrichment.suppressions.manage` | user, sensitive | record or lift "do not enrich" | | `enrichment.usage.read` | read | allowance, usage, estimates | | `hq.enrichment.operate` | staff | cross-tenant usage, overrides, provider price lists, forced retention | A built-in role, "Data quality reviewer", holds `subjects.read`, `requests.create`, `suggestions.review`, `usage.read`. The responsibility `crm.data_quality_review` is the default Attention audience, so review work goes to the staff assigned that responsibility. The responsibility already exists but is unused (**Verified**: [r2 §7](#19-evidence-r2-crm-rhythm-2-0-crm-as-the-target-of-data-enrichment)). ### 2. Routes The API covers policy, individual requests, suggestion review, bulk runs, suppression and usage. A subject is the CRM record being enriched; a suggestion is a proposed field change awaiting a decision or application. Creating a bulk run produces an estimate first. Work begins only when the run is started. Permission keys in the table omit the `enrichment.` prefix, except for the full HQ key. | Method and path | Key | Body or query | Answer | | -------------------------------------------------------------------- | --------------------------------------------- | ------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | `GET /enrichment/{tenantId}/policy` | `policy.manage` or `subjects.read` (redacted) | | the policy, provider settings, field rules, attestations | | `PUT /enrichment/{tenantId}/policy` | `policy.manage` | the policy with `sys_version` | 200 or 409 stale | | `PUT /enrichment/{tenantId}/policy/providers` | `policy.manage` | ordered settings per operation (connection ids from the Library's discovery route) | validates each connection offers the capability with the tier's features | | `PUT /enrichment/{tenantId}/policy/field-rules` | `policy.manage` | rules per object and group | | | `GET /enrichment/{tenantId}/subjects/{objectId}/{recordId}` | `subjects.read` | | subject, open suggestions, provenance per field (with claims the policy function allows), last requests | | `POST /enrichment/{tenantId}/requests` | `requests.create` | `{ id (UUIDv7), object_id, record_id, groups[], wait_ms? ≤ 20000 }` | 201 `{ request_id, status }`; with `wait_ms` the route waits for the single-step sync plan and returns `completed` or `running` | | `GET /enrichment/{tenantId}/requests/{id}` | `subjects.read` | | the request with match evidence and counts; execution ids link to the Library | | `GET /enrichment/{tenantId}/suggestions` | `suggestions.review` | `state, object_id, record_id, kind, field_group, run_id, cursor` | keyset page | | `POST /enrichment/{tenantId}/suggestions/{id}/decisions` | `suggestions.review` | `{ decision: 'accept' \| 'reject' \| 'defer', sys_version, reason?, options? }` | 200 with the new state; accept queues the apply; `options` carries `effective_date`, `previous_affiliation`, `create_organization` for employment changes | | `POST /enrichment/{tenantId}/suggestions/decisions` | `suggestions.review` + object `.bulk` | `{ filter: {...}, decision, dry_run }` | counts; non-dry-run submits one CRM data job per object type | | `POST /enrichment/{tenantId}/runs` | `runs.manage` | `{ id, kind: 'bulk', object_id, record_ids[] \| selection_ref, operation, groups[] }` | 201 with the estimate (a dry run until `/start`) | | `POST /enrichment/{tenantId}/runs/{id}/start` · `/pause` · `/cancel` | `runs.manage` | | the Durable Job's answer | | `GET /enrichment/{tenantId}/runs` · `/runs/{id}` | `runs.manage` or `usage.read` | | counts, status, link to suggestions filtered by run | | `POST /enrichment/{tenantId}/suppressions` · `DELETE …/{id}` | `suppressions.manage` | `{ identifier_kind, identifier, reason, source_ref? }` | the hash is computed server-side; the identifier is never stored | | `GET /enrichment/{tenantId}/usage` | `usage.read` | `period` | allowance, reserved, settled, by provider and trigger | | `POST /enrichment/{tenantId}/estimates` | `usage.read` | `{ object_id, record_ids[] \| selection_ref, operation, groups[] }` | `{ eligible, suppressed, cached, estimate_units, remaining_units }` | | `POST /system/enrichment/tenants/{tenantId}/allowance-overrides` | `hq.enrichment.operate` | `{ credits, from, to, reason }` | an HQ windowed top-up, audited | | `GET /system/enrichment/usage` | `hq.enrichment.operate` | | cross-tenant daily usage | Retries must not create duplicate work. Every `POST` that creates carries a client UUIDv7 and is replay-safe for 30 days through the API workload governance ledger. This is the idempotency contract: repeating the same create returns the existing result rather than creating another item. `requests` also deduplicate on `(subject, identifier_digest)` within the cache window. A second "Enrich now" with nothing new to send returns the existing request. ### 3. Service function (for other macroservices and the workflow executor) Other macroservices and the workflow executor use `platform_enrichment_service`, rather than the staff-facing HTTP routes. Calls use IAM, AWS Identity and Access Management, in trusted-invoke mode and follow the service-calls pattern (**Verified**: [r3 §2.5](#20-r3-platform-core-evidence-for-the-data-enrichment-architecture)). ```ts type EnrichmentServiceOperation = | { op: 'request'; input: { object_id: string; record_id: string; groups: FieldGroup[]; trigger: Trigger; correlation_id: string } } | { op: 'subject'; input: { object_id: string; record_id: string } } | { op: 'suppress'; input: { identifier_kind: 'email' | 'phone' | 'record'; identifier: string; reason: SuppressionReason; source_ref?: string }; } | { op: 'estimate'; input: { object_id: string; record_ids: string[]; operation: Operation; groups: FieldGroup[] } }; ``` Membership, Events and Committees call `request` after their own completion events when they need enrichment sooner than the CRM-level trigger would provide it. CRM's consent ledger calls `suppress` when a member records "do not enrich". ### 4. Events (Platform business events; transactional outbox; at least once) Business events let workflows and other modules respond to enrichment without coupling their work to a provider call. The transactional outbox records an event alongside the state change that produced it, so the change and its notification are committed together. Delivery is at least once: consumers must expect duplicate delivery. | Event | Payload (no personal values) | Consumers | | --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- | | `enrichment.request.completed` | `{ request_id, subject: { object_id, record_id }, trigger, outcome: 'matched' \| 'no_match' \| 'failed' \| 'suppressed' \| 'budget_refused', suggestions_created, units }` | workflow rules ("when enrichment finds a match…"), Intelligence later | | `enrichment.suggestion.created` | `{ suggestion_id, subject, field_key, kind }` | Attention batching, automation | | `enrichment.suggestion.decided` | `{ suggestion_id, subject, field_key, state, decided_by_id, auto }` | audit consumers | | `enrichment.suggestion.applied` | `{ suggestion_id, subject, field_key, applied_revision_ref }` | the provenance subscriber, Intelligence | | `enrichment.employment_change.detected` | `{ suggestion_id, subject, organization_id?, organization_candidate_present }` | Membership (a chapter-transfer prompt, later), Attention | | `enrichment.run.completed` | `{ run_id, kind, counts }` | Notifications | | `enrichment.budget.threshold_crossed` | `{ period, threshold, settled, allowance }` | Notifications to administrators, HQ alarm at 100 % | | `enrichment.subject.suppressed` | `{ subject, reason }` | CRM consent ledger (when it exists), audit | Payloads carry identifiers and digests, never claim values. A claim is a provider's assertion about a field. Keeping those values out of events means the 90-day event ledger holds no licensed data. The module consumes these events: - `crm.contact.created`, `crm.organization.created` (CRM-M15). - `crm.contact.updated` and `crm.organization.updated`, for provenance and portal locking. Until these business events exist, the module uses the change-event stream. - `crm.contact.merged`, to run the participant's after-merge housekeeping. - `membership.membership.activated`. - `platform.data_job.completed`, for import follow-up. - `rhythm.integration.execution.completed`, for async provider steps. - `enrichment.person.changed`, for watch feeds (ENR-M4). ### 5. Workflow action and automation trigger points Staff can use enrichment in workflows and respond to its results without writing code. - Action type `enrichment.enrich_record` (`durable`; params and outputs in [06 §5](#7-06-core-workflows-and-use-cases)). - Trigger points raised by the module for association rules: `enrichment.request.completed`, `enrichment.employment_change.detected`, `enrichment.suggestion.applied`. An association can therefore write "when an employment change is detected for a member of chapter X, create a task for the chapter administrator" without any code. ### 6. Background functions Background functions keep provider calls, bulk work and maintenance out of the staff's API request. SQS is AWS's queue service; EventBridge supplies the daily schedules. Concurrency limits control how many function instances run at once, but provider throughput is ultimately constrained by each connection's rate limit. | Function | Shape | Concurrency | Why | | --------------------------------------- | ----------------- | ------------------- | ----------------------------------------------------------------------------------------------------------------- | | `platform_enrichment_event_intake` | event intake | default | receives business events and queues subjects | | `platform_enrichment_dispatcher` | queue dispatcher | 1 | moves queue rows to SQS (the queued-work standard) | | `platform_enrichment_worker` | SQS consumer | reserved 10 (dev 4) | builds plans, calls the gateway, evaluates claims; the per-connection rate limit is the real throttle | | `platform_enrichment_apply` | SQS consumer | reserved 4 | submits CRM data jobs (ENR-M1) or service-function applies (ENR-M2+) for accepted suggestions, batched per object | | `platform_enrichment_refresh_scheduler` | EventBridge daily | 1 | starts per-tenant refresh runs | | `platform_enrichment_retention` | EventBridge daily | 1 | expires claims, suggestions and cache; honours provider deletion notices | | `platform_enrichment_run_task` | Durable Job task | per job | fans a run out into queue rows in batches of 25 | | `platform_enrichment_api` | HTTP | default | the routes | | `platform_enrichment_service` | service function | default | §3 | Observability follows `docs/observability.md`, with observed boundaries on every function. Metrics track request outcomes, settled units, open suggestions and application failures: - `enrichment.requests.count{outcome}` - `enrichment.units.settled{connection}` - `enrichment.suggestions.open` - `enrichment.apply.failed{reason}` Alarms cover worker DLQ depth, the dead-letter queue holding work that could not be processed, the `budget_refused` rate per association, and apply failures. ### 7. Idempotency, retries, concurrency, outages, security: the rules in one place These rules protect staff changes, member preferences and association budgets when events repeat, providers fail or work overlaps. RLS means row-level security: database rules that restrict access to each association's rows. | Concern | Rule | | --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | A trigger fires twice | event intake dedupes on `envelope.id`; the subject's `identifier_digest` and the cache make a second request within the window free; the gateway's idempotency key (`request::`) makes a retried step return the stored answer (**Verified**: [r1 §4.5](#18-r1-the-integration-platform-as-built-evidence-for-a-data-enrichment-design)) | | A provider times out or is unavailable | the gateway retries within the budget; the step ends `provider_unavailable`; the request ends `failed` with the class; the subject stays `scheduled` and the next trigger or refresh tries again; no budget is settled for a failed step (reservation released) | | An `unknown` outcome | treated as no answer; never reconciled (no side effect to settle); the reservation is released | | Two workers pick the same subject | the queue row is deleted on SQS acceptance; a request row is inserted with a unique `(tenant_id, subject_id) WHERE status IN ('queued','running')` partial index; the loser skips | | A reviewer and an automatic fill race | the apply re-reads the record version; CRM refuses a stale version (W1); the suggestion returns to pending | | The record changed after the suggestion | `current_value_digest` mismatch → `apply_failed: record_changed`, surfaced to the reviewer | | Budget exhausted mid-run | each subject's reservation fails individually (`budget_refused`); the run completes with counts; a run never applies anything itself, so no partly applied record results | | Rate limits | the gateway's per-connection buckets; the worker backs off on 429 using `retry_after`; a Rhythm-managed connection gets a platform-wide bucket (ENR-M5 dependency) | | Credentials | never in the module; the gateway leases them to the adapter (**Verified**: [r1 §3](#18-r1-the-integration-platform-as-built-evidence-for-a-data-enrichment-design)) | | Payload recording | refused for `personal_data` capabilities (the fix in ENR-M1); summaries drop personal-looking fields | | Tenant isolation | forced RLS on every table; the worker runs with the tenant's context from the queue row; cache keyed by tenant and connection | | Audit | `enrichment.policy.changed`, `enrichment.suppression.recorded`, `enrichment.allowance.overridden` as `security_audit_events` (before/after, reason); request and decision history in the module's own tables; CRM's revisions remain the audit of every applied value | ## 12. 11. Console and portal experience ## 11. Console and portal experience The recommendation is to make enrichment part of the existing record and data-quality workflows, with the same experience whichever provider is connected. Staff should see proposed changes and understandable evidence, not provider-specific controls. A staff member should never see a provider's name unless they administer providers. **Status: proposed 2026-10-22. Nothing in this chapter is decided or implemented.** Labels are defined in the [README](#1-rhythm-2-0-data-enrichment-architecture). This chapter describes the proposed screens and interactions; the package's decisions are presented separately in document 15 for Andrew to approve or overturn. Every console screen is designed in v0 first (Andrew, 2026-10-21) and preserves the approved baseline. The portal comes last, using Andrew's components. ### 1. Contact 360 and Organization 360 **The enrichment panel.** Staff review enrichment on the record's overview, in a collapsible panel placed by the record layout like any other panel. Contact and organization layouts are configurable (**Verified**: [r2 §5](#19-evidence-r2-crm-rhythm-2-0-crm-as-the-target-of-data-enrichment)). The panel shows: - A one-line status: "Enriched 12 Mar 2026 · 2 suggestions · next refresh Mar 2027", "Not enriched", "Enrichment paused for this person" for a suppressed record, or "Enrichment is off for this workspace". - Pending suggestions, each as a row with the field's label, current value, proposed value, a confidence word (`High`, `Medium`, `Low`; the underlying score is shown only to administrators on the Sources tab), and three actions: Accept, Reject, Later. - An "Enrich now" button, requiring `enrichment.requests.create`. Pressing it shows "Looking…" for up to the gateway budget, the time allowed for the request to return. It then shows new suggestions, "Nothing new was found", or "Still working; we'll add a badge when it finishes". - A "Where this came from" link that opens the provenance view, showing the origin of the information. **Field badges.** Staff can distinguish accepted enrichment from other data without changing the record layout. Any field value whose provenance is `provider` or `ai_research` has a small source mark. Hovering shows "Suggested by enrichment, accepted by R. Patel on 12 Mar 2026, last verified 12 Mar 2026". Member-entered values have no mark, but their hover says "Entered by the member in the portal, 3 Feb 2026". Staff-entered values show nothing. This applies the HubSpot and Attio pattern (**Market**: [w2 §7](#24-w2-evidence-how-enterprise-crm-erp-platforms-do-data-enrichment)) to Rhythm's existing record layouts without changing them. **Employment change card.** An `employment_change` suggestion appears as a card rather than a row, so staff can see the consequences before accepting it. The card shows: - The current primary affiliation: organization, title and since date. - The proposed affiliation: an organization, or "a new organization: Globex Corp, globex.com". - The evidence: which identifiers matched and when the provider observed the change. - The consequences in plain words: "Acme affiliation will end on 11 Oct 2026; 2 automatic roles will be removed; Globex gains a member". Actions are Accept, with an effective date and a "keep the previous affiliation" option; Pick a different organization; Create organization; and Reject. **Photo suggestion.** A thumbnail preview appears beside the current photo. The preview is proxied, not stored. Accept imports it through the asset pipeline. Reject records the decision so the same image is not suggested again. **Low-confidence drawer.** A "2 low-confidence matches" link opens claims that the match rule did not accept. The claims are read-only, with one action: "This is the right person". That action promotes the claims to suggestions and records who confirmed the match. ### 2. Contact creation and import Enrichment does not interrupt record creation or import: - The New Contact and New Organization forms do not change. Enrichment runs after save. The 1.0 pattern of calling a provider when a user leaves a field is retired (**Verified**: [r5 §1](#22-r5-rhythm-1-0-fullcontact-and-google-civic-enrichment-and-the-customer-360-precedent)). - After an import completes, its results screen gains one line: "Enrichment follow-up queued for 1,980 records (estimate 790 units of 5,000 remaining)", with a link to the run. ### 3. The review queue (Data Quality: Suggestions) Staff can review suggestions across records in a list under CRM's Data Quality area, beside Duplicates. The screen uses the shared one-row toolbar with filter, search and refresh. Business unit is not relevant here. Filters are object type, suggestion kind, field group, confidence, run, age and reviewer responsibility. Each row shows the record as a quick-search-style link, the field, current and proposed values, confidence, and a source-kind word: "Provider", "Public web" or "Member-published". Bulk actions follow the PLAT-M20 dialog: 1. Select rows or "all matching", capped at 10,000. 2. Preview the count and the estimate of what will be written. 3. Apply the action. 4. View the results, with a link to the data job. Bulk rejection requires a reason. "Later" sets a snooze date. Attention items such as "3 suggestions for Jane Doe" deep-link into the record's panel. The responsibility `crm.data_quality_review` is the default audience and can be changed for each association. ### 4. Bulk actions on lists Contact and organization lists gain one bulk action: "Enrich". Its dialog lets staff choose the groups to request, using checkboxes labelled in plain words: - "Job title and employer" - "Public profiles" - "Photo" - "Organization details" - "Verify e-mail addresses" Before starting the run, the dialog shows the estimate: "8,214 records · about 3,300 units · 5,000 remaining this month". Progress appears in the existing jobs area. Completion arrives as a Notification linking to the review queue filtered by run. ### 5. Configuration: the Enrichment settings area Administrators control policy, sources, review rules and spending in an "Enrichment" section under Configuration. Four tabs are administered with `enrichment.policy.manage`: 1. **Overview**: turn enrichment on or off; explain the tiers in plain language; and provide the attestation form for each tier, including "Our privacy notice names enrichment and its providers since … ", "Lawful basis" and "Assessment reference". This tab also contains region rules and the tenant-type question: general, or religious, political or trade-union body. 2. **Sources**: order providers for each operation, choosing from connected integrations in the Library. The Library's own screens handle install, connect and credentials (**Verified**: [r1 §12](#18-r1-the-integration-platform-as-built-evidence-for-a-data-enrichment-design)). Each source row shows its licence summary in words, such as "Staff use only; not shown to members; not exported", its verified capabilities and its unit price. "Test with a record" runs a dry request and shows the claims it would produce without storing them. 3. **Rules**: set behavior per object and field group: "When empty: fill automatically / suggest / ignore" and "When different: suggest / ignore". This tab also defines protected fields; custom-field mappings for provider attributes that CRM has no column for, such as industry and employee range; the reviewer responsibility; and triggers. Triggers cover on create, after import, after portal update, after membership activation, scheduled refresh cadence and daily cap. 4. **Budget and usage**: set the monthly allowance, hard stop or overage, and notification thresholds. Show this month's usage by source and trigger, match rate per source, cost per applied suggestion and the last ninety days. Link to the Library's execution log for the raw calls. A fifth tab, **Suppressions**, lists "do not enrich" entries as masked identifiers (`j***@gmail.com`), with who recorded them and why. Add and lift actions require `enrichment.suppressions.manage`. ### 6. Rhythm HQ The Rhythm HQ staff view provides cross-customer oversight and provider administration: - Usage by association and provider for the month. - Associations near or over allowance. - Windowed allowance top-ups, with a reason and an audit record. - Provider price lists per connection method. - Forced retention sweeps for a retired provider. - The `REGISTERING_ADAPTER_KEYS` and conformance state for each enrichment adapter. These would be the first HQ integration screens. None exist today (**Verified**: [r1 §12](#18-r1-the-integration-platform-as-built-evidence-for-a-data-enrichment-design)). ### 7. Member portal (ENR-M4, built last) The portal gives members an honest explanation and control over their records, but only displays information the association is licensed to show: - On the member's profile page, provider-sourced values appear with a short sentence drawn from the association's own notice: "Some professional details were added from public and partner sources; you can change them or ask us not to do this". A "Keep my record as I enter it" switch records a suppression through CRM's consent ledger and expires pending suggestions. - Displaying a provider-sourced value in the member portal is itself gated by the licence policy function. A value of class `internal_staff` is not shown to the member at all, so the member sees only values the association could lawfully show. - No enrichment runs from portal actions except the profile-update trigger. Nothing in registration or checkout waits for enrichment. ### 8. What stays invisible on purpose Provider names, likelihood numbers, execution ids and licence classes never appear to a reviewer; unit estimates appear only to those who may start runs or read usage. They appear to the administrator on the Sources and Budget tabs and to HQ. The reviewer sees a field, a current value, a proposed value, a confidence word and the evidence in words. This is the "consistent enrichment experience regardless of which provider is configured" requested in the brief. ## 13. 12. Commercial model and economics ## 12. Commercial model and economics **Recommend an included baseline, customer-provided keys for premium data in wave one, and Rhythm-managed premium credits only after provider-specific written agreements.** The framework should support all three models without making resale a launch dependency. Status: proposed 2026-10-22. This chapter's recommendations are not approved or built. References to existing framework capabilities describe what is available to build on, not a completed enrichment product. Decisions **ENR-D4** and **ENR-D8** in [15](#16-15-open-architectural-decisions) govern this chapter. Labels follow the [README](#1-rhythm-2-0-data-enrichment-architecture). **Published** numbers come from vendor pages, with an evidence report and key; **Derived** numbers are arithmetic on published inputs; **Reported** numbers are reported prices; **Assumed** numbers are planning placeholders without vendor evidence. ### 1. The three models, and why the framework must support all three The main distinction is who contracts with and pays the data provider. That determines which services Rhythm can include, which need an association's own account, and which require a resale agreement. | Model | Who holds the provider account | Who pays the provider | Who accepts the provider's terms | Framework mechanism | Verdict | | ---------------------------------------------------------- | ------------------------------------------------------------------ | -------------------------------- | ------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **A. Rhythm-provided baseline, included** | Rhythm (`platform_account`) | Rhythm, from the subscription | Rhythm, for sources whose terms allow multi-customer use | an HQ-granted default licence of the `enrichment` application; allowance from the licence; platform-wide rate bucket | **Include**, limited to sources that are free or near-free and whose terms permit it: Logo.dev logos ("store anything returned, on every plan", **Verified**: [w5 §4](#27-w5-supplementary-provider-checks-read-directly-2026-10-22-11-00-to-11-30-et)), Gravatar avatar probes (free API, responsible-use guidelines, **Verified**: [w5 §5](#27-w5-supplementary-provider-checks-read-directly-2026-10-22-11-00-to-11-30-et)), GLEIF and Wikidata identifiers (CC0, **Market**: [w3 §11.B](#25-w3-evidence-enrichment-provider-landscape-for-an-association-management-system)), e-mail verification at $0.002 to $0.02 per check (**Published**: [w3 §11.D](#25-w3-evidence-enrichment-provider-landscape-for-an-association-management-system)) | | **B. Customer-provided integrations (bring your own key)** | the association | the association, to the provider | the association | the Integration Library as built: install, connect with an `api_key`, verify (**Verified**: [r1 §3](#18-r1-the-integration-platform-as-built-evidence-for-a-data-enrichment-design)); the allowance records units for visibility and the policy's own cap | **The default for premium person data in wave one**, for providers whose terms allow a customer-held key to be used inside a third-party application (PDL and FullContact by contract; Apollo and ZoomInfo do not). It keeps each provider's terms between the association and the provider, which several licences require (Apollo and ZoomInfo bar on-behalf and third-party-app key use; **Market**: [w4 §1](#26-w4-evidence-report-privacy-law-and-data-licensing-constraints-on-rhythm-2-0-data-enrichment)) | | **C. Rhythm-managed premium, billed as credits** | Rhythm (`platform_account` with an `account_qualifier` per tenant) | Rhythm, recharged as credits | Rhythm under a signed OEM or data-licence addendum; the association accepts flow-down terms | the allowance ledger with Rhythm-set `recurring_credits` and HQ overrides; a platform-wide rate bucket; HQ usage screens | **Later (ENR-M5), and only per provider after written terms.** No self-serve licence read permits Rhythm to re-supply results to its customers (**Market**: [w3 §11.B](#25-w3-evidence-enrichment-provider-landscape-for-an-association-management-system)); FullContact's agreement shows the shape such terms take (a named Customer Application, end-user flow-downs, deletion acknowledgments; **Verified**: [w5 §2](#27-w5-supplementary-provider-checks-read-directly-2026-10-22-11-00-to-11-30-et)) | All three models would use the same tables, plan and policy. The differences are the connection's `auth_method`, the source of `recurring_credits`, and whether a platform-wide rate bucket is needed to protect a shared key. That bucket limits requests across all workspaces using the key, rather than letting each workspace consume the provider's shared quota independently. ### 2. What is in the core product and what is optional Every workspace should get useful baseline enrichment and the tools to review and control it. Premium data remains optional, with the association paying its provider directly at first. | Included for every workspace (model A) | Optional, association-paid (model B) | Optional, Rhythm-sold (model C, after terms) | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | Organization logos and identifiers; Gravatar avatar probe; e-mail verification up to an included monthly allowance; the review queue, provenance, policy and budget screens; "Enrich now" with any connected provider | Person and organization enrichment with the association's own PDL, FullContact or other key; scheduled refresh; import follow-up; the workflow action | Rhythm-managed person and organization credits; employment-change watch; AI research credits (which also draw on the AI allowance) | E-mail verification belongs in the baseline because it is inexpensive and improves deliverability and duplicate detection for every association, regardless of tier. Its providers' terms concern the e-mail check, not a person profile. ### 3. Unit economics at 10,000, 100,000 and 1,000,000 contacts These are planning costs, not enterprise quotes. The assumed match rate is the largest driver of premium person-data cost; enterprise pricing remains unknown. Inputs, each tagged: | Input | Value | Tag | Source | | --------------------------------------------------------------------------- | --------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | | Premium person enrichment, per matched record | $0.28 (entry), $0.25 (higher self-serve tier) | Derived from Published ($98 for 350 credits) / Reported | [w5 §1](#27-w5-supplementary-provider-checks-read-directly-2026-10-22-11-00-to-11-30-et), [w3 §11.D](#25-w3-evidence-enrichment-provider-landscape-for-an-association-management-system) | | A miss costs | $0 | Published (PDL: 404 not charged) | [w3 §2.1](#25-w3-evidence-enrichment-provider-landscape-for-an-association-management-system) | | Match rate for association members (personal e-mails, name, some employers) | 40 % | **Assumed**; no vendor publishes one; PDL guidance says e-mail-only input scores low | [w3 §11.A](#25-w3-evidence-enrichment-provider-landscape-for-an-association-management-system) | | E-mail verification, per check | $0.005 | Assumed, within the Published range $0.002 to $0.0195 | [w3 §5](#25-w3-evidence-enrichment-provider-landscape-for-an-association-management-system) | | Organization enrichment (firmographics), per matched organization | $0.10 | Reported (PDL company credit) | [w3 §11.D](#25-w3-evidence-enrichment-provider-landscape-for-an-association-management-system) | | Logos | $0 to $0.02 | Published (Logo.dev CDN free within quota; $0.02 per API logo) | [w5 §4](#27-w5-supplementary-provider-checks-read-directly-2026-10-22-11-00-to-11-30-et) | | Distinct organizations per 10,000 contacts | 1,500 | **Assumed** | | | Annual refresh share | 10 % of contacts | **Assumed** (policy default `annual` cycles 100 %; the planning case refreshes those with a change signal only) | | | AI research (optional), per researched contact | $0.05 | Derived from Published (search $0.005 to $0.01 per request plus a few cents of tokens) | [w3 §11.D](#25-w3-evidence-enrichment-provider-landscape-for-an-association-management-system) | | Enterprise pricing at volume | unknown | PDL and FullContact "custom" | [w5 §1, §2](#27-w5-supplementary-provider-checks-read-directly-2026-10-22-11-00-to-11-30-et) | Results show one-time initial enrichment followed by a steady year (Derived). The match-rate assumption dominates the person-enrichment figures. Firmographics means information about an organization, rather than a person. | Contacts | Verification (all) | Person, 40 % matched at $0.28 | Organizations, 15 % of contacts at $0.10 | Initial total | Annual refresh (10 % × person) | Annual verification re-check (all) | Annual total | | --------- | ------------------ | --------------------------------------------------------- | ---------------------------------------- | -------------------------- | ------------------------------ | ---------------------------------- | ------------- | | 10,000 | $50 | $1,120 | $150 | **≈ $1,320** | $112 | $50 | **≈ $160** | | 100,000 | $500 | $11,200 | $1,500 | **≈ $13,200** | $1,120 | $500 | **≈ $1,600** | | 1,000,000 | $5,000 | $112,000 (at $0.25: $100,000; enterprise pricing unknown) | $15,000 | **≈ $120,000 to $132,000** | $11,200 | $5,000 | **≈ $16,000** | At a 20 % match rate, every person figure halves; at 60 %, it rises by half. Misses are free with the candidate provider. The allowance, rather than the match rate, bounds the risk identified in the brief: "an uncontrolled per-contact expense across millions of records". Optional AI research adds $0.05 per researched contact ($500 per 10,000). It should be used only for records a reviewer asks about or a rule selects, never for the whole contact base. **Pricing implication (Inferred):** under model B, the association pays these costs to the provider and Rhythm sells the capability. Under model C, Rhythm needs roughly a 2× to 3× markup on provider cost to cover support, reviewer tooling and the commitments in an original equipment manufacturer (OEM) or data-licence agreement. That puts a 100,000-contact association at about $25,000 to $40,000 for initial enrichment and $3,000 to $5,000 a year afterward. Whether Rhythm offers this as a product or a professional-services engagement is Andrew's call (ENR-D4). The baseline (model A) costs Rhythm pennies per contact and is intended to give every customer something useful on day one. ### 4. Budget enforcement, metering and cache reuse The proposed controls prevent unexpected spending, avoid paying twice for the same request, and make costs visible to association administrators and Rhythm headquarters (HQ). - **Allowance before the call.** Before contacting a provider, the worker reserves the plan's estimated units against the month's allowance in one conditional update. A bulk run refuses to start if its estimate exceeds the remainder. Thresholds at 80 % and 100 % notify administrators; at 100 %, billable steps stop unless the policy allows overage (`hard_stop`). - **Settlement from the adapter's `cost_units`.** After the call, the actual charge is calculated using the connection's `unit_price_credits`. A free miss settles to zero, and a Logo.dev content delivery network (CDN) hit costs nothing. - **Cache reuse is per association and per connection.** Cached answers, including misses, can be reused within 30 days or the licence's time-to-live (TTL). This prevents paying twice for the same question, such as a re-import or a second click. It does not share answers between customers ([P7](#5-04-architectural-principles)). - **Usage visibility.** The console shows the month's allowance, spending by provider and trigger, the match rate per provider, and the cost per applied suggestion. HQ sees the same information across associations ([11 §6](#12-11-console-and-portal-experience)). - **Migration to licensing v2.** When the entitlements and resource registry exist, `recurring_credits` will come from the licence's resource; the ledger stays. That platform work is decided in direction, not built (**Verified**: [r3 §4.2](#20-r3-platform-core-evidence-for-the-data-enrichment-architecture)). ### 5. OEM and resale agreements: needed or not **OEM or resale agreements are needed only for model C, and separately for each provider.** They are not a prerequisite for launching the proposed baseline and customer-provided integrations, but those models still require licence checks. Model A sources were selected because their published terms permit the use (Logo.dev, Gravatar, GLEIF, Wikidata), or because they are per-check services without a profile licence (verification). Before launch, Rhythm must still read Logo.dev's full terms and the verifiers' retention terms. Model B needs no agreement from Rhythm beyond its customer data processing agreement (DPA) listing the provider as the association's own recipient. Model C needs a written agreement with each of PDL and, if selected, FullContact. It must cover: - Display to end users, meaning association staff, and display in the member portal. - Caching and retention. - Deletion feeds. - AI use. - Allocation of controller and processor responsibilities. - An account qualifier for each association, so usage can be attributed to that association. These requirements are **Market**: [w4 §12.1](#26-w4-evidence-report-privacy-law-and-data-licensing-constraints-on-rhythm-2-0-data-enrichment). Model C would also require the association to accept the provider's flow-down terms, meaning the obligations Rhythm must pass on to its customers. ### 6. Rhythm's own exposure - **Protect shared capacity.** A shared Rhythm key needs both the platform-wide rate bucket and a Rhythm-level monthly cap, set by HQ. Otherwise, one association's bulk run could exhaust the provider quota shared by every other association. - **Do not become a data broker.** Rhythm must not share cached answers across associations, use provider data for secondary purposes, or aggregate provider answers across customers. The proposed design would enforce the first two restrictions; the contract would enforce the third. - **Plan for provider exits.** Provider volatility is real: Clearbit folded into HubSpot and its APIs closed; Data.com and DataFox retired; Proxycurl shut down under a LinkedIn settlement (**Market**: [w2 §2](#24-w2-evidence-how-enterprise-crm-erp-platforms-do-data-enrichment), [w3 §6](#25-w3-evidence-enrichment-provider-landscape-for-an-association-management-system), [w5 §3](#27-w5-supplementary-provider-checks-read-directly-2026-10-22-11-00-to-11-30-et)). The proposed adapter model is intended to limit a provider's exit to one adapter and a retention sweep, meaning a review and removal of stored data under the provider's retention requirements, rather than a product change. ## 14. 13. Phased implementation milestones ## 13. Phased implementation milestones The proposed sequence starts with one customer-provided person provider and staff-reviewed suggestions. It then adds organizations and automatic fill, bulk processing, employment changes, managed premium access, and finally opt-in AI research. This delivers a usable first release without waiting for every CRM foundation or committing to the more complex capabilities. Status: proposed 2026-10-22. Nothing in this sequence is decided or authorized for build. Labels follow the [README](#1-rhythm-2-0-data-enrichment-architecture). The approval decisions are listed in [15](#16-15-open-architectural-decisions). All milestones follow R2-DEV-001: - Nothing is built until Andrew approves a milestone plan written on the standard's template, covering sections 1 to 12, including 7a entity capabilities. - Each milestone ends with a cumulative automated acceptance run and a guided walkthrough. - At most one milestone is In Progress per application. The milestone Ids are **ENR-M1** to **ENR-M6**. Their records will live in services-platform, the module's owning repository, at `docs/milestones/ENR-M-.md`. Sizes are rough agent-day estimates (**Estimate**), not delivery dates. ### 1. What is reusable now, what is missing, what is deferred The existing integration and CRM infrastructure provides much of the execution machinery. This proposal adds the enrichment-specific rules, records and staff experience. The table separates those additions from capabilities deliberately left outside this implementation sequence. | Reusable immediately (Verified) | Missing foundations (this proposal adds) | Deferred on purpose | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | | Integration catalogue, installation, connections, credentials, leases, gateway, executions, rate limits, circuits, SDK, conformance, Library screens ([r1](#18-r1-the-integration-platform-as-built-evidence-for-a-data-enrichment-design)) | contracts category `enrichment` with the claim vocabulary and four capabilities ([07](#8-07-provider-contracts-and-orchestration)) | Clay-style graphs, per-field provider routing, cross-provider merging | | `platform_account` credential method; `cost_units` on executions | the enrichment module: policy, subjects, requests, claims, suggestions, provenance, suppressions, cache, allowance ([08](#9-08-data-model-provenance-and-policies)) | member-facing portal controls beyond "do not enrich" | | CRM data job `crm.contacts update` with `records[]`, version capture, receipts, rollback ([r4 §I-1](#21-r4-data-operations-data-job-engine-and-the-query-layer-evidence-for-rhythm-2-0-data-enrichment)) | CRM: published `created` events, `service` revision source, `apply_enrichment` service function (CRM-M15) | address hygiene (NCOA, deceased) as a separate capability on the same framework | | CRM duplicate normalisation, match keys, synchronous check route ([r2 §6](#19-evidence-r2-crm-rhythm-2-0-crm-as-the-target-of-data-enrichment)) | the identity acceptance rule ([06 §7](#7-06-core-workflows-and-use-cases)) | certifications and licences from credentialing bodies | | Attention, Notifications, business events, Durable Jobs, queued work, workflow actions, schedules ([r3](#20-r3-platform-core-evidence-for-the-data-enrichment-architecture)) | `personal_data` enforcement in payload recording; platform-wide rate bucket | AI research until the LLM module has a web-search tool and the `enrichment.research` operation (ENR-M6) | | The AI usage ledger pattern ([r3 §4.3](#20-r3-platform-core-evidence-for-the-data-enrichment-architecture)) | the licence policy function and classes ([09 §4](#10-09-security-privacy-and-licensing)) | employment watch feeds until a provider's terms and freshness justify it | | The asset pipeline for photos and logos ([r2 §9](#19-evidence-r2-crm-rhythm-2-0-crm-as-the-target-of-data-enrichment)) | HQ screens for usage and overrides | a third record type beyond contacts and organizations | ### 2. ENR-M1: Foundation and first provider (user-facing; ≈ 18 to 24 agent-days) **Objective.** Give staff a complete, review-first enrichment workflow with visible, capped spending. An association administrator enables one customer-provided person provider. Staff can press "Enrich now" on a contact or receive suggestions after creating one. They review, accept or reject suggestions on the Contact 360. Accepted values reach the contact through CRM, with full history and provenance showing where they came from. **Scope.** - **Contracts release:** add category `enrichment`, `enrichment.person.enrich@1` and `enrichment.organization.enrich@1`, the claim vocabulary, features and licence classes. The organization capability is declared but exercised only by the Dummy. Upgrade Platform to this release and refuse payload recording for `personal_data`. - **Adapters:** add both capabilities to `integrations-dummy` in stub mode. Add `integrations-pdl` as the first real provider, using bring-your-own-key (BYOK) authentication through `api_key`. Implement the conformance suite's four extra duties ([07 §4](#8-07-provider-contracts-and-orchestration)) and update the registration key list. - **Module records and policy:** add the tables in [08 §2](#9-08-data-model-provenance-and-policies), except runs, cache misses and watch. Support tiers 1 and 2; declare tier 3 but do not offer it. - **Triggers and execution plan:** support `manual` and `on_create`. Wire `on_create` to CRM's change-event stream until CRM-M15 publishes the business event, so it works in dev now. The plan has one baseline step, a Gravatar probe that stores only "has avatar" and the hash URL, never the image, followed by one provider step. - **Review and provenance:** implement identity acceptance and suggestions for `fill` and `change`. Provide the `employment_change` suggestion in "assisted apply" form: it opens CRM's edit screen pre-filled and is treated as applied when the change event confirms it. Record provenance from CRM's change events. Add the allowance ledger with policy-set credits and Attention items. - **Apply:** submit a CRM data job, using the `records[]` update, as the reviewer. Support scalar fields and mapped custom fields only. - **API:** expose policy, subjects, requests, suggestions with single decisions, usage, suppressions and estimates. - **Console, v0 first:** add the Contact 360 enrichment panel and field badges; Configuration › Enrichment with Overview, Sources, Rules, Budget and usage, and Suppressions; and the review queue list. Keep the Organization 360 panel read-only because there is no organization provider yet. - **Platform participation:** add merge participant functions for the new tables, the workspace lifecycle participant and the two registered datasets. - **Acceptance:** run L1 unit and persistence-layout tests; L2 integration tests with the Dummy in stub mode against the real gateway; L3 end-to-end tests on `acme-acceptance-main` with a seeded PDL sandbox key or the Dummy; and the Chrome walkthrough in `acme-acceptance-guided`. **Decisions required before the plan:** ENR-D1, D2, D3, D4 (BYOK first), D6 (tier 3 withheld), D8 (application key). **Dependencies:** INTG-M2, for calls through the gateway, must be Engineering Verified. Development also needs a PDL developer account, with a free tier of 100 credits a month (**Verified**: [w5 §1](#27-w5-supplementary-provider-checks-read-directly-2026-10-22-11-00-to-11-30-et)), and CRM's existing change-event stream. **Walkthrough preview:** switch on enrichment; connect PDL in the Library; order sources; set a 200-unit budget; create a contact with a personal e-mail, name and employer; see "Looking…" followed by three suggestions; accept the job title and see its badge and CRM history entry; reject the photo; open the employer card and complete the affiliation change in CRM's screen; see usage 1 of 200. ### 3. ENR-M2: Organizations, baseline sources, automatic fill (user-facing; ≈ 12 to 16) This milestone extends enrichment to organizations and allows selected values to be applied automatically, using CRM's service function rather than a staff reviewer's data job. **Scope.** - Add `integrations-logodev` for logos, with licence `open`, and `integrations-gravatar` for an avatar represented as a `self_published` photo claim on acceptance. - Add one e-mail verifier adapter and `enrichment.email.verify@1`, plus GLEIF identifiers for organizations. - Add the organization policy and 360 panel, with auto-fill for `brand` and `identifiers`. - Add the `on_registration` trigger from `membership.membership.activated` and the portal-update locking rule. - Implement automatic apply through CRM-M15's `apply_enrichment` service function, the first non-person apply path. This includes photos and logos through the asset pipeline. - Grant the baseline licence to all workspaces. **Dependencies:** CRM-M15, including service functions, `created` events and the `service` revision source; ENR-M1 accepted. **Decisions:** ENR-D5 (photos), ENR-D8 confirmed. ### 4. ENR-M3: Bulk, imports, schedules and automation (user-facing; ≈ 14 to 18) This milestone lets staff enrich selected lists and makes enrichment available after imports, on schedules and within workflows. Estimates and daily caps keep those larger runs bounded. **Scope.** - Add `enrichment_runs` and Durable Job fan-out to distribute a run across record-level jobs. - Add the list bulk action with an estimate, the import follow-up trigger, and scheduled refresh with cadence and a daily cap. - Support bulk decisions through CRM data jobs. - Add the workflow action `enrichment.enrich_record` and its trigger points. - Add the cache of misses, usage reporting by trigger, and Notifications when a run completes. - Add the adapter `batch` feature for PDL's bulk endpoint. **Dependencies:** PLAT-M21 frozen selections are optional; record-id lists suffice. ENR-M2 must be accepted. **Decisions:** ENR-D11 (refresh default and retention). ### 5. ENR-M4: Employment change, watch feeds, suppression self-service (user-facing; ≈ 14 to 18) This milestone completes the employment-change workflow and gives members a way to keep enrichment from changing their records. Watch feeds remain conditional on provider terms. **Scope.** - Implement the full employment-change path through CRM's `set_primary_organization` and `match_or_create` (CRM-M15), including organization candidates with "create organization". - Add `enrichment.person.watch@1` with the first feed adapter: PDL changelog polling if enterprise terms permit, or none if they do not. - Hand duplicate hints to CRM candidates with origin `enrichment` (CRM-M16). - Propagate erasure to providers' deletion endpoints. - Add the member portal "keep my record as I enter it" switch through CRM's consent ledger when that ledger is built, along with the member-facing provenance sentence. - Add access-request export. **Dependencies:** CRM-M14 (explicit collection operations), CRM-M15 and CRM-M16; the consent ledger milestone CRM proposes under Marketing D-M2; and portal work on Andrew's components. **Decisions:** ENR-D7 (member transparency), ENR-D10 (who reviews employment changes). ### 6. ENR-M5: Rhythm-managed premium and HQ (backend plus HQ screens; ≈ 10 to 14) This milestone adds provider access managed by Rhythm, rather than requiring each association to supply its own key. It depends on signed commercial terms and adds the controls needed to track usage and hand it off for billing. **Scope.** - Add `platform_account` connections with per-workspace account qualifiers for a provider under signed original equipment manufacturer (OEM) terms. - Add the platform-wide rate bucket to limit calls across the platform. - Add licence-fed `recurring_credits`, or HQ-set credits until licensing v2. - Add HQ usage and override screens, provider price lists, and the `customer_app` licence class with its portal display rule. - Provide billing hand-off through usage export for invoicing until Billing exists. **Dependencies:** signed provider terms reviewed by counsel; licensing v2 direction; ENR-M3 accepted. **Decisions:** ENR-D4 (which provider, which commercial model). ### 7. ENR-M6: AI-assisted research (opt-in; ≈ 12 to 16) This milestone offers research for one record at a time. Its results always remain pending for review; bulk research is outside this milestone. **Scope.** - Add the large language model (LLM) module operation `enrichment.research` with an allow-listed search API. - Add the `research` plan step kind and `ai_research` claims with citations. - Require always-pending suggestions, output filters, notice gating and AI allowance metering. - Add the "Research this person" action on the 360 for one record at a time, never in bulk in this milestone. **Dependencies:** the LLM module's cross-service decision, which is not needed if research runs inside Platform as recommended; a search API contract; ENR-M4 accepted. **Decisions:** ENR-D9. ### 8. Order, parallelism and what is explicitly deferred The diagram shows proposed ordering, not dates. CRM work is owned elsewhere. ```mermaid gantt title Enrichment milestones (order, not dates) dateFormat X axisFormat %s section Enrichment ENR-M1 Foundation and first provider :m1, 0, 3 ENR-M2 Organizations, baseline, auto-fill :m2, after m1, 2 ENR-M3 Bulk, imports, schedules, automation :m3, after m2, 2 ENR-M4 Employment change, watch, portal :m4, after m3, 2 ENR-M5 Rhythm-managed premium, HQ :m5, after m3, 2 ENR-M6 AI research :m6, after m4, 2 section CRM (owned elsewhere) CRM-M15 Platform-facing contracts :c15, 0, 2 CRM-M14 Collections operations :c14, after c15, 1 CRM-M16 Duplicates completion :c16, after c14, 1 ``` ENR-M1 can start once its plan is approved and INTG-M2 is verified. It does not wait for CRM-M15: it applies changes through a data job and listens to the existing change-event stream. ENR-M2 waits for CRM-M15. ENR-M5 follows ENR-M4 in sequence, because R2-DEV-001 allows one milestone In Progress per application; its contract work (provider terms, price lists) can be prepared while ENR-M4 is in acceptance. Nothing is built from ENR-M2 onwards until ENR-M1 is Product Accepted, as required by R2-DEV-001 rule 4. Every milestone still needs its own approved plan before build begins. The following remain explicitly deferred: - **Clay-style orchestration:** there is no association buyer for it ([14 §4](#15-14-alternatives-and-tradeoffs)). - **Cross-provider value merging:** unsafe. - **Address hygiene:** it needs a different provider class, auto-apply semantics and USPS cadence. It should have its own proposal on this framework. - **Certifications from credentialing bodies:** there is no provider; these require direct integrations. - **A customer-specific AI-trained field inference:** forbidden by licences and principle. - **Any cross-tenant data product.** ### 9. Testing and operational requirements Acceptance must test both the staff journeys and the safeguards: whether the right person's data is used, whether suppression prevents sending it, and whether spending stays within the reserved amount. - **Unit (L1):** test the pure `decideSuggestions`, the identity acceptance rule, the plan builder, the licence policy function and the allowance arithmetic. Include persistence-layout and naming guards, plus the architecture check. - **Integration (L2):** include persistence specs per concept; the gateway round trip with the Dummy in stub mode for matched, no_match, ambiguous, rate_limited, provider_unavailable and unknown; merge participant plan and apply; the workspace reset participant; the retention sweep; and row-level security (RLS) session-state checks under `DATABASE_POOL_MAX=1`. - **End-to-end (L3):** run the journeys in [06 §8](#7-06-core-workflows-and-use-cases) on `acme-acceptance-main`, with a seeded provider sandbox where terms allow and the Dummy otherwise. Include a cost-assertion test that a run never settles more than it reserved, and a suppression test that a suppressed subject is never sent. - **Conformance for adapters:** test the vocabulary-only rule, likelihood normalisation, cost honesty, free health probes, tier-3 privacy checks and `miss_is_free` truthfulness against the provider's price list. - **Operations:** alarm on dead-letter queue (DLQ) depth, apply failures and budget refusals. Provide a retired-provider runbook covering the retention sweep and rescheduling subjects, and a provider deletion-notice runbook. Provide dashboards for match rate and cost per applied suggestion per association. Apply the `date` rule to every time in records. - **Entity capabilities (R2-ARCH-001):** enrichment adds no custom fields to any entity. It maps provider attributes onto existing contact and organization custom field definitions chosen by the association. The 7a table in each plan will state this. ### 10. Decisions requiring Andrew's approval The package's twelve decisions, ENR-D1 to ENR-D12, are written out in [15](#16-15-open-architectural-decisions) for Andrew to approve or overturn. This document proposes an implementation sequence; it does not decide those questions or authorize build. ENR-M1 needs ENR-D1, D2, D3, D4, D6 and D8 before its plan can be written. ### 11. As ruled (Andrew, 2026-10-22 about 11:50 ET) - **ENR-D1 (inside CRM).** ENR-M1's module, tables and routes live in services-crm; the apply step is a local CRM write, so the "assisted apply" for employer changes and the data-job interim are replaced by CRM's own `set_primary_organization` and field writes in the same transaction; the CRM-M15 dependency on published `created` events is removed (CRM's internal change events raise the trigger); milestone records live in services-crm. CRM-M14's explicit collection operations are still needed before enrichment writes e-mails, phones or addresses. Size estimate unchanged (18 to 24 agent-days). - **ENR-D6 (tier 3 from ENR-M1).** ENR-M1 adds: tier-3 adapter certification in the conformance suite (PDL declares `input.personal_email` and `tier_3_certified`), the attestation form for tier 3 (lawful basis `consent` or a documented assessment, notice date, tenant-type question), region rules that exclude EU, UK, Canadian and Australian members from tier 3 and from tier 2 until a notice date is recorded, the staff suppression entry point on the 360, and `photo` and `social` suggestions with the proxied preview (copy on acceptance stays ENR-M2 and ENR-D5). Adds roughly 3 to 4 agent-days (**Estimate**). - **ENR-D2, D3, D4, D8** as recommended: no change to the plan. - Prerequisites before the ENR-M1 plan is written: the 1,000-record match-rate test and PDL's written terms (ENR-D4), and counsel's confirmation of the tier-3 posture ([09 §8](#10-09-security-privacy-and-licensing)). ## 15. 14. Alternatives and tradeoffs ## 14. Alternatives and tradeoffs Status: proposed 2026-10-22. Labels as in the [README](#1-rhythm-2-0-data-enrichment-architecture). All judgements in this document are **Inferred** from the evidence cited in the chapters. Nothing in the enrichment proposal is decided or built; the **Decided** label below refers to an existing CRM decision. The recommendation is to build enrichment as a Platform module: use each association's provider accounts, suggest changes by default for people, and keep the source and history of each enriched value separate from CRM records. The sections below explain the alternatives, what they would buy, and which costs the recommendation accepts. The package's twelve decisions, ENR-D1 to ENR-D12, are in document 15 for approval or reversal; this document asks for no separate decision. ### 1. Where enrichment lives **Recommend a Platform module**, rather than placing enrichment inside CRM, in the integration layer alone, or in a new macroservice. The ownership analysis is in [05 §2](#6-05-service-ownership-and-integration-platform-alignment). Platform already has most of the supporting infrastructure: the gateway, ledger pattern, Attention, events and jobs. The record address `(object_id, record_id)` is already generic, so enrichment does not need to belong to CRM to address a record. CRM also has the most loaded roadmap. A new service would require a library release and two hops for everything. The accepted costs are specific: Platform has no bulk data-job lane, so fetching provider data would use a Durable Job with queued work rather than a data job. The module would also need its own application key so access can be licence-gated. ### 2. If extraction becomes necessary **Keep a later move to `services-enrichment` mechanical**, without paying the cost of a separate service now. The proposed module boundaries make that possible: - Its tables would share no foreign keys with other Platform modules. Connection ids would be plain text, as they are in the integration tables themselves. - Its only Platform calls would be the gateway's service function, Attention's SQL, the events outbox and the LLM operation, all of which are already cross-service interfaces. - Its routes would form a per-module API with its own stack. Extraction would become worth considering if Platform's stack count or deploy time became a problem, or if Rhythm decided to sell enrichment outside the association management system (AMS). Neither trigger is close. ### 3. Reuse `crm.contact.enrich@1` versus a new category **Recommend a new `enrichment` contracts category.** Reusing the existing contract would avoid a contracts category change, but it would preserve the wrong boundaries: - Provider field names would remain in the contract, which P1 forbids. - The contract would still lack confidence, licence and identifier members. - Person enrichment would remain labelled as a CRM capability, even though organizations are also enriched for Events and Membership. The new category costs one contracts release and a Platform upgrade, which every new capability requires anyway. The existing contract would remain for a district adapter. ### 4. Clay-style orchestration versus an ordered plan **Recommend an ordered plan of at most three steps, with a stop rule.** An association's database administrator needs a policy that says: “use our provider, suggest changes, stay under budget.” The useful waterfall is free, then primary, then fallback. A short plan keeps that policy screen readable. Clay's model is the state of the art for sales operations: a per-column waterfall over a hundred providers, with formulas, validation steps and AI agents. It is also a workspace that a specialist runs. Rhythm should not require that operating model for association enrichment. The simpler plan gives up three capabilities: - **Per-field provider routing.** This could be added later as per-group order without a contract change. - **Cost-optimising provider search.** This is not wanted while there is one paid provider. - **Cross-provider reconciliation.** This is deliberately excluded. Disagreeing claims would be shown, not averaged. If a customer needs Clay, the recommendation is for Clay to feed Rhythm through the data-job import, not for Rhythm to become Clay. ### 5. Automatic application versus suggestions **Recommend suggestions by default for people, with limited automatic filling controlled by field group (ENR-D3, [15](#16-15-open-architectural-decisions)).** This lets an association choose where it is comfortable accepting provider data without review. HubSpot, Attio and Twenty automatically fill empty fields from a confident match, which makes enrichment feel effortless. Microsoft offers suggest-only as an admin choice. That is what a cautious association will want for information about a person. The proposed policy has firm limits: - Every person group defaults to `suggest`. - Only organization brand and identifiers default to `auto_fill`. - An existing value is never overwritten automatically, under any policy. - Employment changes and AI claims are never automatic. The cost is reviewer effort. The mitigation is bulk acceptance by group with an estimate, plus Attention batching per record. ### 6. Provenance as a Platform table versus columns on CRM records **Recommend a Platform provenance table**, which records where a value came from and can cover every entity and every custom field. It would be keyed by `(object_id, record_id, field_key)` and kept accurate by subscribing to the record owner's change events. Adding columns to `crm.contacts`, such as `job_title_source` and `job_title_verified_at`, would make provenance queryable in the contacts dataset without joins. But it would also spread enrichment into CRM's schema, forms, merges and imports. The same work would be needed for organizations and every future enriched entity, while custom-field values would still lack provenance. The Platform table keeps this information out of default contact queries. The accepted cost is one extra primary-key read for a badge on the 360 record view. Finding “contacts whose job title came from a provider” would require a two-dataset query rather than a filter. ### 7. Apply through data jobs versus a service function **Recommend keeping both paths, introduced in stages:** the data job for bulk human decisions, and the service function for policy decisions. ENR-M1 would apply changes through the CRM data job. It already exists, is versioned, writes receipts, is reversible for scalar fields and needs no CRM change. It also fits human review because a person is always the submitter. Its limits matter: it cannot apply collections or affiliations, and it acts with the reviewer's authority. A fully automatic fill therefore cannot use it. ENR-M2 would add service-function application once CRM-M15 ships `apply_enrichment` and widens the revision sources. The data-job path would remain for bulk human decisions. ### 8. Platform-wide versus per-tenant provider accounts **Recommend accounts attributable to each association**, whether BYOK, meaning bring your own key, or `platform_account` with a qualifier. This keeps provider terms, rate limits and usage attributable to one association, as both the licences and the privacy analysis require. A single pooled Rhythm account would be simpler to operate and cheaper per unit. But the licences bar that arrangement without an OEM addendum, and it would make Rhythm look like a data broker. The proposed design would support a Rhythm key only with a per-tenant qualifier and a platform-wide rate bucket. ### 9. Storing photos versus linking **Recommend previewing provider photos without storing them, then importing only an accepted photo whose licence permits a copy.** This reconciles the asset rule with the privacy research. Rhythm 1.0 stored a pointer to the vendor's CDN. The 2.0 asset rule imports external images into Rhythm's own storage before the save (**Verified**: [r2 §9](#19-evidence-r2-crm-rhythm-2-0-crm-as-the-target-of-data-enrichment)). The privacy research recommends link-and-proxy for third-party photos. Under the proposal: - No provider photo would be stored on receipt; staff would see only a proxied preview. - On acceptance, and only under a licence class that permits a copy, the image would enter the asset pipeline as the contact's own photo, with provenance and an expiry. - Gravatar's self-published images would be re-probed on a 180-day cycle. If counsel finds that even on-acceptance copies are not permitted for a provider, that provider's photos would remain preview-only (ENR-D5). ### 10. A tenant-defined custom object for suggestions **Recommend first-class tables for suggestions.** A customer-defined custom object would avoid a deploy and provide an automatic dataset, but it would not support the full review and application process. Suggestions need licence classes, expiry, participation in record merges, apply orchestration and a review screen. Custom objects do not have those capabilities (**Verified**: [r4 §8](#21-r4-data-operations-data-job-engine-and-the-query-layer-evidence-for-rhythm-2-0-data-enrichment)). ### 11. Rules-based identity acceptance versus a learned matcher **Recommend rules-based identity acceptance**, so a reviewer can be told why a match was accepted and the rule can be versioned. CRM's duplicate detection is already decided to be rules-based, explainable and versioned, with no machine learning (**Decided**: [r2 §6.1](#19-evidence-r2-crm-rhythm-2-0-crm-as-the-target-of-data-enrichment)). The proposed enrichment acceptance rule follows that decision for the same reasons. A learned matcher would require labelled association data that Rhythm does not have. It would also be hard to explain to a member who asks why their record changed. ### 12. Building a Rhythm data asset **Rhythm should never accumulate provider answers across customers into a dataset of its own.** The brief did not ask about this alternative, but the market evidence gives a clear answer. Every licence read forbids it. It would make Rhythm a data broker, and HubSpot's reversal shows how customers react. The proposed design would prevent this from happening by accident: the cache key includes the tenant, claims carry the tenant, and no cross-tenant read path exists. ## 16. 15. Open architectural decisions ## 15. Open architectural decisions The recommendation is to build enrichment as a Platform module, start with customer-owned provider accounts, and require review before applying information about people. Baseline features would be available to every workspace; costly features would be licensed separately. Status: proposed 2026-10-22. Every decision below is **open** until Andrew rules. These are proposals, not built enrichment capabilities. Labels are as defined in the [README](#1-rhythm-2-0-data-enrichment-architecture). Enrichment means obtaining suggested information, such as a member's job title or employer, from an external source and letting staff review it. ENR-M1 is the first proposed delivery milestone. Its plan cannot be written until ENR-D1, D2, D3, D4, D6 and D8 are ruled. Each decision below states the recommendation, what it governs, why it matters now, and the alternatives. ### Rulings (Andrew, 2026-10-22 about 11:50 ET) | # | Ruling | Effect | | ------ | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | ENR-D1 | **Inside CRM** (option b), overturning the recommendation | the module is `services-crm/modules/enrichment`, schema `crm`, tables `crm.enrichment_*`; consequences in [05 §8](#6-05-service-ownership-and-integration-platform-alignment) | | ENR-D2 | New `enrichment` category, as recommended | unchanged | | ENR-D3 | Suggest for people, auto-fill organization brand and identifiers, never overwrite, as recommended | unchanged | | ENR-D4 | Bring your own key first, PDL first, Rhythm-managed after terms, baseline included, as recommended | the match-rate test and PDL's written terms remain prerequisites | | ENR-D6 | **Offer tier 3 from ENR-M1 behind attestation** (option b), overturning the recommendation | ENR-M1 gains adapter tier-3 certification, the attestation form, region rules and the suppression entry point; EU, UK, Canadian and Australian members stay excluded until the notice date is recorded; [13 §2](#14-13-phased-implementation-milestones) | | ENR-D8 | Own application key, baseline for every workspace, premium licensed, as recommended | CRM registers the application `enrichment` and publishes its `enrichment.*` keys beside `crm.*` | ENR-D5, D7, D9, D10, D11 and D12 remain open; they gate later milestones. ### ENR-D1. Where enrichment lives **Recommendation.** (a): a Platform module `enrichment` with its own application key. It would sit beside the capabilities it reuses, need no new service identity, and address records generically rather than only CRM records. It could be extracted later if Platform grows too large. **What it governs.** Which service owns the enrichment tables, the API used by the screens, coordination of provider calls, and the enrichment budget. This includes both obtaining suggested job titles or employers and supporting staff review. Today nothing fills these fields, and no service owns this responsibility. **Why now.** Every table name, permission key and route in this proposal depends on ownership. The first milestone cannot be planned without it. **Options.** - (a) A Platform module `enrichment` with its own application key. - (b) Inside services-crm. - (c) Only a contract and adapters, with each application implementing its own approach. - (d) A new macroservice `services-enrichment`. The tradeoffs are in [05 §2](#6-05-service-ownership-and-integration-platform-alignment) and [14 §1](#15-14-alternatives-and-tradeoffs). ### ENR-D2. The provider contract vocabulary **Recommendation.** (a): a new `enrichment` category using Rhythm's own fixed list of fields, or “claim vocabulary.” This is the only option that makes providers interchangeable, keeps screens provider-neutral, and carries licence information with each value. It costs one contracts release, which any new capability would require. **What it governs.** How a provider expresses its answer to Rhythm. Each provider could map its results onto Rhythm's claim vocabulary, or Rhythm could accept the provider's own field names unchanged. Today the one existing contract passes provider field names through, and nothing implements it. **Why now.** The contract is published in the shared contracts package, and adapters compile against it. Changing it later would require a major version for every adapter. **Options.** - (a) A new `enrichment` category with Rhythm's claim vocabulary, four capabilities, discovery based on supported features, and a licence class on the provider manifest. - (b) Extend `crm.contact.enrich@1` with new members and keep provider field names. ### ENR-D3. The default apply policy **Recommendation.** (a): suggest all information about people; automatically fill only an organization's missing brand and identifiers; never automatically overwrite an existing value. Employment changes and AI claims would always require review. Members are individuals, and associations are cautious about their records. A wrong logo is harmless in a way that a wrong title on a member's record is not. A workspace could choose (b) in the Rules tab instead. **What it governs.** What happens to a provider's value when the association has not configured a policy. For example: does an empty job title get filled automatically, does a different job title become a suggestion, and does an organization's missing logo appear without anyone clicking? Today there is nothing to configure. **Why now.** These defaults determine what a customer sees in the first week and what enters the reviewer's queue. They are seeded in ENR-M1. **Options.** - (a) Suggest everything about a person. Automatically fill only an organization's brand and identifiers. Never overwrite an existing value automatically. Always review employment changes and AI claims. - (b) Automatically fill every empty field from a confident match, and suggest only when the value conflicts with an existing value. This is the HubSpot and Attio default. - (c) Suggest everything, including organization logos. ### ENR-D4. The commercial model and the first provider **Recommendation.** (a): start with bring your own key and PDL as the first adapter. Add Rhythm-managed credits only in ENR-M5, after written OEM terms with PDL or FullContact permit Rhythm to resell the service. Include baseline sources for logos, avatars, identifiers and verification for everyone. This keeps each licence between the association and the provider until Rhythm has resale terms. It also lets ENR-M1 ship on a free developer account. PDL is the one provider that is API-accessible, documents personal-e-mail matching with a score, and publishes an entry price. The provider choice remains conditional: before fixing it, complete the 1,000-record match-rate test and obtain PDL's written terms ([03 §6](#4-03-enrichment-provider-comparison)). **What it governs.** Who holds the provider account and pays for premium person data: the association using its own key, or Rhythm reselling credits under an OEM agreement. It also determines which provider gets the first adapter. Today Rhythm has no provider agreement. Rhythm 1.0 used one global FullContact key with no budget. **Why now.** The first adapter is ENR-M1's largest piece of work. A Rhythm-managed offering needs signed terms before it can be sold. **Options.** - (a) Bring your own key first, with PDL as the first adapter. Rhythm-managed credits only in ENR-M5 after written OEM terms with PDL or FullContact. Baseline sources, including logos, avatars, identifiers and verification, included for everyone. - (b) Rhythm-managed from the start, with a FullContact or PDL enterprise agreement negotiated before ENR-M1. - (c) Bring your own key only, with no Rhythm-managed tier ever. ### ENR-D5. Photos **Recommendation.** (a): show a preview when a photo arrives, then copy it through Rhythm's asset pipeline only after acceptance and only where the licence class allows. Re-probe Gravatar images every 180 days. Do not perform face processing anywhere. This recommendation is subject to counsel's confirmation for each provider ([09 §8](#10-09-security-privacy-and-licensing)). Option (b), preview and link only, is the fallback for any provider whose terms forbid copying. **What it governs.** Whether a provider's photo can become the member's photo in Rhythm, and how. It could remain a preview, be copied into Rhythm's storage on acceptance where licensed, or be linked from the provider's server as Rhythm 1.0 did. Today contact photos are uploaded or imported by URL into Rhythm's storage. **Why now.** This determines whether ENR-M2 needs to change the asset pipeline and what the suggestion card shows. **Options.** - (a) Preview-only on receipt; copy through the asset pipeline on acceptance where the licence class allows; Gravatar images re-probed every 180 days; no face processing anywhere. - (b) Preview and link only, never copy, for every source. - (c) Copy on receipt as 1.0 effectively did. ### ENR-D6. Whether tier 3 (personal identity data) is offered in the first year **Recommendation.** (a): include tier 3 in the policy model, but do not offer it until ENR-M4 delivers the member control and counsel confirms the posture. Most of the value for associations is in tier 1 and 2; tier 3 concentrates the privacy exposure (**Market**: [w4 §1](#26-w4-evidence-report-privacy-law-and-data-licensing-constraints-on-rhythm-2-0-data-enrichment)). **What it governs.** Whether an association can ask a provider for a member's photo, social profiles, personal phone or address using the member's personal e-mail as the lookup key. This is tier 3. Tier 2, job title, employer and business profile looked up by work e-mail or name plus employer, would be offered either way. Today none of this exists. **Why now.** Tier 3 needs adapter certification, consent-grade attestations, region rules and a member-facing control before it is safe. Deciding now keeps it explicitly outside ENR-M1's scope. **Options.** - (a) Declare tier 3 in the policy model, but do not offer it until ENR-M4 delivers the member control and counsel has confirmed the posture. - (b) Offer tier 3 from ENR-M1 behind an attestation. - (c) Never offer tier 3. ### ENR-D7. Member transparency and control **Recommendation.** (a): give members a notice and a “keep my record as I enter it” switch in the portal in ENR-M4. Allow staff to record suppression from ENR-M1, and include enrichment history in access-request exports. This puts into practice the transparency that the lawful-basis analysis depends on without making enrichment useless. An association could still require member opt-in under (c) by choosing `lawful_basis: consent`. **What it governs.** What members see and can do about enrichment of their own records: a sentence explaining that some details came from partner sources, a switch to keep the record as they entered it, and the ability to correct or delete information. Today the portal profile pages are not built in CRM. **Why now.** The notice and switch shape both the association's privacy notice and the consent-ledger milestone CRM will propose. Both need this decision. **Options.** - (a) The sentence and switch in the portal (ENR-M4), staff-recorded suppression from ENR-M1, and access-request export of enrichment history. - (b) Staff-only controls, with no member-facing acknowledgement. - (c) Member opt-in before any tier-2 enrichment. ### ENR-D8. Licensing and packaging **Recommendation.** (a): register application `enrichment`, grant its baseline to every workspace, and license the premium features separately. This makes the capability visible on day one while keeping the costly parts priced. It is the only way to gate a Platform-hosted feature (**Verified**: [r3 §2.2](#20-r3-platform-core-evidence-for-the-data-enrichment-architecture)). **What it governs.** Whether enrichment is a licensed application with a baseline for everyone and premium features behind a licence and credits, or part of CRM for every customer. Today Platform capabilities cannot be licence-gated at all. **Why now.** The application key is registered in ENR-M1. The HQ default grant determines who sees the screens. **Options.** - (a) Application `enrichment`: baseline logos, avatars, identifiers, verification allowance, and review tooling with a customer-provided key granted to every workspace. Premium Rhythm-managed credits, scheduled refresh, employment watch and AI research licensed separately. - (b) Everything included in CRM's licence. - (c) Everything premium. ### ENR-D9. AI-assisted research **Recommendation.** (a): retain the `research` step kind in the plan model and put ENR-M6 on the roadmap. Build it as an opt-in, one-record-at-a-time feature inside Platform's LLM module, using a search API and always requiring review. The evidence indicates that it is cheap and useful as a fallback for people no provider knows, but dangerous without citations, review and limits on the personal information collected ([09 §5](#10-09-security-privacy-and-licensing)). **What it governs.** Whether a reviewer can ask a large language model (LLM) with web search to find a member's professional details in public sources and return a cited suggestion. Today the LLM module serves one operation inside Platform and has no web tool. **Why now.** Adding a `research` step kind to the plan model is cheap now and costly later. This decision also determines whether ENR-M6 belongs on the roadmap. **Options.** - (a) Keep the step kind in the model; build ENR-M6 as an opt-in, one-record-at-a-time, always-reviewed feature inside Platform's LLM module with a search API. - (b) Drop it from the roadmap. - (c) Build it first, before any provider. ### ENR-D10. Employment changes: who reviews, and whether ever automatic **Recommendation.** (a): route employment-change proposals to the `crm.data_quality_review` responsibility by default, allow each workspace to change the reviewer, and never apply the change automatically. An employment change ends an affiliation, removes automatic roles and changes an organization's compliance. Routing to the member's primary organization's administrator, option (c), is a later refinement once responsibilities per organization exist. **What it governs.** Who reviews a provider's claim that a member has a new employer: the data-quality reviewer, or the member's chapter or organization administrator. It also determines whether every policy must require a person to approve the change. Today affiliations are edited by staff only. **Why now.** The reviewer responsibility is seeded in ENR-M1, and the Attention audience depends on it. **Options.** - (a) The `crm.data_quality_review` responsibility by default, changeable per workspace; never automatic. - (b) An automatic option for very-high-likelihood matches with an exact organization match. - (c) Route by the member's primary organization's administrator. ### ENR-D11. Refresh cadence and retention defaults **Recommendation.** (a): retain claims for 24 months or the licence's term, whichever is shorter; retain pending suggestions for 90 days; leave scheduled refresh off by default and run it annually when enabled. This matches the licences' deletion windows, bounds spend, and keeps “last verified” meaningful. **What it governs.** How long Rhythm keeps what a provider said, called a claim, and what it proposed for application, called a suggestion. It also determines whether enriched records are re-checked on a schedule by default. Today nothing is stored. **Why now.** Retention is a privacy commitment that the notice must state. Refresh frequency determines ongoing cost. **Options.** - (a) Claims 24 months or the licence's term, whichever is shorter; pending suggestions 90 days; scheduled refresh off by default, annual when on. - (b) Claims for the life of the record. - (c) Refresh on by default. ### ENR-D12. The first adapters **Recommendation.** (a): PDL, Logo.dev, Gravatar, and one of Bouncer, Emailable or ZeroBounce. Choose the e-mail verifier after reading its retention terms. **What it governs.** Which four adapters ENR-M1 and ENR-M2 build: the person provider chosen in ENR-D4, the logo source, the avatar probe and the e-mail verifier. **Why now.** Each adapter requires a repository and a conformance run to check that it meets the contract. **Options.** - (a) PDL, Logo.dev, Gravatar, one of Bouncer, Emailable or ZeroBounce chosen on retention terms. - (b) FullContact in place of PDL. - (c) Brandfetch in place of Logo.dev under a Custom agreement. ### Summary table | # | Decision | Recommendation | Needed by | | ------- | ----------------------------------- | --------------------------------------------------------------------------------------- | ------------------- | | ENR-D1 | Where enrichment lives | Platform module `enrichment` | ENR-M1 plan | | ENR-D2 | Contract vocabulary | New `enrichment` category, Rhythm claims | ENR-M1 plan | | ENR-D3 | Default apply policy | Suggest for people; auto-fill organization brand and identifiers; never overwrite | ENR-M1 plan | | ENR-D4 | Commercial model and first provider | Bring your own key; PDL first; Rhythm-managed after terms | ENR-M1 plan | | ENR-D5 | Photos | Preview, then copy on acceptance where licensed; no face processing or other biometrics | ENR-M2 plan | | ENR-D6 | Tier 3 in year one | Declared, not offered until ENR-M4 and counsel | ENR-M1 plan | | ENR-D7 | Member transparency | Sentence and switch in the portal; staff suppression from M1 | ENR-M4 plan | | ENR-D8 | Licensing | Application key; baseline for all; premium licensed | ENR-M1 plan | | ENR-D9 | AI research | Keep the step kind; ENR-M6 opt-in and reviewed | ENR-M6 plan | | ENR-D10 | Employment changes | Data-quality reviewer; never automatic | ENR-M1 plan | | ENR-D11 | Refresh and retention | 24 months; 90 days; refresh off by default | ENR-M3 plan | | ENR-D12 | First adapters | PDL, Logo.dev, Gravatar, one verifier | ENR-M1 and M2 plans | ## 17. 16. Sources, research dates and how this document was produced ## 16. Sources, research dates and how this document was produced Status: proposed 2026-10-22. Labels as in the [README](#1-rhythm-2-0-data-enrichment-architecture). This chapter records the evidence behind the Data Enrichment proposal, the limits of that evidence, and how the package was produced. The package remains a proposal: none of its proposed decisions has been approved, and nothing has been built. ### 1. The evidence reports Every **Verified** and **Market** label in chapters 01 to 15 points to one of the ten reports below. Those reports contain the supporting file, line and URL citations. They were written on 2026-10-22 through read-only investigations and were not changed by the Humanize editorial pass. | Report | Scope | Repositories or sources, at the revision read | | ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [r1 Integration framework](#18-r1-the-integration-platform-as-built-evidence-for-a-data-enrichment-design) | the Integration Platform as built: catalogue, connections, credentials, bindings, gateway, executions, webhooks, SDK, registration, metering, milestone state, console screens | services-platform `43478ffda`; macroservice-libraries `49728e6` (integration-contracts 0.7.1, read with `git archive`); integrations-dummy `d387998`; react-console `45a47ab6b` | | [r2 CRM](#19-evidence-r2-crm-rhythm-2-0-crm-as-the-target-of-data-enrichment) | contacts, organizations, lists, affiliations, types, custom fields, duplicates and merge, revisions, data operations, assets, write contracts, events, queries, consent, portal | services-crm `e818ee48` (extracted; the local checkout was 354 commits behind); RHYTHM-CRM-ARCH-001 at `d63f10b5d`; households at `67f223139`; services-marketing `b26e4fa`; rhythm-web `776aede` | | [r3 Platform core](#20-r3-platform-core-evidence-for-the-data-enrichment-architecture) | execution context, security, tenant configuration, licensing, audit, business events, jobs, workflow actions, the LLM module, Attention, assets, record addressing, snapshots, conventions, observability, the milestone standard | services-platform `43478ffda`; macroservice-libraries `49728e6` | | [r4 Data operations and queries](#21-r4-data-operations-data-job-engine-and-the-query-layer-evidence-for-rhythm-2-0-data-enrichment) | the Data Job Engine, upsert matching, uploads and runs, the semantic layer, metadata and custom fields, Quick Search, custom objects, merge participation, exports | services-platform `43478ffda`; macroservice-libraries `49728e6ba` (data-jobs 0.12.1); services-crm `e818ee48` | | [r5 Rhythm 1.0 FullContact](#22-r5-rhythm-1-0-fullcontact-and-google-civic-enrichment-and-the-customer-360-precedent) | the 1.0 engines, triggers, fields, config, cache, photos, UI, Google Civic, and the customer-360 internal precedent | `~/rhythm/rolodex-organizations`, `rolodex-contacts`, `rolodex-ngx-console*`, `customer-360-commitments-dashboard` (local checkouts) | | [w1 AMS platforms](#23-w1-evidence-how-association-management-systems-and-the-nearest-nonprofit-crms-handle-data-enrichment-and-data-quality) | iMIS, GrowthZone, MemberSuite, netFORUM, Fonteva, Nimble AMS, Personify, MemberClicks, YourMembership, Wild Apricot, Aptify, Impexium, Novi, Glue Up, Raklet, Higher Logic, Tessitura, Blackbaud, Bloomerang, Neon One, Salesforce platform, MomentiveIQ | vendor documentation and third-party pages, accessed 2026-10-22; §6 lists every URL and every unfetchable page | | [w2 Enterprise CRM and ERP](#24-w2-evidence-how-enterprise-crm-erp-platforms-do-data-enrichment) | Salesforce, HubSpot, NetSuite, Dynamics 365, Zoho, Attio, Clay, Pipedrive, Freshsales, Copper, folk, Twenty | vendor documentation, release notes, pricing pages, Twenty's source on GitHub `main`, accessed 2026-10-22; §9 lists URLs and gaps | | [w3 Enrichment providers](#25-w3-evidence-enrichment-provider-landscape-for-an-association-management-system) | PDL, FullContact, Apollo, ZoomInfo, Clearbit/Breeze, Clay, Cognism, Lusha, RocketReach, Hunter and about twenty more; appenders; verifiers; LinkedIn; organization data; photos; AI research; economics | API documentation, pricing and terms pages, accessed 2026-10-22; §10 lists URLs and unfetchable pages | | [w4 Privacy and licensing](#26-w4-evidence-report-privacy-law-and-data-licensing-constraints-on-rhythm-2-0-data-enrichment) | GDPR and UK GDPR, US state law and data-broker statutes, Canada and Australia, photos, provider licences, scraping cases, appenders, AI research, retention and erasure, cross-tenant caching, contractual patterns | statutes, regulator guidance (several read in full as PDFs), provider terms, law-firm analyses, accessed 2026-10-22; §15 and §16 list sources and gaps | | [w5 Supplementary licence checks](#27-w5-supplementary-provider-checks-read-directly-2026-10-22-11-00-to-11-30-et) | PDL pricing and changelog, FullContact inputs and Services Agreement, Clearbit status, Logo.dev pricing and storage terms, Gravatar API and guidelines | read directly on 2026-10-22, 11:00 to 11:30 ET; lists the pages that could not be read | ### 2. What could not be verified, and what it leaves open These gaps limit what the proposal can conclude. In particular, provider terms still need confirmation, and the assumed match rate still needs testing. - **People Data Labs' own terms of service** are script-rendered and could not be read by any tool used. The evidence for its resale and retention restrictions comes from a reseller's reproduction. Obtaining the terms in writing is a prerequisite of ENR-D4. - **FullContact's current Services Agreement** returned 404. The previous version was read, but the current agreement may differ. - **Logo.dev's full terms** were not read (its pricing page states the storage rights quoted in w5), and the **e-mail verifiers' retention terms** were not read; only pricing pages were. - **No provider publishes a match rate for personal-e-mail-only input**. The 40 percent used in [12](#13-12-commercial-model-and-economics) is an assumption to be replaced by the 1,000-record test. - **Several AMS and CRM help centres** blocked fetching. Claims about them are labelled Reported in w1 and w2. - **Regulator positions** on B2B enrichment are contested. The Italian Garante's Lusha and OpenAI decisions are shown as removed after Court of Rome rulings, and the CNIL's KASPR fine is reported at two amounts. - **The Art. 9 inference** for religious, political and trade-union bodies is the researcher's interpretation, not settled law. - **Google's Civic Information API turndown** comes from general knowledge in r5, not from the repositories. ### 3. Labels The labels distinguish what was observed, what is inferred, what is already required, and what this proposal recommends. A **Recommended** design is not a decision; **Decided** / **Ruled** refers to an existing, dated decision by Andrew. | Label | Meaning | | -------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | | **Verified** | read in code, DDL, a test, a repository document, or a vendor's primary page, at the cited place and revision, on 2026-10-22 | | **Planned** | in a Rhythm proposal or milestone record, not built | | **Decided** / **Ruled** | a dated decision by Andrew, cited where it is recorded | | **Market** | an external fact with a dated URL in an evidence report (itself labelled Verified, Reported or Marketing there) | | **Published** / **Derived** / **Reported** / **Assumed** | economics: a vendor's own number; arithmetic on it; a third party's; a planning placeholder | | **Inferred** | a conclusion from verified facts | | **Recommended** | this proposal's design, open for Andrew to overturn | | **Required** | the brief or a standing Rhythm 2.0 rule | | **Estimate** | a size or cost this proposal worked out, with its basis | | **Absent** / **Unavailable** | searched for and not found; or a page that could not be read | ### 4. How this document was produced The work consisted of research, proposal writing and document production, not implementation. 1. Nine read-only investigations ran in parallel on 2026-10-22 from 10:20 ET: five covered the Rhythm repositories (r1 to r5), and four covered the market and the law (w1 to w4); the architect then read provider pages the researchers could not reach (w5) (w1 to w4). Each produced one evidence report with per-claim labels and citations. The architect then read provider pages the researchers could not reach (w5). 2. The architecture chapters were written against those reports. Every claim about Rhythm's code cites a report section that supplies a path and line at the revisions listed above. Every market claim cites a report that supplies a URL and access date. 3. The prose chapters went through the Humanize pass, an editorial pass that rewrites prose while preserving fixed elements. They were then checked fact by fact against the evidence reports. Tables, DDL, TypeScript, diagrams, identifiers, numbers, dates and links were kept verbatim. The checker in `tooling/check-results.py` counted those elements before and after editing. 4. The single-page artifact beside this README was built by `tooling/build-full-artifact.py` from the humanized chapters and the unedited evidence reports. 5. Nothing was built: no application code, no migration, no console or portal change, and no change to any other proposal or milestone record. No new file was created outside this package. ### 5. Standing decisions and documents this proposal relies on These existing decisions and documents constrain the proposal; they do not approve its proposed design. - R2-DEV-001 (milestones). - R2-ARCH-001 (custom fields opt-in per entity). - R2-ARCH-002 (submission-compatible operations). - `docs/standards/record-write-contracts.md`. - `docs/record-merge-participant.md`. - `docs/queued-work.md`. - `docs/persistence.md`. - `docs/indexing.md`. - `docs/row-level-security.md`. - `docs/naming-conventions.md`. - `docs/proposals/integration-platform/` (chapters 10, 12, 13, 14, 21, 22). - RHYTHM-CRM-ARCH-001 (chapters 04, 05, 06, 07, 10, 12, 15). - `services-crm/docs/affiliations/README.md`. - `services-crm/docs/merging/duplicate-detection.md`. - `services-crm/docs/debt/image-files-and-enrichment.md`. - Marketing decision D-M2 (CRM owns the consent ledger, 2026-10-20). - CRM-ACC-0064 (`.query` and `.read`, 2026-10-21). - The UI-screens-via-v0 rule (2026-10-21). - The portal-last rule. ## 18. R1. The Integration Platform as built: evidence for a Data Enrichment design ## R1. The Integration Platform as built: evidence for a Data Enrichment design Evidence report for the Rhythm 2.0 Data Enrichment architecture proposal. Read-only investigation, 2026-10-22. ### What was read, at which revision | Repository | Revision read | How | | ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `services-platform` | `43478ffda` (origin/master) | the `enrichment-architecture` worktree, which is at `43478ffda` and has no diff from origin/master under `modules/integrations`. The primary checkout `~/rhythm2/services-platform` is at `7d2068d21`, an ancestor of origin/master, so it was not used. | | `macroservice-libraries` | `49728e6` | `git archive 49728e6 packages/integration-*` into a scratch directory. The primary checkout is at `91e6562`, 165 commits **behind** `49728e6`; its `integration-contracts` is 0.5.0 and lacks most of what is quoted here. Anyone re-reading must use `49728e6` or later. | | `integrations-dummy` (adapter) | `d387998` (branch `ip-real-publish-land`; origin/master `045be59`) | the local worktree `~/rhythm2/worktrees/adapters/integrations-dummy` | | `react-console` | `45a47ab6b` | primary checkout, read only | | Consumers (`services-accounting` `6dcab585`, `services-crm` `eac3df30`, `services-commerce` `bd62910a`) | local checkouts | grep only | Citation forms: `services-platform::` (at `43478ffda`), `libs@49728e6::`, `integrations-dummy::`, `react-console::`. Labels: **Verified** = read in code, tests or DDL at the revision above; **Planned** = in a proposal chapter but not built; **Inferred** = my reasoning from what was read; **Absent** = searched for and not found. --- ### Summary for an enrichment architect 1. **A `crm.contact.enrich@1` capability contract already exists, but it is address enrichment only.** Request `{ address, fields_requested[] }`, response `{ matched, fields: { : scalar | string[] } }`; `sync`, idempotency `none`, `personal_data: true`, no features, no lookup counterpart. It was written to replace 1.0's Google Civic district lookup, not for person or company enrichment by e-mail, name or domain. **Verified** `libs@49728e6:packages/integration-contracts/schemas/capabilities/crm/crm.contact.enrich@1.json:1-135`, `libs@49728e6:packages/integration-contracts/src/registry.ts:1082-1100`. 2. **Nothing implements or calls it.** No adapter manifest declares it (the Dummy does not), CRM has no caller, and Platform's `capability_definitions` mirror gets a row only when an adapter registers that major. **Verified** (Dummy manifest; grep of `services-crm`); **Absent** (any `crm.contact.enrich` handler or invoker). 3. **Capability contracts are a versioned npm package, not database configuration.** `@rhythmsoftware/integration-contracts` 0.7.1 holds JSON Schema 2020-12 documents (one per `key@major`), TypeScript types generated from them, and a code registry (mode, timeout, idempotency, sensitivity, personal data, permission, features, error codes, lookup). A new capability or a new major means a contracts release, then a Platform upgrade to that release, because Platform refuses manifests compiled against a newer contracts version than its own. **Verified** `libs@49728e6:packages/integration-contracts/src/registry.ts:1-65`; `services-platform:modules/integrations/services/registration/register-integration-version.ts` header, step 3; `services-platform:package.json:102` (`0.7.1`). 4. **Category and verb vocabularies are closed lists in code.** Categories include `crm` and `marketing` but no `data` or `enrichment`. Verbs include `enrich`, `lookup`, `pull`, `verify` and `describe`; there is **no `search` verb**. A `search`-style capability needs a contracts change. **Verified** `libs@49728e6:packages/integration-contracts/src/naming.ts:12-27`, `:37-76`. 5. **Each provider is its own repository and deployable**: `integrations-`, four Lambda functions from one bundle (`INVOKE`, `WEBHOOK`, `EVENTS`, `SYNC`), deployed by its own pipeline and registered with Platform by `integration publish`. No Platform or consumer redeploy is needed for a new version, **but a brand-new adapter key needs a Platform change**: the registration alias admits only the deploy roles of the keys hard-coded in `REGISTERING_ADAPTER_KEYS` (`dummy`, `stripe`, `payrix`, `zoom`, `mailchimp`, `ups`). **Verified** `services-platform:modules/integrations/integrations.settings.ts:28`; `services-platform:docs/integrations/ip-registration-invoke-grant-notes.md`. 6. **Tenant model: installation, then connection, then binding.** One installation per tenant and integration; many connections (each an authenticated provider account) per installation; a binding says "for capability C at major M, at this scope, use connection K". All tenant tables carry the standard forced row-level-security policy. **Verified** (DDL, section 1). 7. **Resolution always picks exactly one connection.** Scopes are walked most specific first (`resource`, `business_unit`, `application`, `tenant`); `crm.*` capabilities may bind only at `application` and `tenant` scope with fallback `walk`. There is no fan-out to several providers, no ordered "waterfall" of providers, no merge of answers. **Verified** `libs@49728e6:packages/integration-contracts/src/binding-policies.ts:110`; `services-platform:modules/integrations/services/bindings/resolve.ts:1-31`. 8. **Rhythm-managed credentials exist (`platform_account`), but still per tenant.** Rhythm's own provider key is a platform-scoped secret `/rhythm/integrations///`; each tenant still needs its own installation and connection, and the method requires an `account_qualifier` setting on every connection. There is no platform-wide connection shared by tenants. **Verified** `libs@49728e6:.../manifest.schema.json` (`platform_account` requires `method`, `platform_secret`, `account_qualifier`); `services-platform:modules/integrations/services/credentials/credential-values.ts:193-210`. 9. **Rate limits are per connection only.** The Valkey key is `int:{}:rl::`. With a Rhythm-managed key shared by many tenants, nothing enforces the provider's account-wide quota across tenants. **Verified** `services-platform:modules/integrations/services/executions/rate-limits.ts:112`; **Inferred** (the consequence). 10. **No usage metering, quota or billing exists in the integrations module.** The only cost signal is an optional integer `cost_units` an adapter may return, stored on the execution row. No rollup, no allowance, no per-tenant budget. The closest precedent in Platform is the AI usage ledger (`modules/llm`: monthly allowance periods with reserved and settled credits, daily rollups, rate versions). **Verified** `services-platform:modules/integrations/db/integration-executions.table.ts:136-137`; `services-platform:modules/llm/db/llm-allowance-periods.table.ts:1-12`; **Absent** (`quota`, `metering`, `credits` in `modules/integrations`). 11. **The invocation path is solid and reusable as-is for a synchronous enrichment call**: typed `invoke()` from `@rhythmsoftware/integration-client`, a service call to `platform_integrations_invoke_service`, validation, authorization, resolution, execution record, circuit and rate admission, credential lease, adapter invoke, error classification into nine classes, bounded retries (sync 1 s, 4 s, 16 s inside a 25 s budget), idempotency keys, and partitioned execution history kept 13 months. **Verified** (section 4). 12. **Personal-data handling is declared but not enforced.** The registry marks `crm.contact.enrich` `personal_data: true` ("payloads are never recorded"), yet Platform's payload recording checks only `sensitive`; an adapter whose manifest sets `record_payloads: true` would have its enrichment requests and answers written to S3 for 30 days. Request and response summaries do drop personal-looking fields. **Verified** `services-platform:modules/integrations/services/executions/payload-recording.ts:31-37`; **Absent** (`personal_data` anywhere in `services-platform/modules/integrations` or `lib`). 13. **Adapters cannot read Rhythm data yet.** The data-grant callbacks (`data.query`, `data.project`) answer 501; the sync engine (WS10) is not on master. An enrichment adapter can only answer what it is sent. **Verified** `services-platform:modules/integrations/aws/api/gateway.routes.ts:155-161`; **Absent** (any sync-profile table). 14. **What is accepted versus built.** INTG-M1 (find, install, connect) is Product Accepted (2026-10-21). INTG-M2 (calls through the gateway) is In Progress. INTG-M3 (webhooks, events, deliveries), M4 (OAuth), M5 (sync), M6 (HQ operations) are Proposed, although most of M3/M4 code is already on master. INTG-M7 (Stripe) and M8 (Payrix) are In Progress. **Verified** (milestone records, section 11). 15. **FullContact is not mentioned anywhere** in the integration proposal, its evidence, the milestone records or the code. The only enrichment precedent in the documents is Google Civic (1.0, retired, Andrew 2026-10-17, Q-6). **Absent** (grep for `fullcontact`, `full contact`, `clearbit`, `zoominfo`); **Verified** `services-platform:docs/proposals/integration-platform/25-migration.md:53`. --- ### 1. Domain model: tables, keys, ids, tenancy, RLS #### 1.1 The id rule **Verified** `services-platform:modules/integrations/db/integration-ids.ts:1-16`, `:78-135`. Configuration tables (what Rhythm or a customer names) are keyed by an immutable lowercase string configuration id of at most 63 characters, checked in the database (`_id_configuration_id_check`); records (what happens: executions, events, deliveries, credentials, audit rows) are keyed by UUIDv7. Ids are composed from parent ids and natural keys: | Row | Id composition | Example | | -------------------------------------------------------------- | ----------------------------------------------------- | ------------------------------ | | integration definition | its key | `stripe` | | integration version | `_v` | `stripe_v1_2_3` | | capability definition | `_v` | `payments_authorize_v1` | | integration capability | `_` | | | deployment | `__` | | | installation | the definition's key (one per tenant and integration) | `stripe` | | binding | `_v_[_]` | `payments_authorize_v1_tenant` | | availability grant, rate limit, webhook endpoint, subscription | composed from parents | | Long ids keep 54 characters plus an 8-hex SHA-256 suffix (`integration-ids.ts:56-71`). #### 1.2 Global catalog tables (no tenant, no RLS) | Table | Holds (key columns) | Id | Cite | | ------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | | `platform.integration_definitions` | one row per adapter: `key` (= `id`), `name`, `summary`, `description_md`, `publisher`, `categories[]`, `auth_methods` (jsonb, as the manifest declares), `availability` (`public`/`restricted`/`internal`, default `internal`), `status` (`active`/`hidden`/`retired`), `current_version_id`, logo/docs/support URLs | string = key, pattern `^[a-z](?:[a-z0-9] | _(?=[a-z0-9])){2,40}$` | `integration-definitions.table.ts:1-23`, `:33-93` **Verified** | | `platform.integration_versions` | one immutable release: `version` (semver), `manifest` (jsonb, whole), `manifest_hash` (SHA-256 hex of canonical JSON), `sdk_version`, `contracts_version`, `status` (`draft`, `validated`, `published`, `deprecated`, `retired`, `rejected`), `conformance_report`, publish/deprecate/retire timestamps, `release_notes_md`. Immutable from `published` (a re-registration with another hash is 409) | composed string | `integration-versions.table.ts:1-92` **Verified** | | `platform.capability_definitions` | Platform's **mirror** of the contracts registry: `key`, `major`, `minor`, `category`, `execution_mode` (`sync`/`async`/`either`), `request_schema`, `response_schema`, `error_schema`, `features` (jsonb), `idempotency` (`required`/`optional`/`none`), `permission_key`, `sensitive`, `status`, `deprecated_at`, `sunset_at`. Written only by registration when a manifest names a major the table lacks | composed string | `capability-definitions.table.ts:1-89` **Verified** | | `platform.integration_capabilities` | join "version V implements `key@major`": `handler`, `features[]`, `constraints` (jsonb: currencies, amounts, countries), `execution_modes[]` | composed string | `integration-capabilities.table.ts:1-78` **Verified** | | `platform.integration_deployments` | where a version runs: `entrypoint` (Lambda alias ARN), `webhook_entrypoint`, `events_entrypoint`, `transport` (`lambda`; `https` later), `runtime` (`lambda`/`fargate`), `principal`, `region`, `environment` (`dev`/`prod`), `status` (`active`/`draining`/`inactive`), `health`. One `active` per version and environment. The `sync` entrypoint is validated but not stored | composed string | `integration-deployments.table.ts:1-78` **Verified** | | `platform.integration_availability_grants` | the tenants a `restricted` definition is offered to: `integration_definition_id`, `tenant_id`, `granted_by_id`, `granted_at`, `note`; at most 200 tenants per definition. **No RLS by decision**: `tenant_id` names the tenant the grant is about, not an owner | composed string | `integration-availability-grants.table.ts:1-56`; `services-platform:db/row-level-security.ts:81-82` **Verified** | | `platform.integration_event_consumers` | the Rhythm applications that consume integration events: `principal` (primary key), `function_arn`, `event_types[]` (exact, or prefix `.*`), at most 100 types | principal | `integration-event-consumers.table.ts:1-35` **Verified** | #### 1.3 Tenant tables (standard forced tenant RLS policy) Every table below states "Tenant-owned: the standard forced tenant policy" in its header. Tenant-plane foreign keys are composite `(tenant_id, parent_id)` so no row can point at another tenant's parent (`services-platform:docs/integrations/README.md`, "Decisions the chapters left open", item 3). **Verified**. | Table | Holds (key columns) | Id | Cite | | -------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | | `integration_installations` | "this tenant installed X": `integration_definition_id` (= `id`), `status` (`enabled`/`suspended`/`uninstalled`), `settings` (jsonb per the manifest's `installation_settings_schema`), `accepted_permissions` (`{ capabilities, events, data_scopes, rhythm_permissions }`), `accepted_by_id`, `accepted_at`, `suspended_reason`, `uninstalled_at`. PK `(tenant_id, id)`; reinstalling re-enables the same row | string = definition key | `integration-installations.table.ts:1-90` **Verified** | | `integration_connections` | one authenticated provider account: `installation_id`, `integration_definition_id`, `integration_version_id` (**the pin**), `name`, `external_account_ref`, `auth_method`, `account_environment` (`live`/`test`), `credential_id`, `configuration` (jsonb per `connection_settings_schema`), `verified_features` (`{ "@": { supported, features[], verified_at, notes? } }`), `status` (8 states), `health` (`{ state, checked_at, last_error, success_rate_1h, p95_ms_1h, executions_1h, metrics_at, consecutive_failures }`), `labels[]`, `business_unit_id`, disconnection columns. Only `active` and `degraded` are eligible | string, customer-chosen ids may not start `sys_` | `integration-connections.table.ts:1-205` **Verified** | | `integration_credentials` | a **pointer** to a secret: `connection_id`, `kind` (`api_key`, `oauth2_tokens`, `client_credentials`, `basic`, `custom`), `secret_ref` (Secrets Manager name), `key_version`, `scopes[]`, `expires_at`, `refresh_expires_at`, `last_refreshed_at`, `last_used_at`, `rotation_due_at`, `status` (`active`/`superseded`/`revoked`/`destroyed`), `secret_deletion_due_at`, `destroyed_at`, `metadata` (non-secret: method, token type, client id, hint, refresh failures). One `active` per connection; at most 50 rows per connection | UUIDv7 | `integration-credentials.table.ts:1-160` **Verified** | | `integration_oauth_states` | one OAuth attempt: PK `state_hash`, `connection_id`, `user_id`, `code_verifier_ref` (PKCE verifier as a secret), `requested_scopes[]`, `issued_at`, `expires_at` (10 min), `used_at` | hash | `integration-oauth-states.table.ts:1-63` **Verified** | | `integration_bindings` | routing choice: `capability_key`, `capability_major`, `scope_kind` (`resource`/`business_unit`/`application`/`tenant`), `scope_ref` (`*`, app key, BU id, `:`), `connection_id`, `required_features[]` (≤ 20), `policy` (`{ on_ineligible: 'refuse' | 'fall_through' }`), `revision`, `status` (`active`/`invalid`/`superseded`), `invalid_reason`, `note`. One `active` per (tenant, capability, major, scope_kind, scope_ref) | composed string | `integration-bindings.table.ts:1-150` **Verified** | | `integration_binding_revisions` | append-only copy of every binding change: `binding_id`, `revision`, `snapshot` (jsonb), `reason` (`created`, `replaced`, `deleted`, `invalidated`, `restored`, `disconnected`, `migrated`), `changed_by_id`, `changed_at`; ≤ 200 per binding | UUIDv7 | `integration-binding-revisions.table.ts:1-67` **Verified** | | `integration_rate_limits` | per connection, bucket and source (`manifest`/`provider_header`/`operator`): `limit_per_window`, `window_seconds`, `concurrency`, `refilled_at`; `tokens`/`concurrent_now` left null (live counts are in Valkey) | composed string | `integration-rate-limits.table.ts:1-64` **Verified** | | `integration_circuit_states` | one breaker per (deployment, connection): state `closed`/`open`/`half_open`, `opened_at`, `probe_at`, `failure_count` | | `integration-circuit-states.table.ts:1-67` **Verified** | | `integration_executions` | one logical invocation (section 4.3). **Partitioned by month on `started_at`**, 13 months online then archived as Parquet | UUIDv7, PK `(id, started_at)` | `integration-executions.table.ts:1-140` **Verified** | | `integration_execution_attempts` | one delivery to the adapter; partitioned | UUIDv7 | `integration-execution-attempts.table.ts:1-65` **Verified** | | `integration_execution_keys` | idempotency rule spanning months: PK `(tenant_id, capability_key, idempotency_key)` → `execution_id`, `request_digest` | | `integration-execution-keys.table.ts:1-32` **Verified** | | `integration_execution_responses` | full stored answer (≤ 256 KB) of a settled execution for idempotent replay; partitioned on `stored_at` | | `integration-execution-responses.table.ts:1-30` **Verified** | | `integration_execution_dispatches` | queue table for async attempts and sync retries that outgrew the caller (request ≤ 200 KB) | | `integration-execution-dispatches.table.ts:1-45` **Verified** | | `integration_work_dispatches` | queue table for `webhook_event`, `integration_event`, `delivery` work | | `integration-work-dispatches.table.ts:1-48` **Verified** | | `integration_webhook_endpoints`, `integration_webhook_events`, `integration_webhook_event_keys` | section 5 | | **Verified** | | `integration_events`, `integration_subscriptions`, `integration_deliveries`, `integration_delivery_keys` | section 5 | | **Verified** | | `integration_audit_events` | section 7 | UUIDv7 | **Verified** | #### 1.4 Job roots (no RLS, by the job convention) `integration_verification_jobs` and `integration_health_jobs` (with `_steps` and `_events` tables) are durable job roots; their `tenant_id` is the connection's; the job is read through the job scope rule. **Verified** `services-platform:db/row-level-security.ts:83-86`; `integration-verification-jobs.table.ts:1-46`; `integration-health-jobs.table.ts:1-40`. #### 1.5 The proposal's picture Chapter 09 names thirteen concepts (definition, version, capability definition, integration capability, deployment, installation, connection, credential, binding, invocation/execution, subscription, webhook endpoint, synchronization profile). All but the synchronization profile (and its `SYNC_RUN`, `EXTERNAL_ID`) exist as tables. **Verified** `services-platform:docs/proposals/integration-platform/09-domain-model.md:9-56`; **Absent** (sync profile tables on master). --- ### 2. Capability contracts, provider declarations, binding resolution #### 2.1 Where capabilities are declared **Verified.** `@rhythmsoftware/integration-contracts` 0.7.1 (`libs@49728e6:packages/integration-contracts/package.json`): - `schemas/capabilities//@.json`: JSON Schema 2020-12 documents with `$id` `https://contracts.rhythmsoftware.com/capabilities//`, `$defs/request`, `$defs/response`, `$defs/feature`, and `examples` (used by conformance and the console). 54 capability documents at this revision, including `crm/crm.contact.enrich@1.json`. - `src/generated/types.ts`: TypeScript generated from all schemas (envelope, manifest, every capability). `Capabilities[K][M]` indexes request, response, features, mode (`src/capabilities.ts:7-36`). - `src/registry.ts`: `CAPABILITY_REGISTRY`, one `CapabilityDefinition` per `key@major`, with what the schema does not say (interface quoted in the appendix; `:31-65`). Defaults from `define()`: major 1, minor 0, timeout 900 s for `async`, 20 s for `payments`, 25 s otherwise (`:146-159`, `TIMEOUTS_MS` at `:29`). - `src/naming.ts`: closed category list and verb list; key grammar two to five segments, first a category, last a verb (`:12-97`). - `src/binding-policies.ts`: for each capability, the scopes the consuming application allows and the fallback (`refuse`/`walk`) (`:14-60`, policies to `:115`). - `src/errors.ts`: nine error classes with retry rules, `COMMON_ERROR_CODES`, `ProviderError` (`:13-125`). Platform mirrors a contract into `capability_definitions` only when a registration names it (section 1.2). The proposal says the same: "Each contract is a versioned document in the `@rhythmsoftware/integration-contracts` package. At publication, it is also copied into `capability_definitions`." **Verified** `services-platform:docs/proposals/integration-platform/10-capability-contracts.md:9-11`. **Versioning** (**Verified** as code for majors/minors; the minor-stripping rule is **Planned**): identity is `key@major`; a minor may add optional fields, features, codes, events; a major is a new row and a new handler; deprecation sets `deprecated_at` and `sunset_at` ≥ 180 days later (`MINIMUM_SUNSET_DAYS`, `registry.ts:67`), after which resolution refuses `contract_retired` (`resolve.ts:21-22`). The gateway's stripping of unknown fields for an adapter on an older minor is described in `10-capability-contracts.md:84-86`; I did not find it in the gateway (**Inferred** not built; not searched exhaustively). #### 2.2 How a provider declares what it supports The adapter's **manifest** (`IntegrationManifest`, generated from `schemas/manifest.schema.json`, `$id` `https://contracts.rhythmsoftware.com/manifest/1`). Required properties: `key, name, version, publisher, summary, categories, contracts_version, runtime, auth_methods, installation_settings_schema, connection_settings_schema, capabilities, webhooks, network, events, sync_resources, rate_limits, rhythm_permissions, data_scopes, execution, support`. **Verified** (schema read with a script; `libs@49728e6:packages/integration-contracts/src/generated/types.ts:1433-2450`). Per capability the manifest gives `{ key, major, features?, modes, constraints? }`, where constraints are a closed vocabulary (`currencies`, `min_amount_minor`, `max_amount_minor`, `countries`) (`types.ts:2215-2250`, quoted in the appendix). Then, per connection, `connection.verify` reports which declared features the actual account supports; this is stored in `integration_connections.verified_features` and checked at resolution. **Verified** `10-capability-contracts.md:66-80`; `services-platform:modules/integrations/services/connections/verify-connection.ts:1-24`. Example, the Dummy adapter's manifest (`integrations-dummy:integration.manifest.ts:13-109`, **Verified**): `contracts_version: '0.7.1'`, two auth methods (`api_key`, `platform_account` with `account_qualifier: { field: 'account_id' }`), 17 capabilities (payments, virtual rooms, three `connection.*`), one manual HMAC webhook kind, `events.produces` and `events.subscribes: ['crm.contact.updated']`, one sync resource `contact`, one rate bucket `{ bucket: 'api', per_second: 200, burst: 400, capabilities: ['payments.*', 'events.*', 'connection.*'] }`, `execution: { sync_timeout_ms: 20000, record_payloads: false }`. #### 2.3 How an application discovers and binds - **Discovery** (for configuration screens): `GET /integrations/{t}/connections?capability=&major=&features=` and `GET /integrations/{t}/catalog?capability=`. **Verified** routes exist (`services-platform:modules/integrations/aws/api/integrations.routes.ts`, route list in section 4.6); design in `10-capability-contracts.md:126-132`. - **Binding** (data): an administrator creates `integration_bindings` rows (`PUT /integrations/{t}/bindings`), offered only at the scopes the capability's policy allows. **Verified** `react-console:components/integrations/bindings-screen.tsx` header; `binding-policies.ts`. - **Resolution** (runtime): `resolveBinding(ctx, { capability, major, context, required_features })` (`services-platform:modules/integrations/services/bindings/resolve.ts:107`) walks the policy's scopes most specific first; for each scope it finds the current binding and runs nine eligibility checks (`binding_active`, `connection_state`, `installation_enabled`, `version_deployed`, `implements_major`, `features`, `constraints`, `circuit`, `authorization`); the first eligible binding wins; otherwise `refuse` (money, or a binding with `on_ineligible: 'refuse'`) or `walk`; nothing bound → `no_binding`. The full trace is returned. **Verified** `resolve.ts:1-31`; `EligibilityCheck` in `types.ts:446-475`. - **Explicit connection**: `connection_id` on the request skips resolution (`resolveExplicitConnection`, `resolve.ts:239`); the connection must be active or degraded and its pinned version must implement the capability. **Verified**. - **Policy for `crm.*`**: `{ capabilities: keysOf('crm.'), major: 1, scopes: [APPLICATION, TENANT], fallback: 'walk' }`. No `resource` or `business_unit` scope. **Verified** `binding-policies.ts:110`. **Inferred consequence for enrichment.** A tenant can have exactly one effective enrichment connection per capability per application (plus a tenant default). Choosing among providers by field, by cost, by match quality, or calling several and merging is not expressible in bindings and would sit in the consuming application or a new layer. --- ### 3. Connections and authentication #### 3.1 Methods and where secrets live **Verified** `services-platform:docs/proposals/integration-platform/11-connections-and-authentication.md:43-58`, `:99-110`, matching code in `services/credentials/*`: | Method | Stored | Lease `values` (`credential-values.ts:7-15`) | | ----------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | | `api_key` | tenant secret with the manifest's fields | the same fields | | `basic` | `{ username, password }` | same | | `custom` | properties of `credential_schema` | same | | `oauth2_client_credentials` | `{ client_id, client_secret }` (kind `api_key`) | `{ access_token, token_type }` fetched on lease, cached in the container | | `oauth2_authorization_code` | access/refresh tokens (kind `oauth2_tokens`); client id/secret are a **platform-scoped** secret `/rhythm/integrations///oauth-client` | `{ access_token, token_type }` | | `platform_account` | nothing per connection; Rhythm's platform secret `/rhythm/integrations///` | the platform secret's fields plus `account_qualifier` (the connection's `external_account_ref`, else the configured setting) | | `aws_assumed_role`, `azure_client_secret` | for storage destinations | short-lived session values | - Tenant secrets: AWS Secrets Manager, one secret per credential row, `/rhythm/integrations////`, KMS alias `platform-integrations-credentials`; only Platform's role may read; adapters never read secrets. **Verified** `integration-credentials.table.ts:1-10`; `services/credentials/credential-store.ts:1-20`; `aws/resources/integration-credentials.kms.ts`, `.iam.ts` exist. - No secret column in any `integration_*` table, proved by `tests/integration/db/integration-secret-columns.integration.spec.ts` (named in `integration-credentials.table.ts:9`). **Verified** (the claim; the spec file name). - Platform-scoped secrets are written by staff through `PUT /hq/integrations/{key}/secrets/{purpose}`, only for purposes some manifest declares. **Verified** `services/credentials/platform-secret-tooling.ts:1-12`. #### 3.2 Rhythm-managed versus customer-provided - **Customer-provided**: `api_key`, `basic`, `custom`, `oauth2_client_credentials`, `oauth2_authorization_code` (the customer authorizes; Rhythm's OAuth client). **Verified**. - **Rhythm-managed**: `platform_account`. The schema requires `account_qualifier` (**Verified**, schema read). `platformAccountLeaseValues` throws if the connection names no qualifier (`credential-values.ts:198-208`). **Inferred**: a Rhythm-paid enrichment provider with one Rhythm API key would fit `platform_account` only by inventing a per-tenant qualifier (for example the tenant id, or a provider sub-account), and every tenant would still need an installation and a connection before anything resolves. #### 3.3 Per-tenant versus platform-wide Every connection is tenant-owned (PK `(tenant_id, id)`, RLS). There is **no platform-wide connection** that tenants share, and no "default connection Rhythm provides" concept in bindings. **Verified** (DDL); **Absent** (search for a shared/system connection in `modules/integrations`). #### 3.4 OAuth state `POST …/connections/{id}/oauth/start` issues a signed `state` bound to tenant, connection and user, valid 10 minutes; the row `integration_oauth_states` (PK = SHA-256 of the state) makes it single-use; PKCE verifier stored as a tenant secret; the public callback `GET /integrations/public/oauth/callback` learns the tenant from the verified signature. **Verified** `integration-oauth-states.table.ts:1-21`; `services/credentials/oauth-state.ts`, `oauth-flow.ts` exist. #### 3.5 Leases, refresh, rotation - The **broker** (`services/credentials/broker.ts:1-30`) is the only code that turns a stored credential into a lease `{ method, lease_id, expires_at, values }`: 5 minutes default, 60 max; OAuth tokens expiring within 60 s are refreshed inline under the connection's lock; each lease writes an audit row (`credential.leased`) and a log line naming the lease id only. `renew`, `rotate`, `destroy`, webhook-secret leases. **Verified**. - Rotation: a new credential supersedes the old; the old secret's deletion is scheduled 7 days out; the connection is re-verified. **Verified** `credential-store.ts:1-20`. - Scheduled: refresh sweep every minute; daily maintenance marks destroyed secrets, reports rotation due (dormant: nothing sets `rotation_due_at`), warns 14 days before refresh-token expiry. **Verified** `services/credentials/credential-passes.ts:1-22`. --- ### 4. Invocation gateway and execution path #### 4.1 How an application calls **Verified** `libs@49728e6:packages/integration-client/src/invoke.ts:1-31`, `:337-403`: - `invoke(ctx, call, options)` (typed by `K@M`) and `invokeOnConnection(ctx, connectionId, call)`. - Wire: a **service call** (`invokeService` of `@rhythmsoftware/macroservice-http`) `POST /integrations//invoke` to the Lambda alias in `PLATFORM_INTEGRATIONS_INVOKE_SERVICE_FUNCTION_ARN`, as the principal in `RHYTHM_SERVICE_PRINCIPAL`. The **invoke grant** is `grantServiceFunctions(role, 'platform', ['integrations_invoke'])` in the caller's CDK. - One attempt, never retried by the client; no answer → `IntegrationInvokeFailed` (treat like `unknown`: retry with the same idempotency key); a non-`InvokeResult` 4xx/5xx → `IntegrationInvokeRefused`. - The caller's wait must exceed the contract timeout plus 1 s (gateway grace) plus 1 s margin (`GATEWAY_GRACE_MS`, `DEFAULT_CALLER_MARGIN_MS`, `:72-76`). - `context.application` defaults to the principal's application (`svc_crm` → `crm`); only `svc_platform` may name any (`:83-101`, `:223`). - A console route for people also exists (`GatewayInvoker` kind `console`, permission `platform.integrations.invoke`), used by "Try a call" on test accounts. **Verified** `services-platform:modules/integrations/services/executions/invoke-capability.ts:47-58`. Current consumers of the client: Accounting (payments, exchange rates, integration events) and Commerce (UPS shipping fixture, service-call wiring). **Verified** (grep, file list in the method section above). CRM: none. **Absent**. #### 4.2 The pipeline **Verified** `invoke-capability.ts:1-19`, `:66`; `attempt-loop.ts:1-28`: validate (≤ 1 MB, envelope and request against the contract, mode, key when required, `timeout_ms` ≤ contract) → authorize (service principal's application = `context.application`; console caller holds the permission) → resolve (binding or explicit connection) → record execution `accepted` (+ execution key) → **async stops here** (dispatch row, 202 `accepted`) → admit (circuit, then rate buckets) → lease → invoke the adapter's `INVOKE` alias with the envelope and deadline → normalize/classify → retry or settle → store the full answer. #### 4.3 Execution records `integration_executions` columns (**Verified** `integration-executions.table.ts:95-137`): `capability_key`, `capability_major`, `connection_id`, `binding_id`, `binding_revision`, `integration_version_id`, `deployment_id`, `invoker_principal`, `originating_user_id`, `correlation_id`, `idempotency_key`, `subject_ref` (the Rhythm record, `object_id:record_id`), `execution_mode`, `status`, `attempt_count`, `request_digest`, `request_summary`, `response_summary` (≤ 4 KB each), `provider_ref`, `error_code`, `error_message`, `started_at`, `finished_at`, `reconcile_after`, `lookup_count`, `cancel_requested_at`, **`cost_units`** ("The provider calls consumed, as the adapter reported them"). Statuses: `accepted`, `running`, `retrying`, `succeeded`, `failed`, `unknown`, `needs_attention`, `cancelled` (`:45-54`). Attempt outcomes: `running`, `succeeded`, `accepted`, `retryable_error`, `terminal_error`, `timeout`, `unknown`, `cancelled`, at most 10 attempts (`integration-execution-attempts.table.ts:29-42`). Retention: 13 months online, then archived to S3 as Parquet; indexes include `(tenant_id, subject_ref, started_at)` for "a record's integration activity". **Verified**. Summaries: one rule for all capabilities until contracts declare projections: top-level scalar fields whose names say nothing secret or personal, strings cut to 200 characters; nothing named like a token, key, instrument, account number, person's name or contact. **Verified** `services/executions/summaries.ts:1-7`. #### 4.4 Sync versus async, timeouts, retries - Contract mode `sync`/`async`/`either`; the gateway's sync budget is the contract's `timeout_ms` (25 s, payments 20 s), async attempt 900 s. **Verified** `registry.ts:29`, `:146-159`. - Sync retry schedule: 4 attempts, delays 1 s, 4 s, 16 s (jittered), inside the caller's budget; beyond it the answer is `retry_later` and the execution continues in the queue, **except** sensitive capabilities, which settle `failed` (`retry_later.budget_exhausted`) and never continue in the background. Async: 6 attempts, 1 min, 5 min, 30 min, 2 h, 6 h. **Verified** `services-platform:lib/invocation-runtime/retry-policy.ts:8-37`; `attempt-loop.ts:19-26`. - Error classes and treatment: `invalid_request`, `refused`, `not_found`, `conflict` (never retried, failed); `rate_limited` (after `retry_after`); `provider_unavailable`, `timeout` (backoff); `unknown` (never retried, reconciled through the lookup capability); `adapter_error` (once, alarmed). **Verified** `libs@49728e6:packages/integration-contracts/src/errors.ts:25-56`. - `unknown` is reconciled every minute by invoking the contract's `lookup` (30 s, then 2 min, 10 min, 1 h, 6 h); a capability with **no lookup goes to `needs_attention` at once**. **Verified** `services/executions/reconciler.ts:1-27`. `crm.contact.enrich` has `lookup: null` (**Verified**), so any `unknown` answer from an enrichment adapter would land in the operator queue (**Inferred**). - A timeout pass settles in-flight executions nothing will finish. **Verified** `reconciler.ts:22-27`; `services/executions/timeouts.ts`. #### 4.5 Idempotency Rule: unique per (tenant, capability, idempotency_key), max 128 characters; reuse with a different request digest is refused `idempotency_key_reused`; a repeat while running answers `in_progress`; a settled repeat answers the stored `integration_execution_responses` row. The provider's idempotency key is always the execution id, never the caller's key. A capability with `idempotency: 'none'` is never deduplicated (`invoke-capability.ts:72`: `deduplicated = !!request.idempotency_key && definition.idempotency !== 'none'`). **Verified** `registry.ts:21-26`; `integration-execution-keys.table.ts:1-15`; `libs@49728e6:packages/integration-sdk/src/context.ts:35-41`. **Inferred**: `crm.contact.enrich` is `idempotency: 'none'`, so two identical enrichment calls are two provider calls (two charges, for a paid provider). Any "don't pay twice for the same lookup" cache must be built outside the gateway or the contract must change. #### 4.6 Rate limits, circuits, outages, concurrency - **Rate limits** (`services/executions/rate-limits.ts:1-12`, `:112`): manifest buckets per connection (sliding window of leases in Valkey via `@rhythmsoftware/macroservice-admission`), lowered by `operator` rows, emptied by `provider_header` `retry_after`. Key `int:{${tenant_id}}:rl:${connection_id}:${bucket}`. Sync over the limit → 429 `retry_after`; async waits. Without a cache endpoint, admission **fails open**. **Verified**. - **Circuit** per (deployment, connection): opens after 5 consecutive failures, open 60 s doubling to 15 min, one half-open probe. **Verified** `services-platform:lib/invocation-runtime/circuit.ts:30-39`, `:222-229`; table `integration_circuit_states`. - **Outages**: `provider_unavailable` retried with backoff; connection health degrades an `active` connection after consecutive failed probes; binding invalidation on connection state change. **Verified** (sections 6, 2.3). - **Async consumer concurrency**: `FnIntegrationsExecutions` (async dispatcher, its queue consumer, the reconciler) has `reservedConcurrentExecutions: 5`, timeout 15 min. **Verified** `services-platform:modules/integrations/aws/logic/integrations-gateway.lambda.ts:92-100`. **Inferred**: bulk async enrichment of large lists would be throttled here before the provider's limits matter. - Proposal sizing note: "A tenant doing 22,000 payments a month … averages under one invocation a minute. The gateway's budget is thousands a minute." **Verified** as doc text `13-invocation-gateway.md:213-226`. #### 4.7 Routes (all verified present at `43478ffda`) Tenant: `/integrations/{t}/` `catalog`, `catalog/{key}`, `installations` (+ suspend/resume/uninstall), `connections` (+ `credential`, `oauth/start`, `reauthorize`, `verify`, `health`, `suspend`, `resume`, `disconnect`, `update_version`, `move_bindings`, `try`, `webhook_endpoints`), `bindings` (+ `resolve`, `migrate`, `{id}/revisions`), `invoke`, `executions` (+ `attempts`, `cancel`, `resolve`, `retry`), `events`, `subscriptions` (+ `replay`), `deliveries` (+ `retry`), `webhook_events` (+ `replay`), `webhook_endpoints/{id}/rotate|replay`, `audit_events`, `connection-notices`, `health`. Adapter callbacks: `/integrations/adapter/…` (section 8.4). Public: `/integrations/public/webhooks/{token}`, OAuth callback. Registration: `/integrations/registration`, `/integrations/registration/consumers`. HQ: `/hq/integrations/catalog`, `events`, `executions`, `circuits/{id}/reset`, `versions/{id}/publish|deprecate|retire`, `{key}/availability`, `{key}/secrets[/{purpose}]`. **Verified** (grep of `modules/integrations/aws/api/*.routes.ts`). --- ### 5. Webhooks, events, subscriptions, deliveries All **Verified** in code on master (WS8); the milestone that would accept them (INTG-M3) is **Proposed**. #### 5.1 Inbound provider webhooks 1. **Endpoints**: one per (connection, kind) the manifest declares, created on first successful verification, with an unguessable `public_path_token`, the verification scheme (`hmac_sha256`, `hmac_sha1`, `api_key_header`, `basic`, `none`), `secret_ref` and `previous_secret_ref` (rotation overlap), `provider_endpoint_ref` when registered through `webhooks.register`. `scope: 'app'` gives one endpoint for a provider app routed by account reference. `integration-webhook-endpoints.table.ts:1-68`; manifest `webhooks` (`types.ts` after `:2250`). 2. **Ingress** `POST /integrations/public/webhooks/{token}`: endpoint lookup; > 1 MB → 413; per-endpoint token bucket → 429; raw body and headers written to S3 first; row `pending`/`queued` + queue row in one transaction; 200, or an inline challenge answer. Nothing is parsed or trusted at ingress. `services/webhooks/ingress.ts:1-18`. 3. **Worker**: leases the endpoint's secrets, invokes the adapter's `WEBHOOK` entry (`verify` then `parse`), deduplicates through `integration_webhook_event_keys` (`UNIQUE (endpoint_id, provider_event_id)`, or `UNIQUE (endpoint_id, body_digest)` within 24 h), writes `integration_events` rows, retries 6 times then `dead`; operator replay bypasses dedup. `services/webhooks/webhook-worker.ts:1-20`; `integration-webhook-event-keys.table.ts:1-41`. 4. Retention: rows 90 days (13 months for the `payments` kind); partitioned monthly. `integration-webhook-events.table.ts:15-56`. #### 5.2 Integration events (Platform to Rhythm applications) `integration_events`: CloudEvents 1.0 shape, `type` `rhythm.integration...`, `source` `/integrations/connections/`, `data` validated against the type's schema; origins `webhook`, `async_completion`, `sync_run`, `adapter_emit`, `reconciliation`; delivery statuses `pending`, `delivered`, `retrying`, `failed`, `dead`, `no_consumer`; 90 days. The router invokes each registered consumer's function (`integration_event_consumers`, registered by `POST /integrations/registration/consumers`) with `POST /integrations//events/consume`; at-least-once, consumer dedups by event id; `execution.completed` goes only to the invoking application. **Verified** `integration-events.table.ts:1-86`; `services/events/event-router.ts:1-20`; client side `libs@49728e6:packages/integration-client/src/events.ts:45-151` (`defineIntegrationEventConsumer`). #### 5.3 Outbound domain events (Rhythm to adapters) `integration_subscriptions`: an installation subscribes to a Rhythm domain event type its manifest lists in `events.subscribes` (and the administrator accepted), optional filter in the shared conditions grammar, `delivery_mode` `push` or `pull` (cursor), status `active`/`paused`/`paused_by_failures` (after 50 consecutive dead). The router consumes owning services' change events (CRM's change-events framework contract), projects them (never the raw row), writes one `integration_deliveries` row per (subscription, event) through `integration_delivery_keys` (`UNIQUE (subscription_id, event_ref)`), and the delivery worker invokes the adapter's `EVENTS` entry; 6 attempts then `dead`; `skipped_filter` recorded. Payload ≤ 64 KB; 90 days. **Verified** `integration-subscriptions.table.ts:1-65`; `integration-deliveries.table.ts:1-74`; `integration-delivery-keys.table.ts:1-24`; `services/subscriptions/change-event-handler.ts:1-20`; `delivery-worker.ts:1-16`. Domain events in the catalog that matter to enrichment: `crm.contact.created|updated|merged|deleted`, `crm.organization.created|updated|merged|deleted`. **Verified** (`schemas/domain-events/` listing). **Inferred**: an enrichment adapter could be driven by subscription (Rhythm pushes `crm.contact.updated` to the adapter), but the adapter then has no route to write the answer back into CRM (no CRM callback, data grants not built); the documented pattern is the opposite: CRM invokes the capability and writes the answer itself (`21-application-examples.md:134`). --- ### 6. Health and verification jobs **Verified**: - **Verification** (`connection.verify` through the adapter transport; Step Functions workflow `platform-integrations-connection-verification`): purposes `created`, `reauthorized`, `requested`, `scheduled`, `update_version`. A `verifying` connection becomes `active` (with `external_account_ref`, `verified_features`, webhook endpoints) or `failed_verification`; `update_version` moves the pin only on success and only for the same account; scheduled/requested may degrade or recover. Lost features write `connection.features_changed`, invalidate bindings that need them, and notify administrators. `services/connections/verify-connection.ts:1-24`; `integration-verification-jobs.table.ts:1-46`. - **Health** (`connection.health`, cheap, read-only; workflow `platform-integrations-connection-health`): outcomes `healthy`, `failing`, `degraded`, `recovered`. `integration-health-jobs.table.ts:1-40`. - **Scheduled pass** (`platform.integrations.health`): every 15 minutes a health job for each active/degraded connection that failed in the last day or is degraded, plus folding the last hour of executions into `health.success_rate_1h`, `p95_ms_1h`, `executions_1h`; daily a `scheduled` verification of every active/degraded connection. Walks tenants one by one inside a time budget. `services/health/health-pass.ts:1-21`. **Inferred**: for a metered enrichment provider, the daily verification and 15-minute probes are provider calls; the adapter should make `connection.verify`/`health` free endpoints (account info), or they will consume paid quota per tenant connection. --- ### 7. Audit: `integration_audit_events` **Verified** `integration-audit-events.table.ts:1-75`; actions from grep of `modules/integrations/services`. - Columns: `id` (UUIDv7 default), `tenant_id`, `actor_id`, `actor_access_type` (`console`/`service`/`system`), `originating_user_id`, `action`, `target_kind` (`installation`, `connection`, `webhook_endpoint`, `binding`, `execution`, `circuit`, `subscription`, `webhook_event`, `delivery`), `target_id`, `details` (jsonb, never a secret), `at`. Append-only, no system columns, not partitioned, read by tenant administrators at `GET /integrations/{t}/audit_events`. - Installation and connection lifecycle changes are also written to `platform.security_audit_events` in the same transaction. - Actions recorded include: `installation.enabled|permissions_accepted|reinstalled|resumed|suspended|uninstalled|updated`; `connection.created|updated|suspended|verification_passed|features_changed|version_update_requested|version_updated|version_update_failed|test_call`; `credential.stored|authorization_started|authorized|leased|lease_renewed|refreshed|refresh_expiring_notified|revoked|destroyed`; `binding.changed`; `circuit.reset`; `execution.cancel_requested|cancelled|resolved_by_operator|retried|settled_by_reconciliation`; `subscription.created|changed|removed|paused_by_failures|paused_by_acceptance`; `delivery.replayed|retried`; `webhook_endpoint.registered|registration_failed|rotated|rotation_finished|secret_created|secret_shown`; `webhook_event.replayed`. - **An ordinary invocation writes no audit row**: its record is the execution and attempt rows (plus a `credential.leased` audit row per lease). The proposal says the same (`13-invocation-gateway.md:207-211`). --- ### 8. Provider microservice pattern and developer SDK #### 8.1 One provider, one repository, one deployable **Verified** as design `14-provider-microservice.md:9-16` and as practice: adapter repositories on GitHub `integrations-dummy`, `-stripe`, `-payrix`, `-zoom`, `-mailchimp`, `-ups`, `-aws-s3`, `-azure-blob`, `-adapter-template` (plus 1.0-era `-*-accounts`, `-ngx-api`, `-omni`) (`gh repo list`, names only). Each is created from `integrations-adapter-template`, depends only on the npm `integration-*` packages and `@rhythmsoftware/macroservice-cdk`, and deploys stacks `r2-integration--resources|logic|ci` via `defineAdapterStacks`: **four Lambda functions from one bundle**, each with a `v1live` alias, alias resource policies admitting Platform's role, SSM parameters for `integration publish`, alarms. **Verified** `integrations-dummy:aws/adapter.ts:1-60`. No database; no queue unless the adapter asks (`14-provider-microservice.md:202-214`). #### 8.2 The SDK **Verified** `libs@49728e6:packages/integration-sdk/src/handlers.ts:1-132`, `integration.ts:1-140`, `context.ts:18-66`: - `defineManifest(...)` (authoring and validation), `defineCapabilityHandler(key, major, (request, ctx) => response)` typed by the contract, `defineConnectionHandlers({ verify, health?, describe? })`, `defineWebhook(kind, { verify, parse })`, `defineEventReceiver(types, handler)`, `defineSyncResource(name, { describe, pull?, push? })`, `defineIntegration({ manifest, connection, capabilities, webhooks?, events?, sync?, stub? })` → `{ INVOKE, WEBHOOK, EVENTS, SYNC }` Lambda entries. - Entry wrapper: envelope version and schema check; deadline (`ctx.deadline`, `ctx.signal`); result validated against the contract; thrown `ProviderError` classified, anything else `adapter_error` (or `unknown` if a provider request was already sent); one structured log line per invocation without bodies; lease values deleted at the end. Stub mode `INTEGRATION__MODE=stub`, refused in deployed environments. - `HandlerContext`: `execution_id`, `provider_idempotency_key` (= execution id), `attempt`, `capability`, `major`, `idempotency_key`, `tenant { id, time_zone }`, `connection { id, external_account_ref, account_qualifier, configuration, installation_settings }`, `credential` (lease), `correlation`, `deadline`, `signal`, `logger` (redacting), `rhythm` (callbacks), `features`, `manifest`. The tenant's name and business data are never sent. - Network: the SDK's HTTP client may reach only `network.allowed_hosts` (SSRF guard). **Verified** manifest schema text. - `integration-cli` commands: `validate`, `conformance`, `invoke` (local runner, `--stub` or `--live`), `publish --env [--release]`, `scaffold`. **Verified** `libs@49728e6:packages/integration-cli/src/commands.ts:1-14`. - `integration-testing` conformance: request examples validate; each example invoked through `INVOKE` against the stub provider; every `must_map` error code has a fixture; feature coverage; deadline behaviour; webhook valid/tampered/replayed/challenge. **Verified** `libs@49728e6:packages/integration-testing/src/conformance.ts:1-25`. #### 8.3 Registration **Verified** `services-platform:modules/integrations/services/registration/register-integration-version.ts:1-22`; `libs@49728e6:packages/integration-cli/src/registration.ts:1-9`: `integration publish` reads `/rhythm/integrations///entrypoints/` and `/manifest-hash` from SSM, refuses if the deployed hash differs from the checkout's, and invokes Platform's registration function (`/rhythm/platform//integrations/registration-arn`) as the adapter's principal: `POST /integrations/registration` with `{ manifest, manifest_hash, deployment: { environment, entrypoints, principal, runtime, region }, conformance_report, release }`. Platform then: (1) checks the principal (`svc_integration_` may register only ``; `svc_platform` only outside production); (2) validates the manifest against the schema; (3) checks `contracts_version` ≤ Platform's and every named major and feature exists; (4) recomputes the hash; (5) in one transaction upserts the definition, writes missing `capability_definitions` rows, writes the version (`validated`, or `published` with `release: true`) and its capabilities, and the deployment (previous active → `draining`). Two gates outside the adapter's control (**Verified**): - The registration Lambda alias carries one `lambda:InvokeFunction` permission per adapter deploy role, for the keys in `REGISTERING_ADAPTER_KEYS` only (`integrations.settings.ts:28`; `docs/integrations/ip-registration-invoke-grant-notes.md`). - The adapter principal is admitted by pattern in `services/principals.ts` pending the service-principal registry (WS0b). #### 8.4 Adapter callbacks (`ctx.rhythm`) **Verified** `libs@49728e6:packages/integration-sdk/src/rhythm-client.ts:1-20`; `services-platform:modules/integrations/services/adapter-callbacks/adapter-callbacks.ts:1-21`; `gateway.routes.ts:150-196`. Served by `platform_integrations_adapter_service`; every call names its connection (`x-integration-connection-id`), whose tenant Platform enters, and must come from the principal of that connection's pinned deployment. | Call | Route | State at `43478ffda` | | ------------------------------------ | ---------------------------------------------------------- | --------------------------------------------------------------------------- | | `credentials.renew(lease_id)` | `POST /integrations/adapter/credentials/{lease_id}/renew` | built | | `executions.progress(id, progress)` | `POST /integrations/adapter/executions/{id}/progress` | built; answers whether cancellation was requested | | `executions.complete(id, result)` | `POST /integrations/adapter/executions/{id}/complete` | built; validates against the contract, settles, emits `execution.completed` | | `events.emit(events)` | `POST /integrations/adapter/events` | built; types must be in the manifest's `produces`; ≤ 100 a call | | `data.query(grant_id, page)` | `POST /integrations/adapter/data/{grant_id}/query` | **501, not built (WS9)** | | `data.project(grant_id, record_ids)` | `POST /integrations/adapter/data/{grant_id}/project` | **501, not built (WS9)** | | `sync.cursor.get/set(profile_id)` | `GET`/`PUT /integrations/adapter/sync/{profile_id}/cursor` | built only for `pull` subscriptions; sync profiles are WS10 | --- ### 9. Extension points, and enrichment mentions in the documents #### 9.1 Chapter 22 (Extension Points) **Verified** as doc text `services-platform:docs/proposals/integration-platform/22-extension-points.md`: - Boundary: integrations connect Rhythm to external providers through Rhythm-defined capabilities; extension points customize Rhythm's internal behaviour (pricing, eligibility, approval). Separate product concepts (`:17-35`). - Shared: `lib/invocation-runtime/` (envelope and result shape, transports, execution records, error classification and retries, timeouts, circuit breaking, correlation and metrics, JSON Schema validation, conformance, service principals) (`:37-56`). **Verified** that `lib/invocation-runtime/` exists with `circuit.ts`, `classification.ts`, `deadline.ts`, `envelopes.ts`, `lambda-transport.ts`, `retry-policy.ts`, `transport.ts`. - Not shared: catalog, installation and consent, connections and credentials, bindings by capability, webhooks, subscriptions, synchronization (`:58`). - An extension that needs an external system uses an integration through a scoped integrations client (`:35`). - Explicitly listed extension points (proposed names, all **Planned**): `commerce.product.pricing`, `commerce.order.pricing`, `commerce.order.preprocessing`, `commerce.eligibility.evaluate`, `membership.application.generate`, `.preprocess`, `membership.order.generate`, `membership.approval.decide`, `membership.activation.after`, `membership.sections.describe`, `membership.chapters.suggest`, `membership.renewal.transform`, `certifications.requirement.evaluate`, `events.priority_points.accumulate`, `platform.record_action.execute`, `submissions.reviewers.assign` (`:70-82`). None is about enrichment. The Extension Points registry itself is **Absent** on master (no `extension_executions` or registry table found in `modules/`; `modules/extensions` exists and was not examined for this question). #### 9.2 Enrichment in the integration documents - `03-rhythm1-landscape.md:100-109` (**Verified** as doc text): section "3.5 Accounting, automation, enrichment, shipping, files"; 1.0 **Google Civic** (`partners-google-civic`), "District enrichment on contact change", API key in SSM; "The Representatives API was deprecated by Google (Inferred)". - `10-capability-contracts.md:42`: the `crm` category's first taxonomy is `contact.enrich`, `contact.resource` and `organization.resource` (sync resources), sources "1.0 Google Civic, Higher Logic, Mobilize". - `21-application-examples.md:127-137`: CRM consumes `crm.contact.enrich` (sync); "a CRM rule invokes `crm.contact.enrich` (`{ address, fields_requested }` → `{ fields }`) on an address change … CRM writes the returned fields through its own persistence with `app.source = 'integration'`. The engine's digest provides the no-diff guard"; "the CRM data-quality screen lists `crm.contact.enrich` connections". **Planned** (no CRM rule, no data-quality screen found). - `25-migration.md:53`: Google Civic → `crm.contact.enrich`, "none: retired (Andrew, 2026-10-17, Q-6)". - **FullContact: Absent.** No mention in chapters 03 or 21, or anywhere in `docs/` or `modules/` (grep `full.?contact`, case-insensitive; the only hits are "full contact upserts" in the data-engine proposal, unrelated). No Clearbit, ZoomInfo, Melissa, Cicero mentions either. - **`lookup`** capabilities in the registry are reconciliation counterparts (`payments.lookup`, `events.virtual_room.lookup`, `accounting.journal.lookup`, `files.deliver.lookup`, `shipping.shipment.lookup`), not data lookups. **Verified** `registry.ts:69-73`. - **`search`**: not a verb, not a capability. **Absent** (`naming.ts:37-76`). - Sync resources `contact` and `organization` exist (`schemas/sync-resources/contact.json`, `organization.json`) for bidirectional sync profiles, which are not built. **Verified** (files) / **Planned** (engine). --- ### 10. Metering and usage - **Integrations module**: the only usage-like column is `integration_executions.cost_units integer` ("The provider calls consumed, as the adapter reported them"), filled from `AdapterResult.cost_units` on success (`attempt-loop.ts:328`; `adapter-callbacks.ts:146`; `integration-executions.ts:697`). The adapter result schema describes it as "Provider quota units the call consumed, where the provider counts them" (`types.ts:381-400`). Nothing reads, sums or limits it. Connection `health` keeps last-hour counts (`executions_1h`) for health only. **Verified**. - **Absent** in `modules/integrations`: `quota`, `metering`/`metered`, `credits`, any per-tenant allowance, any billing hand-off, any usage rollup table. (Hits for "budget" are time budgets.) - **Licensing**: `modules/licensing` has the application catalog and HubSpot sync; no usage metering. The only "credits" hit is the Learning application's description. **Verified** `modules/licensing/catalog/applications.ts:112-121`. - **Precedent elsewhere in Platform**: `modules/llm` meters AI usage: `llm_allowance_periods` (one row per tenant per month with `reserved_credits`, `settled_credits`, `in_flight_count`; authorization is one conditional `UPDATE` against `LEAST(platform_ceiling, recurring_base_credits + overrides)`), `llm_tenant_limits`, `llm_allowance_overrides`, `llm_allowance_adjustments`, `llm_usage_daily` (rollup by tenant, application, feature, provider, model, rate version), `llm_usage_threshold_crossings` (80 % and 100 % notices), `llm_model_rate_versions` (append-only prices; an unpriced model is unusable). **Verified** (table headers, `modules/llm/db/*.table.ts`). PLAT-M1 ("AI usage is metered, limited and visible") is **Awaiting Plan Approval** (`docs/milestones/PLAT-M1-ai-usage-metered-and-visible.md:7`). --- ### 11. Built versus planned #### 11.1 Milestone state **Verified** `services-platform:docs/milestones/INTG-M*.md` status rows; `docs/proposals/integration-platform/31-milestone-plan.md:11-33`. | Milestone | Scope | Status | | --------- | ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | INTG-M1 | Find, install, connect (api_key), verify, health, suspend/resume, disconnect, uninstall | **Product Accepted** 2026-10-21 ~20:28 ET (Engineering Verified 16:34 ET, run m1live10d, 19/19 scenarios) | | INTG-M2 | Applications call through the gateway: bindings, executions, retries, unknown/reconcile, async, "Try a call" | **In Progress**; first live run m2live1 (2026-10-22 05:10 ET) satisfied 1 of 22 scenarios (Dummy lacked async and scripted behaviours; harness issues) | | INTG-M3 | Webhooks, events, deliveries | **Proposed** (plan not written) | | INTG-M4 | OAuth and credential lifecycle | **Proposed** | | INTG-M5 | Data synchronization engine | **Proposed** | | INTG-M6 | Rhythm HQ operations | **Proposed** | | INTG-M7 | Stripe test-mode card payments | **In Progress** | | INTG-M8 | Payrix card payments | **In Progress** | "Every milestone is proved with the Dummy adapter (`integrations-dummy`) in the dev environment" (`31-milestone-plan.md:13`). Real-provider work is outside the INTG job (`:31-33`). #### 11.2 Component state | Component | Code on master `43478ffda` | Accepted | | -------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | --------------------------------------------------- | | Catalog, versions, capability mirror, deployments, availability grants, registration | built | M1 accepted (registration of the Dummy in dev) | | Installations, connections, state machine, api_key credentials, verification and health jobs, audit | built | M1 accepted | | `platform_account` credentials | built (broker, values) | used by M7/M8 in progress | | OAuth authorization code, refresh sweep, rotation | built (WS3) | M4 proposed | | Bindings, resolver, revisions, invalidation, migrate | built (WS4, M2 WP1) | M2 in progress | | Gateway: sync and async invoke, attempts, idempotency, responses, reconciler, timeouts, circuits, rate limits, payload recording | built (WS5/WS5b, M2 WPs) | M2 in progress | | Webhook ingress and worker, integration events, router, subscriptions, deliveries, pull cursor | built (WS8) | M3 proposed | | Data grants (`data.query`, `data.project`) | **501 stubs** | Planned (WS9) | | Sync profiles, runs, external ids, mapping | **Absent on master** (`agent/ip-ws10*` branches unlanded per `31-milestone-plan.md:29`) | M5 proposed | | HQ tooling routes (`/hq/integrations/*`) | routes present | M6 proposed; **no HQ console screens** (section 12) | | Extension runtime sharing `lib/invocation-runtime` | runtime built; extension registry not part of this module | Planned (chapter 22) | | Minor-version stripping for older adapters | not found | Planned (`10-capability-contracts.md:84`) | | Contract-declared summary projections | not built (one generic rule) | Planned (`summaries.ts:1-6`) | | Rotation policy (`rotation_due_at`) | column exists, nothing writes it | Planned (WS3 decision 10) | --- ### 12. Console screens (react-console `45a47ab6b`, read only) All tenant screens live in the Integration Library under `/configuration/platform/integrations/…`; `app/(console)/console/[tenantId]/integrations/page.tsx` only redirects there (`:1-9`). Routing: `react-console:lib/integrations/library-routes.ts:1-53`; dispatcher `react-console:components/integrations/integrations-library.tsx:28-71` (behind `permissions.configuration.integrations.read`). Client: `react-console:lib/integrations/client.ts`; mock double `mocks/handlers/integrations.ts`. **Verified**. | Path segment | Screen (file) | What it shows | | ----------------------------------- | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `` (home) | `library-home.tsx` | summary row; one card per installed integration with connections, health, last activity, anything reconciling | | `catalog` | `catalog-screen.tsx` | provider search by category and by capability (`GET catalog?q&category&capability`); restricted integrations only when granted; Install | | `catalog/{key}` | `provider-detail.tsx` | what the integration does, permissions it asks for, versions; Install opens consent and installation settings (FormEngine over the manifest schema) | | `{key}` | `installation-detail.tsx` | connections, settings, accepted permissions and who accepted, event subscriptions, versions; Suspend, Resume, Uninstall | | `{key}/connections/new` | `connection-wizard.tsx` | name and settings, authenticate by method (`credential-entry.tsx`, OAuth), watch verification, bind in one click | | `{key}/connections/{id}` | `connection-detail.tsx` (1,267 lines) | account, verified features, credential summary, bindings using it, webhook endpoints, subscriptions, activity, audit; Verify, Update, Re-Authorize, Suspend, Resume, Disconnect; `try-a-call.tsx` on test accounts | | `{key}/connections/{id}/executions` | `activity-screen.tsx` | that connection's executions | | `bindings` | `bindings-screen.tsx` | bindings grouped by capability; Add Binding drawer limited to the policy's scopes and eligible connections; Explain panel (`resolution-explain.tsx`) drawing the resolver trace | | `activity` | `activity-screen.tsx` | every execution, filtered by status, capability, connection, date | | `activity/{executionId}` | `execution-detail.tsx` | connection, binding revision, invoker, attempts, resolution trace, bounded summaries, provider request id; Retry (inert), Cancel, Resolve | | `webhooks` | `webhooks-screen.tsx` | received provider notifications, verification, processing, emitted events; Replay | | `events` | `events-screen.tsx` | subscriptions (Pause, Resume, Remove, new) and deliveries (Retry) | | `sync`, `sync/{profileId}` | `sync-screens.tsx` | sync profiles read-only; mapper and run controls marked not built. **Inferred**: these call `/sync_profiles` (`client.ts:611-612`), which Platform does not serve, so they only work against the mock double | | `needs-attention` | `needs-attention-screen.tsx` | queue grouped by problem, one fixing action per row | - `lib/integrations/policies.ts:168` and `binding-policies.snapshot.json:270` already list `crm.contact.enrich` (from the contracts snapshot). **Verified**. - **No Rhythm HQ integration screens**: nothing under `components/console/hq/` references integrations, and no component calls `/hq/integrations`. **Absent**. - No CRM "data quality" screen listing enrichment connections. **Absent** (Planned in `21-application-examples.md:137`). --- ### Appendix: quoted interfaces All quotes are verbatim from the files cited. Where a generated type's doc comments were removed to save space, the quote says so; no field was added, renamed or reordered. #### A.1 Capability definition (registry) `libs@49728e6:packages/integration-contracts/src/registry.ts:15-65` ```ts /** sync answers within the gateway's budget; async through an execution record and an event; either lets the caller choose. */ export type ExecutionMode = 'sync' | 'async' | 'either'; /** required: the caller must send a key; optional: deduplicated when sent; none: a read, never deduplicated. */ export type IdempotencyRequirement = 'required' | 'optional' | 'none'; /** The rule for a key reused with a different request digest, the same for every capability (chapter 13 section 7). */ export const IDEMPOTENCY_RULE = Object.freeze({ unique_per: ['tenant', 'capability', 'idempotency_key'] as const, reuse_with_different_digest: 'refused' as const, refusal_reason: 'idempotency_key_reused' as const, max_length: 128, }); /** The gateway's synchronous budgets (chapter 10 section 4, chapter 13 section 8), in milliseconds. */ export const TIMEOUTS_MS = Object.freeze({ payments: 20_000, sync: 25_000, async_attempt: 900_000 }); export interface CapabilityDefinition { readonly key: string; readonly major: number; /** The latest minor of this major. */ readonly minor: number; readonly category: Category; /** One sentence: what the capability does, from Rhythm's side. */ readonly summary: string; readonly mode: ExecutionMode; /** The mode when the caller names none (equal to `mode` unless it is `either`). */ readonly default_mode: 'sync' | 'async'; /** The most a caller may allow for a sync invocation, or one async attempt. */ readonly timeout_ms: number; readonly idempotency: IdempotencyRequirement; /** Money moves: drives the audit level, payload recording (never) and the `refuse` fallback; requires a lookup counterpart. */ readonly sensitive: boolean; /** Personal data moves to or from the provider: payloads are never recorded. */ readonly personal_data: boolean; /** The Knox permission a console invoker needs, and its resource (the category); trusted service invokes evaluate ALL. */ readonly permission: { readonly key: 'platform.integrations.invoke' | 'platform.integrations.manage'; readonly resource: string }; /** The named optional behaviours, as `$defs/feature` of the schema declares them. */ readonly features: readonly string[]; /** The integration event types (without `rhythm.integration.`) the capability can produce asynchronously. */ readonly events: readonly string[]; /** The capability the reconciler invokes for an `unknown` outcome, on the same connection. */ readonly lookup: { readonly key: string; readonly major: number } | null; /** The capability's own error codes and their classes; `COMMON_ERROR_CODES` apply besides. */ readonly errors: Readonly>; /** The codes conformance requires a stub fixture for: an adapter must map each. */ readonly must_map: readonly string[]; /** Set when this major is deprecated; `sunset_at` is then at least 180 days later, after which resolution refuses `contract_retired`. */ readonly deprecated_at: string | null; readonly sunset_at: string | null; } ``` #### A.2 The existing enrichment capability `libs@49728e6:packages/integration-contracts/src/registry.ts:1080-1100` ```ts // crm define({ key: 'crm.contact.enrich', summary: 'Fields a provider knows about an address (districts, representatives, geocodes), for CRM to write through its own persistence.', mode: 'sync', idempotency: 'none', sensitive: false, personal_data: true, permission: invoke('crm'), features: [], events: [], lookup: null, errors: { 'invalid.address': 'invalid_request', 'not_found.address': 'not_found', 'invalid.field_requested': 'invalid_request', 'rate_limited.provider': 'rate_limited', }, must_map: ['invalid.address', 'not_found.address', 'rate_limited.provider'], }), ``` Request and response `$defs`, `libs@49728e6:packages/integration-contracts/schemas/capabilities/crm/crm.contact.enrich@1.json:38-109` (verbatim excerpt): ```json "request": { "description": "What to enrich.", "type": "object", "additionalProperties": false, "required": ["address", "fields_requested"], "properties": { "address": { "$ref": "/common/1#/$defs/address", "title": "Address", "description": "The address to enrich." }, "fields_requested": { "title": "Fields requested", "description": "The provider's field names to answer (the adapter's describe lists them).", "type": "array", "items": { "type": "string", "pattern": "^[a-z][a-z0-9_]*$", "maxLength": 100 }, "minItems": 1, "maxItems": 100, "uniqueItems": true } }, "title": "Request" }, "response": { "description": "What the provider knows.", "type": "object", "additionalProperties": false, "required": ["matched", "fields"], "properties": { "matched": { "title": "Matched", "description": "Whether the provider recognised the address.", "type": "boolean" }, "fields": { "title": "Fields", "description": "The requested fields the provider could answer, by name; a field it could not answer is absent.", "type": "object", "additionalProperties": { "anyOf": [ { "type": "string", "maxLength": 1000 }, { "type": "number" }, { "type": "boolean" }, { "type": "null" }, { "type": "array", "items": { "type": "string", "maxLength": 1000 }, "maxItems": 100 } ] }, "maxProperties": 100 } }, "title": "Response" } ``` (Whitespace compacted; keys and values as in the file.) #### A.3 Typed request/response accessors `libs@49728e6:packages/integration-contracts/src/capabilities.ts:11-36` ```ts /** Every capability key. */ export type CapabilityKey = keyof Capabilities; /** The majors defined for a capability key. */ export type MajorOf = keyof Capabilities[K] & number; type Contract> = Capabilities[K][M] extends { request: unknown; response: unknown; features: unknown; mode: unknown; } ? Capabilities[K][M] : never; /** The request of `K@M`. */ export type RequestOf> = Contract['request']; /** The response of `K@M`. */ export type ResponseOf> = Contract['response']; /** The features of `K@M` (never when it declares none). */ export type FeatureOf> = Contract['features']; /** The modes `K@M` may run in. */ export type ModeOf> = Contract['mode']; ``` #### A.4 Invoke request (caller to gateway) `libs@49728e6:packages/integration-contracts/src/envelope.ts:31-40` ```ts /** What a consuming service sends the gateway for `K@M`. */ export type InvokeRequest> = Omit< AnyInvokeRequest, 'capability' | 'major' | 'request' | 'required_features' > & { capability: K; major: M; required_features?: FeatureOf[]; request: RequestOf; }; ``` `libs@49728e6:packages/integration-contracts/src/generated/types.ts:76-157` (doc comments removed; fields verbatim): ```ts export interface InvokeContext { application: ApplicationKey; business_unit_id?: string; resource?: { object_id: string; record_id: string; }; subject_ref?: string; extra?: { [k: string]: string; }; } export interface AnyInvokeRequest { capability: string; major: number; mode?: InvocationMode; idempotency_key?: string; context: InvokeContext; connection_id?: string; required_features?: string[]; request: {}; timeout_ms?: number; correlation_id?: string; } ``` `ApplicationKey` (`types.ts:60-71`): `'accounting' | 'events' | 'learning' | 'marketing' | 'crm' | 'membership' | 'certifications' | 'committees' | 'objects' | 'commerce' | 'platform'`. The client's call shape and entry points, `libs@49728e6:packages/integration-client/src/invoke.ts:107-109`, `:337-341`, `:395-400`: ```ts export type InvokeCall> = Omit, 'context'> & { context?: Omit & { application?: ApplicationKey }; }; ``` ```ts export async function invoke>( ctx: ExecutionContext, call: InvokeCall, options: InvokeOptions = {}, ): Promise> { ``` ```ts export async function invokeOnConnection>( ctx: ExecutionContext, connectionId: string, call: Omit, 'connection_id'>, options: InvokeOptions = {}, ): Promise> { ``` #### A.5 Invoke result (gateway to caller) `libs@49728e6:packages/integration-contracts/src/envelope.ts:61-68` ```ts /** What the gateway answers the caller for `K@M`. */ export type InvokeResult> = | (Omit, 'response'> & { response: ResponseOf }) | Branch | Branch | Branch | Branch | Branch; ``` `libs@49728e6:packages/integration-contracts/src/generated/types.ts:499-620` (doc comments removed; fields verbatim): ```ts export type RefusalReason = | 'no_binding' | 'binding_ineligible' | 'connection_disconnected' | 'connection_unavailable' | 'contract_retired' | 'idempotency_key_reused' | 'in_progress'; export type AnyInvokeResult = | { outcome: 'succeeded'; execution_id: string; connection_id: string; response: {}; warnings: string[]; } | { outcome: 'accepted'; execution_id: string; connection_id: string; } | { outcome: 'failed'; execution_id: string; connection_id?: string; error: ContractError; } | { outcome: 'unknown'; execution_id: string; connection_id: string; error: ContractError; } | { outcome: 'retry_later'; execution_id: string; connection_id: string; retry_after_s: number; } | { outcome: 'refused'; reason: RefusalReason; trace: ResolutionTrace; execution_id?: string; }; ``` `ContractError`, `types.ts:25-50` (doc comments removed): ```ts export interface ContractError { class: ErrorClass; code: ErrorCode; message: string; retry_after_s?: number; provider_request_id?: string; details?: {}; } ``` #### A.6 Adapter envelope and result (gateway to adapter and back) `libs@49728e6:packages/integration-contracts/src/generated/types.ts:261-435` (doc comments removed; fields verbatim): ```ts export interface AnyAdapterEnvelope { envelope_version: 1; execution_id: string; attempt: number; capability: string; major: number; minor: number; mode: InvocationMode; tenant: { id: string; time_zone: string; }; connection: { id: string; external_account_ref: string | null; configuration: {}; installation_settings: {}; features?: { [k: string]: string[]; }; }; credential: CredentialLease; request: {}; idempotency_key: string; correlation: { correlation_id: string; trace_id?: string; originating_user_id?: string; }; deadline_at: string; rhythm: { service_function_arn: string; principal: string; }; } export type AnyAdapterResult = | { outcome: 'succeeded'; response: {}; provider_ref?: string; provider_request_id?: string; cost_units?: number; } | { outcome: 'accepted'; provider_ref?: string; } | { outcome: 'failed'; error: ContractError; provider_request_id?: string; } | { outcome: 'unknown'; error: ContractError; provider_ref?: string; }; ``` #### A.7 Credential lease `libs@49728e6:packages/integration-contracts/src/generated/types.ts:159-192` (doc comments removed): ```ts export type CredentialMethod = | 'oauth2_authorization_code' | 'oauth2_client_credentials' | 'api_key' | 'platform_account' | 'basic' | 'custom' | 'aws_assumed_role' | 'azure_client_secret'; export interface CredentialLease { method: CredentialMethod; lease_id: string; expires_at: string; values: { [k: string]: string; }; } ``` #### A.8 Manifest: capabilities and rate limits `libs@49728e6:packages/integration-contracts/src/generated/types.ts:2215-2250`, `:2372-2390` (doc comments removed): ```ts capabilities: { key: string; major: number; features?: string[]; modes: ['sync' | 'async', ...('sync' | 'async')[]]; constraints?: { currencies?: 'any' | [string, ...string[]]; min_amount_minor?: number; max_amount_minor?: number; countries?: [string, ...string[]]; }; }[]; ``` ```ts rate_limits: { bucket: string; per_second?: number; per_minute?: number; per_hour?: number; burst?: number; concurrency?: number; capabilities: [string, ...string[]]; }[]; ``` `platform_account` method entry (doc comments removed), same file in the `auth_methods` union: ```ts | { authentication_instructions?: { markdown: string; }; method: 'platform_account'; platform_secret: string; account_qualifier: { field: string; label: string; pattern?: string; }; } ``` #### A.9 Connection (Drizzle table) `services-platform:modules/integrations/db/integration-connections.table.ts:79-105`, `:109-152` ```ts export interface VerifiedCapability { readonly supported: boolean; readonly features: readonly string[]; readonly verified_at: string; readonly notes?: string; } /** The whole of `verified_features`: by `@`. */ export type VerifiedFeatures = Readonly>; ``` ```ts export const integrationConnections = platformSchema.table( 'integration_connections', { /** A configuration id (`integration-ids.ts`): immutable, in the grammar of `configuration-ids`. */ id: text('id').notNull(), tenant_id: text('tenant_id').notNull(), installation_id: text('installation_id').notNull(), /** Denormalized from the installation, for the catalog's and discovery's queries. */ integration_definition_id: text('integration_definition_id').notNull(), /** The pin: the version that serves this connection (`09-domain-model.md` §5). */ integration_version_id: text('integration_version_id').notNull(), name: text('name').notNull(), /** The provider's own id of the account (`acct_…`); set by verification. */ external_account_ref: text('external_account_ref'), /** One of the definition's `auth_methods`. */ auth_method: text('auth_method').notNull(), // (account_environment, with its doc comment, at :125-129) account_environment: text('account_environment', { enum: CONNECTION_ACCOUNT_ENVIRONMENTS }), /** The active credential (`integration_credentials`); null until there is one. */ credential_id: text('credential_id'), /** The connection-level settings the manifest's `connection_settings_schema` declares. */ configuration: jsonb('configuration') .notNull() .default(sql`'{}'::jsonb`), verified_features: jsonb('verified_features') .$type() .notNull() .default(sql`'{}'::jsonb`), status: text('status', { enum: CONNECTION_STATUSES }).notNull().default('pending_auth'), health: jsonb('health').$type(), labels: text('labels') .array() .notNull() .default(sql`'{}'::text[]`), /** A hint: which Accounting business unit the account belongs to (a plain text id, no foreign key across services). */ business_unit_id: text('business_unit_id'), disconnected_at: timestamptz('disconnected_at'), disconnected_by_id: text('disconnected_by_id'), disconnect_reason: text('disconnect_reason'), ...snakeSystemColumns(), }, ``` (The `// (account_environment …)` line is mine, replacing the five-line doc comment of that column.) #### A.10 Binding (Drizzle table and contracts policy) `services-platform:modules/integrations/db/integration-bindings.table.ts:96-120` ```ts export const integrationBindings = platformSchema.table( 'integration_bindings', { /** A configuration id (`integration-ids.ts`): immutable, in the grammar of `configuration-ids`. */ id: text('id').notNull(), tenant_id: text('tenant_id').notNull(), /** `payments.authorize`. */ capability_key: text('capability_key').notNull(), capability_major: integer('capability_major').notNull(), scope_kind: text('scope_kind', { enum: BINDING_SCOPE_KINDS }).notNull(), /** `*`, an application key, a business-unit id, or `:`. */ scope_ref: text('scope_ref').notNull(), connection_id: text('connection_id').notNull(), /** Features every invocation through this binding needs, checked against the connection's `verified_features`. */ required_features: text('required_features') .array() .notNull() .default(sql`'{}'::text[]`), policy: jsonb('policy').$type().notNull(), revision: integer('revision').notNull().default(1), status: text('status', { enum: BINDING_STATUSES }).notNull().default('active'), invalid_reason: text('invalid_reason', { enum: BINDING_INVALID_REASONS }), note: text('note'), ...snakeSystemColumns(), }, ``` `libs@49728e6:packages/integration-contracts/src/binding-policies.ts:14-32` ```ts export type ScopeKind = 'resource' | 'business_unit' | 'application' | 'tenant'; export type BindingScope = | { readonly kind: 'resource'; /** The object whose records are bound (`accounting.merchant_accounts`). */ readonly object_id: string } | { readonly kind: 'business_unit' } | { readonly kind: 'application' } | { readonly kind: 'tenant' }; export interface BindingPolicy { /** The capability keys the policy governs, at the major. */ readonly capabilities: readonly string[]; readonly major: number; /** The allowed scopes, most specific first; empty for an explicit-connection-only capability. */ readonly scopes: readonly BindingScope[]; readonly fallback: 'refuse' | 'walk'; } /** The order scopes are checked in, most specific first. */ export const SCOPE_ORDER: readonly ScopeKind[] = Object.freeze(['resource', 'business_unit', 'application', 'tenant']); ``` `binding-policies.ts:110`: ```ts { capabilities: keysOf('crm.'), major: 1, scopes: [APPLICATION, TENANT], fallback: 'walk' }, ``` #### A.11 Adapter-side handler and context (SDK) `libs@49728e6:packages/integration-sdk/src/handlers.ts:25-45` ```ts export type Handler = (request: Req, ctx: HandlerContext) => Promise; export interface CapabilityHandler = MajorOf> { readonly [KIND]: 'capability'; readonly key: K; readonly major: M; readonly handle: Handler, ResponseOf>; } /** A capability's handler, typed by its contract; refuses a key and major the contracts do not define. */ export function defineCapabilityHandler>( key: K, major: M, handle: Handler, ResponseOf>, ): CapabilityHandler { if (!capabilityDefinition(key, major)) throw new AdapterError('handler_invalid', `the contracts define no capability ${key}@${major}`); if (key.startsWith('connection.')) throw new AdapterError('handler_invalid', `${key} is a connection handler: define it with defineConnectionHandlers`); if (typeof handle !== 'function') throw new AdapterError('handler_invalid', `the handler of ${key}@${major} is not a function`); return Object.freeze({ [KIND]: 'capability' as const, key, major, handle }); } ``` `libs@49728e6:packages/integration-sdk/src/integration.ts:105-114` (the input of `defineIntegration`) ```ts export interface IntegrationDefinition { readonly manifest: DefinedManifest | IntegrationManifest; readonly connection: ConnectionHandlers; readonly capabilities: readonly CapabilityHandler[] | Readonly>>; readonly webhooks?: readonly WebhookHandler[] | Readonly>; readonly events?: readonly EventReceiver[] | Readonly>>; readonly sync?: readonly SyncResourceHandler[] | Readonly>>; /** The adapter's stub provider (`src/stub/stub-provider.ts`), which stub mode routes the HTTP client to. */ readonly stub?: () => HttpTransport; } ``` `HandlerConnection` and `HandlerContext`: `libs@49728e6:packages/integration-sdk/src/context.ts:19-66` (fields listed in section 8.2; quoted here without doc comments): ```ts export interface HandlerConnection { readonly id: string; readonly external_account_ref: string | null; readonly account_qualifier: string | null; readonly configuration: Readonly>; readonly installation_settings: Readonly>; } export interface HandlerContext { readonly execution_id: string; readonly provider_idempotency_key: string; readonly attempt: number; readonly capability?: string; readonly major?: number; readonly idempotency_key: string; readonly tenant: { readonly id: string; readonly time_zone: string }; readonly connection: HandlerConnection; readonly credential: CredentialLease; readonly correlation: { readonly correlation_id: string; readonly trace_id?: string; readonly originating_user_id?: string }; readonly deadline: Date; readonly signal: AbortSignal; readonly logger: RhythmLogger; readonly rhythm: RhythmClient; readonly features: readonly string[]; readonly manifest: DefinedManifest; } ``` #### A.12 Payload recording rule (the personal-data gap) `services-platform:modules/integrations/services/executions/payload-recording.ts:30-37` ```ts /** Whether an attempt of the capability is recorded: the manifest asks for it and the registry does not mark it sensitive. */ export function recordsPayloads( manifest: { readonly record_payloads: boolean }, definition: Pick, ): boolean { return manifest.record_payloads && !definition.sensitive; } ``` --- ### Gaps relevant to enrichment (Inferred judgement) Everything in this section is **Inferred** from the evidence above unless it cites a Verified fact. #### Reusable as-is 1. **Provider packaging and lifecycle.** One repository per enrichment provider (`integrations-`), the SDK, conformance, `integration publish`, versions pinned per connection, deprecation and retirement. A FullContact-style adapter is an ordinary adapter: `api_key` (customer's key) or `platform_account` (Rhythm's key), one or more capability handlers, `connection.verify`/`health`/`describe`. 2. **Installation, consent, connection, credential storage, leases.** Secrets Manager, broker leases, rotation, audit; nothing enrichment-specific is needed for a customer-provided key. 3. **The synchronous gateway call.** `invoke(ctx, { capability, major, context: { subject_ref }, request })` from the consuming application gives typed requests and responses, classified errors, retries within 25 s, circuit breaking, per-connection rate limiting, and an execution row per call with `subject_ref` linking it to the contact. That is enough for "enrich this one contact now". 4. **Async mode** (`either`/`async` contracts), `executions.complete` and `execution.completed` events, for providers that answer later (bulk jobs, webhooks back). 5. **Integration events and consumer registration**, if a provider pushes updates (for example a change in a person's employer) by webhook: ingress, verification, dedup, routing to the consuming application. 6. **Execution history and Library screens** (Activity, Execution detail, Connection detail, Bindings with Explain, Needs attention) show enrichment calls with no console work beyond labels. 7. **The CRM ownership rule** in chapter 21: the integration returns fields; CRM writes them with `app.source = 'integration'`. That keeps enrichment writes inside CRM's persistence, constraints and change events. #### Needs extending or deciding 1. **The capability contract.** `crm.contact.enrich@1` is address-in, fields-out. Person/company enrichment needs a different request (e-mail, phone, name, social handle, domain, company name) and response (typed, per-field confidence and source, possibly multiple candidates). Options: a new major `crm.contact.enrich@2`, new keys (`crm.organization.enrich`, `crm.contact.match`), or a new category. Any of them is a contracts release plus a Platform upgrade to that contracts version (Verified gate, registration step 3). A `search` verb does not exist (Verified). 2. **Idempotency and caching.** The contract is `idempotency: 'none'` and has no lookup, so repeated enrichment of the same input is never deduplicated and an `unknown` outcome goes straight to `needs_attention` (Verified reconciler rule). A paid provider needs either `optional` idempotency with a content-derived key, a cache of answers outside the gateway, or both. 3. **Multi-provider choice.** Bindings choose exactly one connection per scope; `crm.*` allows only `application` and `tenant` scopes. A waterfall (try provider A, then B for missing fields), per-field provider preference, or answer merging is not expressible and would need either an orchestration layer above the gateway (in CRM or a Platform enrichment module) or a resolver extension. 4. **Rhythm-managed provider accounts.** `platform_account` serves "Rhythm's key", but every tenant still needs an installation and a connection with an `account_qualifier`, and rate limits are per connection, so a shared provider quota is not protected across tenants (Verified key shape). Needs either a platform-wide bucket (a new `rate_limits` scope or a key without the connection id) or an automatic per-tenant connection provisioned by Rhythm, plus an answer to "what is the qualifier". 5. **Metering, allowance and billing.** Integrations record only `cost_units` and never act on it (Verified). Enrichment sold or included as credits needs a ledger: allowance per tenant per period, reservation before the call, settlement after, threshold notices, staff overrides, daily rollups for HQ. `modules/llm` already implements this pattern (Verified tables) and is the natural model; whether to generalize it or copy it is a decision. 6. **Personal data.** `personal_data: true` is not enforced (Verified): Platform must refuse `record_payloads` for personal-data capabilities before an enrichment adapter ships. Retention of enrichment answers, provider terms (storage limits, attribution), consent and data-subject requests also have no home in the integration framework today; CRM owns the consent ledger (memory note: Andrew 2026-10-20, Marketing D-M2). 7. **Bulk and list enrichment.** The sync path is per record. Batch enrichment of a list (thousands of contacts) would run through async executions whose consumer has reserved concurrency 5 (Verified), per-connection rate limits, and one execution row per contact; the data job engine (PLAT-M19..M22) or a batch capability (`either` mode with a list request) is the likely carrier. The sync engine (WS10) and data grants (WS9) that would let an adapter read CRM records are not built (Verified). 8. **Health probes cost money.** Daily verification and 15-minute health probes call the adapter per connection (Verified schedule); the enrichment adapter's `connection.verify`/`health` must use free provider endpoints. 9. **Registration gate for a new provider key.** Adding `REGISTERING_ADAPTER_KEYS` entries is a Platform code change and deploy (Verified), so "add an enrichment provider without touching Platform" is not yet true for a new key. 10. **Console.** No CRM data-quality or enrichment screen exists; the Library shows connections and executions generically; there are no HQ integration screens for cross-tenant views of enrichment spend or failures (Verified absence). 11. **Contract scope of `fields_requested`.** The current contract lets the provider's own field names leak through (`fields_requested` are "the provider's field names"; response `fields` keyed by them). That makes CRM's mapping provider-specific. A Rhythm-defined field vocabulary in the contract would keep providers swappable, which is the framework's stated purpose (chapter 10 §1: "The contract is the only thing they share"). ## 19. Evidence r2-crm: Rhythm 2.0 CRM as the target of Data Enrichment ## Evidence r2-crm: Rhythm 2.0 CRM as the target of Data Enrichment Read-only investigation for the Data Enrichment architecture (filling and refreshing contact and organization data from external providers, the successors of Rhythm 1.0's FullContact integration). Written 2026-10-22. **Sources and the commits read** | Short name in citations | What it is | Commit | | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------ | | `services-crm:` | services-crm `origin/master`, extracted with `git archive` (the local checkout was 354 commits behind and was not used) | `e818ee48` | | `crm-arch:` | RHYTHM-CRM-ARCH-001, worktree `services-platform/crm-architecture`, `docs/proposals/crm/` | `d63f10b5d` (2026-10-19) | | `households:` | worktree `services-platform/households-architecture`, `docs/proposals/households/` | `67f223139` (2026-10-21) | | `services-platform:` | services-platform `origin/master` | `43478ffda` | | `pkg:@` | the shared library as installed in services-crm `node_modules`; the version equals the one pinned by services-crm `package.json` at `e818ee48` (`package.json:82-102`) | as stated | | `marketing-arch:` | services-platform branch `origin/marketing-architecture` | branch head | | `services-marketing:` | services-marketing `origin/master` | `b26e4fa` | | `rhythm-web:` | rhythm-web working tree | `776aede` | **Labels.** **Verified**: read in code, DDL or tests at the cited path. **Planned**: in a proposal or milestone document, not built. **Ruled**: a dated decision by Andrew recorded in a document. **Inferred**: my reading, not stated anywhere. **Absent**: searched for and not found. --- ### Summary for the enrichment architect 1. **The fields an enrichment provider fills already exist and nothing fills them.** Contacts carry `image_url`, `job_title`, `facebook`, `twitter`, `instagram`, `linked_in`, `gender`, `pronouns`, `date_of_birth`, three civic districts with OCD ids, and two provider-era timestamps `date_profile_last_updated` and `date_social_profile_last_updated` (`services-crm:modules/entities/db/contacts.table.ts:84,111-130`). Organizations carry `description`, `website`, `domain_name`, `alternate_domain_names`, `logo_url`, `facebook`, `twitter`, `linked_in`, `date_profile_last_updated` (`organizations.table.ts:66-89`). There is **no** industry, employee count, revenue, founded year, headline/biography or location-of-person column. CRM's own debt note says "The fields are stored; nothing fills them" (`services-crm:docs/debt/image-files-and-enrichment.md:8`). **Verified.** 2. **The architecture already says where enrichment goes.** RHYTHM-CRM-ARCH-001 recommends "Integration Platform connectors writing through CRM's service functions; no CRM-internal vendor calls. Low priority" (`crm-arch:05-functional-architecture.md:349-350`), lists enrichment among "Later candidates (not scheduled until Andrew decides)" (`crm-arch:15-completion-roadmap.md:217`) and as gap L1-12 (`crm-arch:14-gap-matrix.md:273`). **Planned (recommendation, not a ruling).** 3. **The service function enrichment would write through does not exist.** CRM publishes one service function, `tasks` (`services-crm:aws/crm.service-functions.ts:21-27`); the `contacts` and `organizations` functions (read, upsert, match-or-create, lookup by domain) are "**Missing**" (`crm-arch:06-api-and-service-architecture.md:47-48`) and are CRM-M15 scope (`crm-arch:15-completion-roadmap.md:131-146`). **Verified absent; Planned.** 4. **A service-principal write to a contact would fail today.** The execution context maps a service caller to change source `service` (`pkg:macroservice-execution@0.9.0 dist/execution-context.js:186-198`), but `crm.record_revisions` only accepts six sources and deliberately leaves `service` out "until a captured record gets a service writer" (`services-crm:modules/entities/db/record-revisions.table.ts:67-81`, check at `:157`). Because the revision write fails closed (`docs/record-history.md:96-106`), the business write would be refused. Enrichment needs this widened (or to run as a CRM job). **Verified.** 5. **Provenance exists only for addresses and phones, and only for organization-sourced values.** `source_mode` is `owned | copied | linked | disconnected` with `copied_from_organization_id`, `copied_at` and `disconnected_*` (`contact-addresses.table.ts:63,119-131`; `contact-phone-numbers.table.ts:50-58`). Scalars (names, socials, photo, job title) have **no** field-level source, confidence or last-verified column. The record-level `origin_service` / `origin_service_record_id` say where the record came from, not where a value came from (`contacts.table.ts:131-132`). **Verified; field-level provenance Absent.** 6. **Change attribution is strong at the save level.** Every committed save writes one `crm.record_revisions` row with field-level before/after, actor id and display-name snapshot, `sys_source`, `sys_operation_name`, correlation and request ids, and the data job and batch (`record-revisions.table.ts:110-145`). Contacts and organizations both keep history (`services-crm:db/change-event-installs.ts:44-78`). This is where "who/what changed this field" is answered today; enrichment would show as a distinct source/operation/job. **Verified.** 7. **A "needs verification" pattern already exists**: D-25 disconnected addresses/phones answer `needs_verification: true` and offer explicit `verify` / `relink` / `remove` mutations at the version read (`services-crm:docs/disconnected-information.md:33-67`). It is the closest precedent for "suggested value awaiting review". **Verified.** 8. **Identity matching is mature and reusable.** Deterministic match keys (`email`, `phone`, `name_sound`, `name_sound_postal`, `name_dob`, `origin_id`; organizations `org_name`, `org_name_sound_postal`, `org_domain`, `org_phone`, `origin_id`) stored in `crm.contact_match_keys` / `crm.organization_match_keys`, explained scoring with bands, free-mail domain exclusion, a frequency cap, and a synchronous check route `POST /duplicates/{tenantId}/{objectType}/checks` (`services-crm:docs/merging/duplicate-detection.md:27-160`; `modules/duplicates/aws/api/duplicates.routes.ts:98,539`). No ML, no LLM (decided). **Verified.** 9. **Bulk "apply enrichment" can ride the Data Job Engine.** CRM registers contact `update` (contract = the PATCH body), `insert`/`upsert`, organization `insert`/`upsert`/`update`, with per-row outcomes, a `prepare` stage for files, version capture, job attribution on every revision, and rollback planned from revisions (`services-crm:modules/data-jobs/services/operations/data-operations.ts:1-25`, `contact-update.ts:1-30`, `record-files.ts:1-25`). Rollback restores a record's own fields only; collections and custom fields are `not_reversible_operation` (`docs/debt/rollback-of-collections.md:1-20`). **Verified.** 10. **Photos and logos must go through the asset pipeline.** A new external URL is imported to permanent storage under SSRF guards, 10 MiB / 10 s limits and content sniffing, before the record's transaction; Rhythm 1.0 stored FullContact avatars as pointers and 2.0 deliberately does not (`services-crm:docs/migrations/asset-persistence/README.md:51,137,172-197`; `services-platform:docs/assets.md:202-219`). **Verified.** 11. **Custom fields are available for provider attributes CRM has no column for** (industry, size, revenue), on contacts and organizations only (R2-ARCH-001), stored in `crm.contact_custom_field_values` / `crm.organization_custom_field_values` and written through the record's own save via the `custom_fields` map (`services-platform:docs/standards/custom-field-capability.md:32-34,127-141`). **Verified.** 12. **Business events are thin**: only `crm.contact.merged` is published; `created/updated/deleted` for contacts and organizations are planned (`services-crm:docs/business-events.md:12-16`; `crm-arch:06-api-and-service-architecture.md:81`). Internally, change events drive duplicate detection, regions, compliance and Quick Search on every save (`services-crm:aws/crm.change-events.ts:11-35`). An enrichment trigger on "contact created" would today need a CRM change-event handler or the planned business event. **Verified / Planned.** 13. **Consent ledger is ruled to CRM but not built.** Andrew's Marketing D-M2 (2026-10-20) puts subscription types, subscriptions, consent history and global unsubscribe in services-crm (`marketing-arch:docs/proposals/marketing/decisions-2026-10-20.md:9`); what exists is `email_opt_out`, `do_not_mail`, `do_not_call` and `contact_communication_preferences` (`contacts.table.ts:133-136`). **Ruled; ledger Absent.** 14. **Member self-service is not built in CRM.** rhythm-web's member portal calls `/contacts/{tenantId}/portal/me...` routes that are "requests" (`rhythm-web:packages/web-contacts/API-ASSUMPTIONS.md:1-12`); no such route exists in services-crm (search of `modules/*/aws/api`). Member-entered values would be distinguishable only by revision `sys_source = portal`, never per field. **Verified absent.** 15. **Merge treats enrichment-type fields as plain scalars**: socials, photo, job title, profile timestamps are `kind: 'scalar'` (staff choose), consent flags are `or` (never widened) (`services-crm:modules/merges/services/planning/contact-merge-model.ts:88-106`). A future enrichment table keyed by contact id must register a merge policy or the coverage test fails (`crm-arch:07-data-quality-duplicates-and-merge.md:80-81`). **Verified.** --- ### 1. Contact and organization records #### 1.1 Contacts (`crm.contacts`) Declared at `services-crm:modules/entities/db/contacts.table.ts:67-173`; the full column list is quoted in Appendix A.1. **Verified.** | Group | Columns (line) | Notes | | ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Key and tenant | `tenant_id` (70), `id` (71), `number` (73) | Key `(tenant_id, id)` (142); `number` is the server's per-tenant sequence, never changes (72); a client-less create gets a UUIDv7 (15-16) | | Lifecycle | `status` enum `active, duplicate, deceased, inactive` (52, 74); archive columns `archived_at`, `archived_by_id`, `archive_reason`, `unarchive_reason` (138; `archive-columns.ts:32-37`) | `inactive`/`deceased` release the primary e-mail and pickers (55-60). Archive is not a status (`archive-columns.ts:1-4`) | | Name parts | `prefix`, `salutation`, `first_name`, `middle_name`, `last_name`, `maiden_name`, `nickname`, `suffix`, `credentials` (75-83) | First or last name required (`contacts_first_or_last_name_check`, 165-168). A name is never parsed (20) | | Computed names | `name`, `full_name`, `name_last_first` (86-90) | Server-owned, recomputed every write; a client may not send them (17-20, 247-256) | | Work | `job_title` (84) | Kept equal to the primary affiliation's title in both directions (`docs/affiliations/README.md:404-407`; `contacts.ts:2380`) | | Classification | `type_id` (92), `region_id` (93), `override_automatic_region_assignment` (94) | Type → `crm.contact_types`; region set by the regions module | | Civic districts | `national_district`, `state_upper_district`, `state_lower_district` and their `_ocd_id` (111-116) | Declared `read_only` in metadata "from the civic district lookup" (`metadata/contacts.object.ts:61-72`); nothing writes them (Google Civic not ported) | | Relationship | `spouse_id` (118) | Interim, Ruled D-19 (`crm-arch:05-functional-architecture.md:64-65`) | | Photo | `image_url` (119) | Metadata `dataType: 'image'` → managed by the asset pipeline (`contacts.object.ts:56`, `:142`) | | Personal | `gender` enum `male, female` (62, 120), `pronouns` (121), `date_of_birth` (122), `date_of_death` (123), `notes` (124) | DOB, DOD, gender, pronouns are `neverIndexed` in Quick Search as "personal data" (`queries/contacts.query.ts:355-361`) | | Social | `facebook` (125), `twitter` (126), `instagram` (127), `linked_in` (128) | Free text, no format rule found | | Provider-era timestamps | `date_profile_last_updated` (129), `date_social_profile_last_updated` (130) | Client-writable timestamps (`aws/api/contacts.schemas.ts:191-192`); no server code sets them (search of `modules/`, `lib/`) | | Record source | `origin_service` (131), `origin_service_record_id` (132) | Read-only on replace ("the record source ... is bookkeeping (M1-D5)", `contacts.ts:358-372`); excluded from metadata and history (`contacts.object.ts:132-140`; `db/change-event-installs.ts:47-53`); kept on the merge alias (`merged-record-aliases.table.ts:54-56`) | | Consent flags | `do_not_mail`, `do_not_call`, `email_opt_out` (133-135), `communication_preferences_last_confirmed` (136) | Never widened by a merge (D6) | | System bookkeeping | `sys_region_address_fingerprint` (103), `sys_create_digest` (110) | Never on the wire | | Standard system columns | `sys_version`, `sys_created_at`, `sys_created_by_id`, `sys_last_modified_at`, `sys_last_modified_by_id`, `sys_external_id`, `sys_locked` (139) | `pkg:macroservice-records@0.5.1 dist/system-columns.js:63-71`. `sys_external_id` holds an import's source identity (`contact-import.ts:21-26`) | **Not stored on the contact** (`contacts.table.ts:22-30`): the organization (it is the primary affiliation's), any address/phone link, `organization_name`, `preferred_*` copies, `user_ids`, `merged_ids`, `custom_field_values` (own table), the legacy `sys_search_*` keys. **Verified.** **Absent on contacts** (searched the table and migrations): headline, biography/bio, personal website, location/city of the person outside addresses, seniority, department, employment history beyond affiliations, GitHub or other social networks, social handles' verification, avatar source, enrichment status/date, provider id, confidence. #### 1.2 Organizations (`crm.organizations`) Declared at `services-crm:modules/entities/db/organizations.table.ts:56-129`; quoted in Appendix A.2. **Verified.** | Group | Columns (line) | Notes | | ------------ | --------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Key | `tenant_id` (59), `id` (60), `number` (62) | Separate number sequence from contacts (53-54) | | Identity | `name` NOT NULL (63), `acronym` (66), `type_id` (65) | Type → `crm.organization_types`, composite FK | | Descriptive | `description` (67), `notes` (68) | Quick Search never indexes `notes`, `description` (`queries/contacts.query.ts:520`) | | Lifecycle | `status` `active, duplicate, inactive` (47, 69); archive columns (93) | | | Web | `website` (70), `domain_name` (71), `alternate_domain_names text[]` (72) | Website and domain fill each other on save (`db/organization-domain-name.ts:1-19`); `alternate_domain_names` excluded from metadata ("an open string list", `metadata/organizations.object.ts:79`) | | Hierarchy | `parent_organization_id` (74) | Never a descendant (trigger, migration 0018) | | Logo | `logo_url` (75) | `dataType: 'image'`, managed file (`organizations.object.ts:45`) | | Other | `flags text[]` (76, excluded from metadata), `tax_exempt` (77), `region_id`, `override_automatic_region_assignment` (78-79) | | | Social | `facebook` (86), `twitter` (87), `linked_in` (88) | No `instagram` on organizations | | Timestamp | `date_profile_last_updated` (89) | Client-writable (`organizations.schemas.ts:76`) | | Server-owned | `contact_role_compliance_status` (91), `sys_region_address_fingerprint` (85) | | | System | `snakeSystemColumns()` (94) | | **Absent on organizations**: industry, NAICS/SIC, employee count, revenue, founded year, ticker, organization e-mail addresses (Recommended addition, `crm-arch:05-functional-architecture.md:77-79`), phone `is_preferred` (Recommended, same lines), any provider id or enrichment timestamp beyond `date_profile_last_updated`. The proposal's domain lookup by e-mail domain is "Partial" — normalisation, alternate domains, free-mail list and the `org_domain` key exist, no lookup route or service function (`crm-arch:14-gap-matrix.md:37`; `05-functional-architecture.md:90-93`). **Verified / Planned.** #### 1.3 What a provider would typically fill, mapped to what exists (Inferred) | Provider attribute (typical of FullContact successors) | CRM home today | Label | | -------------------------------------------------------- | --------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | | Person full name, given/family name | name parts; computed names are server-owned | exists | | Photo / avatar | `contacts.image_url` via asset import | exists | | Job title | `contacts.job_title` = primary affiliation `job_title` | exists | | Employer name / domain | an affiliation to an organization (no free-text employer) | exists, but needs organization resolution (I-10, `crm-arch:04-domain-and-data-model.md:127`) | | LinkedIn, Twitter/X, Facebook, Instagram URLs | `linked_in`, `twitter`, `facebook`, `instagram` | exists | | Other networks (GitHub, YouTube, TikTok...) | none | Absent → custom field or new table | | Gender, pronouns | `gender` (`male`/`female` only), `pronouns` | exists, narrow enum | | Age range / birth date | `date_of_birth` | exists (sensitive; `neverIndexed`) | | Person location (city/region/country) | only as an address row of a type | exists only as a typed address | | Phones, e-mails | typed phone rows; three untyped e-mail slots | exists, with limits (§2) | | Bio / headline | none | Absent | | Organization logo, description, website, domain, socials | `logo_url`, `description`, `website`, `domain_name`, `linked_in`, `twitter`, `facebook` | exists | | Organization industry, size, revenue, founded | none | Absent → custom field | | Organization address / phone | `organization_addresses`, `organization_phone_numbers` by type | exists | | "Last enriched" / "last verified" date | `date_profile_last_updated`, `date_social_profile_last_updated` (semantics unspecified) | partially exists, unused | --- ### 2. E-mail, phone and address lists All are contact-owned tables: key `(tenant_id, id)`, owner FK `(tenant_id, contact_id)` `ON DELETE CASCADE` (`contacts.table.ts:194-217`), written only as collections of the contact save. **Verified.** #### 2.1 E-mail addresses (`crm.contact_email_addresses`) - At most three; `position` 1..3 is the slot, server-assigned from the order sent, contiguous; position 1 is primary (`contact-email-addresses.table.ts:2-5,36,46,59`). **Verified.** - **No type** (work/personal). Whether to add configurable e-mail types is open decision **CRM-Q6**, recommended yes (`crm-arch:13-decisions.md:116-127`), scheduled under CRM-M14 (`crm-arch:15-completion-roadmap.md:123-129`). **Planned (open).** - Canonical form: the domain `crm.email_address` refuses anything but lowercase without whitespace; `normalizeEmailAddresses` canonicalises and drops blanks (RB-1699); `emailAddressProblems` validates format and count (`contact-email-addresses.table.ts:7-9`; `contacts.ts:538-560`). **Verified.** - **Primary uniqueness**: a primary address belongs to one non-released contact per tenant; inactive/deceased contacts release it (CRM-M1 M1-D1, Andrew 2026-10-19); enforced by constraint triggers with a tenant+address advisory lock, refused as `IllegalOperation` (`contact-email-addresses.table.ts:11-23`; `contacts.ts:564-580`). Secondary slots are not compared. **Verified / Ruled.** - **No verification, deliverability or bounce status** on the row (columns: Appendix A.3). Bounce/complaint suppression lives in Platform communications ("today Platform communications suppresses only bounces and complaints", `crm-arch:05-functional-architecture.md:56-58`). **Absent in CRM.** #### 2.2 Phone numbers (`crm.contact_phone_numbers`) - One row per tenant phone-number type (`contact_phone_numbers_one_per_type_uq`), at most one `is_preferred` (`contact-phone-numbers.table.ts:64-67`). Types are configuration (`crm.phone_number_types`: `name`, `applies_to` contacts/organizations/both, `display_order`; `db/collection-types.ts:37-44`). **Verified.** - `phone_number` formatted by the server (US national format), required unless the row is `linked` (`contact-phone-numbers.table.ts:47-48,120-123`; `contacts.ts:611-620`). Blank numbers dropped; default preferred = first by the tenant's type order. **Verified.** - **Source mode** and provenance: `source_mode` (`owned`, `copied`, `linked`, `disconnected`), `source_affiliation_id`, `source_organization_id`, `source_organization_phone_number_id`, `copied_from_organization_id`, `copied_at`, `disconnected_from_organization_id`, `disconnected_at`, `disconnected_reason` (`:50-58`), with a CASE check holding each mode's columns (`:99-114`). **Verified.** - No line type (mobile vs landline as a fact), carrier, validity or DNC status columns. **Absent.** #### 2.3 Addresses (`crm.contact_addresses`) - One row per tenant address type, at most one preferred, seasonal `MM-DD` start/end (E-06), a seasonal address is never preferred (`contact-addresses.table.ts:92-224`, checks `:152-170`). **Verified / Ruled E-06.** - Values: `line1..line3`, `city`, `state`, `postal_code`, `county`, `country` (default `US`), `latitude`, `longitude`, `time_zone_id` (`:98-108`). Country and state cleaned; a non-ISO country refused (M1-D3) (`contacts.ts:631-657`). - Same `source_mode` / link / copy / disconnected provenance as phones (`:119-131`); a linked row holds **no** values (`contact_addresses_linked_values_check`, `:216-221`) and is resolved through `crm.contact_addresses_effective`. - **Address verification (a postal service) is out of scope** for the CRM architecture "and can be added behind the same validator" (`crm-arch:05-functional-architecture.md:166-168`). No verified/standardised/geocoded-by flag exists. **Planned (as an extension point) / Absent.** #### 2.4 Organization addresses and phones `crm.organization_addresses` (one per type, one preferred; same value columns, no source mode) and `crm.organization_phone_numbers` (one per type, **no preferred flag**) (`organization-addresses.table.ts:30-57`; `organization-phone-numbers.table.ts:25-40`). Removing one turns linked contact rows `disconnected` (`docs/disconnected-information.md:44-48`). **Verified.** #### 2.5 The "one blank row" convention This is a **console** convention, not a server rule: Andrew 2026-10-22 ~09:30 ET, "Don't make me click add email ... if you save it and it's empty, then obviously it saves nothing" — show one blank row, never save, validate or dirty it (recorded in the session memory `start-with-one-blank-row.md`; no services-crm document states it). The server side matches it independently: a blank e-mail, phone or address element is **dropped**, not refused (`contacts.ts:538-542,611-620`; schema comment on DEF-CRM-0013 at `aws/api/contacts.schemas.ts` `contactCollectionSchemas`). **Ruled (UI) / Verified (server drop).** #### 2.6 How writes validate them One validator for console, portal, import and API is the principle (`crm-arch:05-functional-architecture.md:166-167`): `normalizeContact` then `assertContactRules` run every table's normalisation and refusals before anything is written, answering `ObjectInvalid` with per-path problems (`contacts.ts:685-716`). The zod boundary derives element schemas from the tables, `.strict()`, with write-only `source_organization_*_type_id` (`aws/api/contacts.schemas.ts:1-110`). **Verified.** Collection semantics: a present collection is the whole collection (`[]` clears); omitted collections on a `PUT` are **Ruled D-09/E-09 to mean "unchanged"** but CRM still replaces an omitted `affiliations` (DEF-CRM-0174) (`services-platform:docs/standards/record-write-contracts.md:134-161`; `crm-arch:04-domain-and-data-model.md:118,125`). Owned rows keep their ids on edit (in-place writes), so history records an edit as `updated` and a replacement as `deleted`+`inserted` (Ruled D-22, `docs/record-history.md:47-77`). **Verified / Ruled.** --- ### 3. Affiliations #### 3.1 Model (`crm.contact_organization_affiliations`) Columns (`contact-organization-affiliations.table.ts:51-79`): `organization_id`, `affiliation_type_id`, `is_primary`, `job_title`, `start_date` (null = unknown), `end_date` (inclusive, null = open), `notes`, plus system bookkeeping `sys_automatic_roles_due` and `sys_default_links_pending` (never on the wire, never in events or revisions, `:64-77,141-156`). Roles are `crm.contact_organization_roles` (`affiliation_id`, `contact_id`, `contact_role_id`, `start_date`, `end_date`; `contact-organization-roles.table.ts:41-54`). **Verified.** Database rules (`:20-31`, constraints `:99-128`): - **D1** at most one primary per contact (`contact_organization_affiliations_one_primary_uidx`); - **D2** a primary affiliation has no end date; - **D3** end ≥ start; - **D4** no overlapping affiliations of one contact to the same organization **of the same type** (GiST exclusion, migration-only). No `is_active`, no lifecycle status: current / past / future are derived from dates on the tenant's business date (`:15-18,163-171`; `docs/affiliations/README.md:286-321`, D-10, D-12, Andrew 2026-10-19). Limits: 100 current and upcoming per contact (ended ones unlimited, M2-D5, Andrew 2026-10-21) and 25 roles per affiliation (`docs/affiliations/README.md:337-340`; `MAX_AFFILIATIONS` `:161`). **Verified / Ruled.** #### 3.2 Affiliation types `crm.affiliation_types`: configuration id, `name`, `description`, `is_active` (`affiliation-types.table.ts:36-50`). Every tenant has `employer`, `school`, `other`; the Organization field defaults to the tenant's `crm_settings.default_affiliation_type_id` (null → `other`) (`docs/affiliations/README.md:247-265`; `crm-settings.table.ts:42-46`). Changing the default is admin-only (Ruled E-03, `crm-arch:11-security-audit-and-operations.md:27`). Each type may map organization address/phone types to contact destination types (`crm.affiliation_type_contact_information_defaults`, `affiliation-type-contact-info-defaults.table.ts:58-...`), applied when an affiliation is created, never overwriting an existing row (`docs/affiliations/README.md:116-122`). **Verified / Ruled.** #### 3.3 Rules in `docs/affiliations/README.md` that constrain enrichment - **The contact stores no organization**; the primary affiliation is the Organization (`:1-9`, `:223-235`). A provider's "employer: Acme" can only land as an affiliation to an existing `crm.organizations` row, never as text (I-10, "No free-text organization name exists on a contact", `crm-arch:04-domain-and-data-model.md:127`). **Ruled 2026-10-13.** - **Written only through the contact aggregate** ("no route, service or job writes an affiliation without saving its contact", `contact-organization-affiliations.table.ts:9-13`; README §4 `:342-371`). **Verified.** - **Picking an organization** (`set_primary_organization: { organization_id | null, effective_date?, previous_affiliation?: 'end' | 'keep' }`) creates or reuses an affiliation of the default type and makes it primary; changing it ends the old primary the day before by default (README §5.1-5.2 `:375-391`, §6 `:498-510`; `ContactInput.set_primary_organization`, `contacts.ts:335-336`). - **A future-dated primary is refused** ("cannot start after today", README §4 rule 3). A provider's "started at Globex in 2027" cannot become primary. - **Job title sync**: `contacts.job_title` ↔ primary affiliation `job_title` both ways; past affiliations keep their title (README §5.5 `:404-407`). Setting `job_title` through enrichment changes the primary affiliation's title too. **Verified.** - **Automatic roles** follow effective affiliations (D-46, Andrew 2026-10-19; README §5.6a `:418-446`), so an enrichment-created affiliation can grant CRM roles and change an organization's compliance (Inferred consequence). - **Organization matching for imports**: a row naming an organization matching two or more organizations is reported, never guessed (1.0 P5f, `crm-arch:10-data-operations.md:49-50`); domain auto-linking only when exactly one active organization matches (`crm-arch:05-functional-architecture.md:90-93`). **Planned (Recommended).** #### 3.4 Write contract for adding and ending an affiliation - Today: by saving the contact — the `affiliations[]` collection (full replacement or JSON Patch, e.g. `replace /affiliations/0/end_date`), or `set_primary_organization` (`aws/api/contact-api-mapper.ts:196-197`). An omitted `affiliations` on `PUT` is treated as `[]` (DEF-CRM-0174). **Verified.** - Ruled target (D-09, E-09; record-write-contracts W3): relationship collections are add/update by id; ending or removing is an **explicit operation with its own permission and audit**; a general save that would end one is refused (`services-platform:docs/standards/record-write-contracts.md:134-161`; `crm-arch:06-api-and-service-architecture.md:26,29`; `crm-arch:11-security-audit-and-operations.md:26`). The explicit routes are **not built** (no affiliation routes in `modules/entities/aws/api/contacts.routes.ts`). **Ruled / not built.** - Reads: `GET /organizations/{tenantId}/{id}/affiliations?effective=...` (`organizations.routes.ts:211`). **Verified.** #### 3.5 Events No affiliation-specific event. Affiliations and roles are collections of the contact's change event and revision (`db/change-event-installs.ts:44-60,129-138`); a save that changes one bumps the contact's version (README §4 `:368-371`). The contact stream's handlers include organization compliance recalculation (`aws/crm.change-events.ts:14-19`). Membership's `membership_crm_change_handler` for affiliation changes is built in Membership but not invoked (`crm-arch:06-api-and-service-architecture.md:80`; `crm-arch:12-cross-application-contracts.md:27`). **Verified / Planned.** --- ### 4. Contact types and organization types - **Contact types** (`crm.contact_types`): configuration id, `name`, `description`, `is_active`, nullable `display_order`, and layout/form assignments (`console_page_layout_id`, `portal_page_layout_id`, `console_create_form_id`, `console_edit_form_id`, `portal_form_id`) (`contact-types.table.ts:1-16,30-53`; `layout-slots.ts:128-133`). A contact holds at most one (`contacts.type_id`, optional). Typical values Member, Staff, Prospect (`:2`). **Verified.** - **Organization types** (`crm.organization_types`): the same plus `restrict_contact_roles`, `applicable_contact_role_ids`, `enable_contact_role_compliance`, `has_contact_role_compliance_options`, `notify_contacts_of_compliance_violation`, `notification_contact_role_ids`, and compliance rules (`organization-types.table.ts:52-117`, `organization_type_compliance_rules` `:118-...`). **Verified.** - **No member / non-member flag** exists on either type, or on contacts. Searched `modules/entities/db/*.ts` for `is_member`, `member_status`, `membership`: nothing. Membership status lives in services-membership (`memberships.contact_id/organization_id`, `crm-arch:12-cross-application-contracts.md:27`). An enrichment eligibility rule "members only" must therefore read Membership (or a type id the tenant chooses), not a CRM flag. **Absent in CRM (Verified by search).** - Other eligibility-relevant facts CRM does hold: `status` (active/inactive/deceased/duplicate), `archived_at`, `sys_locked`, `email_opt_out`/`do_not_*`, `type_id`, `region_id`, affiliation types (Inferred usefulness). --- ### 5. Custom fields - **Which entities**: contacts and organizations only in CRM (R2-ARCH-001, Andrew 2026-10-19 ~18:24 ET; `services-platform:docs/standards/custom-field-capability.md:5-6,32-34`). Activities, affiliations, roles, addresses, phones and e-mails have none. CRM declares `capabilities: { customFields: true, configurableForms: true, configurablePageLayouts: true }` (`metadata/contacts.object.ts:26`; `metadata/organizations.object.ts:23`). **Ruled / Verified.** - **Definitions** live in Platform (`platform.custom_fields`, object `crm.contacts` / `crm.organizations`), created through Platform's `POST /custom_fields` (refused for an object without the capability) (`custom-field-capability.md:101-110,130-133`). Only Andrew decides an entity gets custom fields (`:23-24`). **Ruled.** - **Values**: `crm.contact_custom_field_values` and `crm.organization_custom_field_values`, built by `customFieldValuesTable`: PK `(tenant_id, record_id, custom_field_id)`, FK to the record `ON DELETE CASCADE`, one typed value column filled (`string_value`, `integer_value`, `numeric_value`, `boolean_value`, `file_url_value`, `list_value`, `date_value`, `date_time_value`, `reference_value`, `html_value`), `sys_last_modified_at`, `sys_last_modified_by_id`; key matches `CUSTOM_FIELD_KEY_CHECK`, ≤ 100 chars (`pkg:macroservice-custom-fields@0.2.9 dist/custom-field-values.js:46-94`; `contact-custom-field-values.table.ts:1-33`). Quoted in Appendix A.6. **Verified.** - **Writes**: only through the record's save, as the `custom_fields` map (a value sets, `null` clears, an unnamed key is untouched), resolved against the tenant's definitions in the same transaction; unknown/archived/invalid keys refused by name (`contacts.ts:326-342` `ContactInput.custom_fields`; `createContact` doc `:2067-2095`; `prepareContactCustomFields` `:2027`). Image/document custom fields go through the asset pipeline like the photo. **Verified.** - **Queries, history, merge**: the query registration names the value table (`queries/contacts.query.ts:200`); custom field values are a history collection identified by `custom_field_id` (`db/change-event-installs.ts:58`); the merge plan covers them. **Verified.** - **For enrichment (Inferred)**: provider attributes without a column (industry, size, revenue, other socials) can map to tenant-defined custom fields with no CRM schema change; text values have no length-indexed limit (the string index holds the field, not the value, after the 2026-10-18 rubber.org import failure, `custom-field-values.js:69-77`). There is no field-level provenance on a custom value either (only `sys_last_modified_*`). --- ### 6. Duplicate detection and merging #### 6.1 Matching keys and algorithm (Verified) Design `services-crm:docs/merging/duplicate-detection.md` (status line "draft for review (2026-10-11)", built and accepted in phases, `crm-arch:07-data-quality-duplicates-and-merge.md:3-5`). Code `modules/duplicates/services/matching/`. - **Normalisation in TypeScript** (because forced RLS forbids `lower()`/trigram indexes): names NFKD, diacritics stripped, honorifics removed, nickname canonicalisation from the 1.0 table (1,963 names), Double Metaphone; e-mails as stored; phones E.164 via `awesome-phonenumber`; postal US5; address line 1 USPS abbreviations; organization names minus legal tokens; domains from `domain_name`, `alternate_domain_names`, website host, **free-mail domains excluded** (`duplicate-detection.md:27-37`; `normalization.ts:36-442`; `free-mail-domains.ts`). - **Match keys** (`match-keys.ts:38,42`; table `duplicate-detection.md:41-52`): | Key | Object | Value | | ----------------------- | ------- | ---------------------------------------------------- | | `email` | contact | each e-mail address | | `phone` | contact | each phone in E.164 | | `name_sound` | contact | last-name metaphone + canonical first-name metaphone | | `name_sound_postal` | contact | last-name metaphone + postal5 | | `name_dob` | contact | first 3 letters of last name + date of birth | | `origin_id` | both | `origin_service` + `origin_service_record_id` | | `org_name` | org | sorted normalised token set | | `org_name_sound_postal` | org | metaphone of first two significant tokens + postal5 | | `org_domain` | org | each non-free-mail domain | | `org_phone` | org | each phone in E.164 | Linked addresses/phones are excluded from a contact's own evidence ("would make every employee of one organization look alike", `match-keys.ts:78-84`). - **Frequency cap** 200 records per key value; stale-evidence re-scoring Ruled D-47 (Andrew 2026-10-19) (`duplicate-detection.md:54-77`). - **Scoring**: deterministic, explainable, versioned (`RULE_SET_VERSION = 1`, `match-keys.ts:35`); contact weights (same origin +60, primary e-mail +40, any e-mail +30, mobile +20, names ±, DOB +25/−40, suffix conflict −30, same employer +8, spouse −30) and caps (no name agreement ≤ 49) (`duplicate-detection.md:91-116`); organization weights (origin +60, name exact +35, domain +30, ...) (`:117-131`). Bands very high ≥ 90, high 70-89, medium 50-69, low 35-49 (`:133-141`; `evidence.ts:29-38`). No ML, no LLM (decided; `crm-arch:07-...:19-21`). **Verified / Decided.** #### 6.2 Tables (Verified; Appendix A.7) `crm.contact_match_keys` / `crm.organization_match_keys` (`tenant_id`, `record_id`, `key_type`, `key_value`, `rule_set_version`) (`modules/duplicates/db/match-keys.table.ts:56-98`); `crm.contact_duplicate_candidates` / `crm.organization_duplicate_candidates` (pair, `score`, `band`, `reasons jsonb`, `status` open/dismissed/confirmed, `origin` detected/manual/portal_sign_up, dismissal fields, `group_id`) (`duplicate-candidates.table.ts:46-105,159-181`); `crm.duplicate_rules` (`record_type`, `match_on` email/name_zip/name_phone/org_name_domain, `action` warn/block/review_queue, `check_portal`, `position`, `minimum_band`) (`duplicate-rules.table.ts:40-82`); `crm.duplicate_block_overrides` (`duplicate-block-overrides.table.ts:42-59`); groups and rescans. #### 6.3 When detection runs Asynchronously from contact/organization change events when a match-relevant field changed; imports and bulk updates the same way; a synchronous read-only check route for forms; `block` rules synchronous on **create only**, refusing `409 possible_duplicate` unless an admin-tier key holder sends `confirmed_not_duplicate: true` with a reason (Ruled D-32, Andrew 2026-10-19) (`duplicate-detection.md:152-190`). **Verified / Ruled.** Consequence for enrichment (Inferred): enrichment that adds an e-mail or phone to a contact will re-key it and may raise new candidate pairs; enrichment that **creates** contacts is subject to `block` rules. #### 6.4 Merge flow and participant protocol - N-way (2–10), planner recommends a survivor, staff choose per field and per collection item, one transaction across CRM and every participant schema, merged-away ids answer 410 with the survivor, merge record keeps original values for the tenant's life, no unmerge (`crm-arch:07-data-quality-duplicates-and-merge.md:44-89,134-144`). **Verified.** - **Participant protocol**: each application holding references publishes `.merge_plan_(survivor, merged[]) → jsonb` and `merge_apply_(merge_id, survivor, merged[], decisions) → jsonb`, SECURITY DEFINER, EXECUTE granted only to `rhythm_crm_app`, called inside CRM's transaction with the merge's session context (`app.source = job`); contract `@rhythmsoftware/macroservice-record-merge` (`services-platform:docs/record-merge-participant.md:9-30,114-129`). Platform blocks on two portal logins or resource-scoped grants (`:64-76`). **Verified.** - **CRM-ADR-12 (Andrew, CRM-Q3, 2026-10-19 17:48 ET, option 1)**: a merge is blocked when a merged-away record is referenced by an application with no participant for that object type; the plan names the application and counts; no override; lifts when the participant ships; a CI test fails on an uncovered new referencing column (`crm-arch:13-decisions.md:38`; `services-crm:docs/milestones/CRM-M4-duplicates-and-merge.md:31`). **Ruled.** - **Field policies relevant to enrichment** (`modules/merges/services/planning/contact-merge-model.ts:61-115`): `image_url`, `facebook`, `twitter`, `instagram`, `linked_in`, `job_title`, `date_profile_last_updated`, `date_social_profile_last_updated` are `scalar` (staff choose; `:88-100`); civic districts follow `region` (`:79-86`); `origin_service*` and `sys_external_id` keep the survivor's, merged-away ones kept on the alias (`:101-102,113`; `merged-record-aliases.table.ts:54-56`); `do_not_mail`, `do_not_call`, `email_opt_out` are `or` ("consent is never widened by a merge (decision D6)", `:103-105`); `communication_preferences_last_confirmed` keeps the latest (`:106`). **Verified.** #### 6.5 Identity infrastructure enrichment could share (Inferred from the Verified pieces) - The match keys and the synchronous check (`POST /duplicates/{tenantId}/{objectType}/checks`, `MAX_CHECK_RESULTS = 5`, `duplicates.routes.ts:98,539`; `duplicate-checks.ts:50-105`) answer "which existing contact or organization is this provider record?" with the same rules staff see. - `org_domain` keys plus the free-mail list are the basis of the planned domain lookup (OR-05). - The SQL port of the scorer used by portal sign-up (`crm.portal_contact_candidates`, migration `0077`; `duplicate-detection.md:500-564`) shows how a cross-service caller can get scored candidates without a Lambda hop. - `origin_id` key: a provider-sourced record carrying `origin_service`/`origin_service_record_id` would match its own re-delivery at +60. - The planned **match-or-create** service function: "One function decides whether an incoming person or organization already exists, using the duplicate rules ... Every entry path uses it" (`crm-arch:06-api-and-service-architecture.md:55-58`). **Planned.** --- ### 7. Record history, audit and change tracking #### 7.1 `crm.record_revisions` (Verified; Appendix A.8) - One row per committed save per record, written only by the commit-time settle of the change-event stream (SECURITY DEFINER), application role SELECT-only, no foreign keys, kept for the tenant's life (`record-revisions.table.ts:1-60,110-166`; `docs/record-history.md:1-44`). Approved by Andrew 2026-10-11. - Content: `changes` jsonb with record fields `{before, after}` and collection entries (`op`, `identity`, `fields`), plus `redacted`; `changed_fields` array for filtering (`docs/record-history.md:47-77`). - **Attribution**: `sys_actor_id` (from `app.user_id`), `sys_actor_label` (display-name snapshot), `sys_source` ∈ `console, portal, public, integration, job, maintenance`, `sys_operation_name`, `sys_correlation_id`, `sys_request_id`, `sys_job_id`, `sys_job_batch_id` (`record-revisions.table.ts:129-144`). All from the verified execution context, never from a request (`:43-51`). - **Fail closed**: no `app.source` or no `app.user_id` → the revision write raises and the business change is refused; maintenance runs use `withMaintenanceContext(ctx, { operatorId, reason }, fn)` (`docs/record-history.md:96-106`; migration `0046_record_revisions.sql:683-687`). - **Which records keep history**: contacts (with e-mails, phones, addresses, communication preferences, affiliations, roles, custom field values) and organizations (with addresses, phones, custom field values) and activities (`db/change-event-installs.ts:44-78,113-150`; organizations since migration `20261020130548_organization_history.sql`). `docs/record-history.md:14-17` still says only contacts and activities: stale. Contact history excludes `origin_service`, `origin_service_record_id` and the three `_ocd_id` columns; redacts nothing (DOB kept, Andrew D2 2026-10-11) (`change-event-installs.ts:28-55`). **Verified.** - **History API**: `GET /contacts/{tenantId}/{id}/history[/{revisionId}|/actors]`, filters `actor_id`, `operation`, `source`, `from`, `to`; permission `crm.contacts.history` (tier read, Andrew D1) (`docs/record-history.md:128-171`). Data-job revisions answer `source.job_id` and a label like "Data Job: Delete Contacts" (`:143-146`; Ruled E-08 wants a link to the job, `crm-arch:10-data-operations.md:30`). #### 7.2 Source attribution of a change (Verified) `changeSourceOf` (`pkg:macroservice-execution@0.9.0 dist/execution-context.js:186-198`): maintenance → `maintenance`; a job origin or `system` caller → `job`; portal caller → `portal`; anonymous → `public`; service-function caller → `service`; otherwise `console`. `integration` is "reserved for API-client callers (the platform has no such caller type yet, so nothing maps to it today)" (`pkg:macroservice-db@0.15.0 dist/session-context.js:28-35`). A service caller also carries `app.originating_user_id` (the person the service acts for) (`execution-context.js:178-179`). **Gap**: `RECORD_REVISION_SOURCES` excludes `service` "until a captured record gets a service writer" (`record-revisions.table.ts:67-81`) and the check constraint enforces it (`:157`). A contact write from a future `contacts` service function (the enrichment path the architecture recommends) would therefore be refused until a migration widens the check and the console's `ChangeSourceKind` gains the word. **Verified (gap).** A non-user system actor is possible: portal sign-up writes contacts "attributed to the system actor `portal-sign-up` (source `job`, operation `portal-sign-up`)" (`docs/merging/duplicate-detection.md` §15; migration `0077_portal_sign_up_contacts.sql:11`). The actor label then has no user to resolve and shows as "(Unknown User)" by the stamp rule (`docs/record-stamps.md:3-9`) (Inferred for an enrichment actor). #### 7.3 Field-level provenance, "last verified", other audit (Verified / Absent) - **Field-level provenance**: only `source_mode` and its columns on contact addresses/phones (§2). Nothing on scalars or custom values. **Absent elsewhere.** - **"Needs verification"**: only disconnected address/phone rows (`needs_verification: true`, `source_status: "disconnected"`), with explicit `verify` / `relink` / `remove` mutations under `crm.contacts.update` at the version read (`docs/disconnected-information.md:33-67`; routes `contacts.routes.ts:465-497`). Queryable (`:69-74`). **Verified.** - **Last verified / confirmed timestamps**: `communication_preferences_last_confirmed`, `date_profile_last_updated`, `date_social_profile_last_updated` — client-written, no server semantics (§1.1). **Verified.** - **Stamps**: `sys_created_by_id` / `sys_last_modified_by_id` shown by name (`docs/record-stamps.md:3-9`). - **Lock audit**: `crm.record_lock_changes` (`record-lock-changes.table.ts:27-44`). Customer locks `deletes` / `updates & deletes` refuse writes (`lockedCheck`, `contacts.table.ts:171`; `docs/record-locks.md:9-20`): an enrichment write to a locked contact is refused (Inferred). - **Platform audit events**: `record.viewed`, `record.merged` (`crm-arch:11-security-audit-and-operations.md:51-56`). --- ### 8. Data operations #### 8.1 Engine and operations (Verified) CRM has no import framework; it registers operations in the shared Data Job Engine (`@rhythmsoftware/macroservice-data-jobs` 0.12.1), which owns lanes, admission, uploads, progress, cancellation and results (CRM-ADR-07; `crm-arch:10-data-operations.md:6-17`). Registered (`services-crm:modules/data-jobs/services/operations/data-operations.ts:1-25`): | Object | Operations | Matching keys | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | | `crm.contacts` | `insert`, `upsert` (whole person with collections, affiliations, roles, custom fields), `update` (PATCH contract), `delete`, rollback of `update` | `id`, `sourceKey` (`contact-import.ts:91`; `contact-update.ts:95`); delete `id` only | | `crm.organizations` | `insert`, `upsert`, reference-only `update`, `delete`, rollback of `update` (`organization-update-rollback.ts`) | `id`, `sourceKey` (`organization-import.ts:119`) | | configuration types (7) | `insert`, `upsert`, `update`, `delete` | `id`, `sourceKey` | | activities | `insert`, `upsert`, `update`, `delete` | `id`, `sourceKey` | `crm-arch:10-data-operations.md:75-77` says organizations have no rollback; `organization-update-rollback.ts` and its integration spec exist at `e818ee48` (`docs/data-jobs.md:899-927`), so that statement is stale. #### 8.2 How an import maps and matches (Verified) - An import item is the same body `POST /contacts/{tenantId}` takes; `createContact` / `updateContact` write it with all the contact's rules, change event and revision ("an import can do nothing a single create could not", `contact-import.ts:1-9`). - References are **source keys** resolved through the crosswalk `crm.data_id_mappings` (`tenant_id`, `source_system`, `object_type`, `source_key`, `record_id`, `last_source_revision`, `sys_last_applied_version`, job ids) (migration `0068_data_job_engine_0_3.sql:146-159`); an unloaded key blocks the item `dependency_missing` (`contact-import.ts:10-13`). - Spreadsheet e-mail/phone/address columns become typed collections before validation (Ruled D-40; `collectionMapping`, `contact-import.ts:88`); untyped columns resolve to the tenant's configured default type (Ruled E-15) (`crm-arch:10-data-operations.md:37-46`). - The source identity is kept in `sys_external_id`; the contact number is always the server's (`contact-import.ts:21-26`). - **An upsert of a known key replaces the contact**: "the fields it omits are cleared exactly as its own PUT clears them" (`contact-import.ts:206-207`). Matching by e-mail is "through upsert rules" only, and configurable match keys (id, source key, e-mail, unique custom field) are J-9, Partial (`crm-arch:10-data-operations.md:31`). - Historical imports suppress only the compliance notice e-mail (`contact-import.ts:29-31,97`). #### 8.3 Bulk update, versions and rollback (Verified / Ruled) - `crm.contacts` / `update` takes the same field map as `PATCH /contacts`, validated by the same schema; `prevalidate` resolves ids under the submitter's access; `apply` is the contact's own update inside the batch transaction (`contact-update.ts:1-30`). - Every revision a job writes carries `sys_job_id`/`sys_job_batch_id` (`record-revisions.table.ts:137-144`), indexed for rollback (`record_revisions_job_idx`, `:155`). - Ruled D-37/D-38 (version capture at validation, source-revision ordering) are the standard (`services-platform:docs/standards/record-write-contracts.md:163-200`) and "Not built" in CRM (`crm-arch:10-data-operations.md:24-25`). - **Rollback** restores only what is still as the job left it (Ruled D-39); collections and custom-field values are `not_reversible_operation` (`record-rollback-plan.ts:24-26,130-133`; `docs/debt/rollback-of-collections.md:1-20`). - Files in imports: a `prepare` stage outside the transaction runs the asset pipeline per item; a refused file is that item's rejection (`record-files.ts:1-25`). (The asset-persistence record's "Bulk and import write paths do not exist yet", `docs/migrations/asset-persistence/README.md:334`, is stale.) - Bulk updates trigger the same side effects as a console edit, including duplicate detection via change events (`crm-arch:10-data-operations.md:59-60`). #### 8.4 How a bulk "apply enrichment" could ride on it (Inferred) A job of `crm.contacts` / `update` per enrichment batch would give per-record outcomes, version checks (once D-37 lands), duplicate re-keying, revisions attributed to the job, and rollback of scalar fields. Limits: anything written to e-mail, phone, address, affiliation or custom-field collections is not rolled back; the PATCH contract cannot express "fill only if empty" except by the producer reading first; `source_revision` ordering is not yet enforced on `update`; a photo URL is imported per item in `prepare` with the 10 s / 10 MiB limits. --- ### 9. Photos and logos (asset pipeline) Verified, `services-crm:docs/migrations/asset-persistence/README.md` (accepted 2026-10-08) and `services-platform:docs/assets.md`. - **Managed fields** are decided by metadata data type `image`/`file`: contacts `image_url`, organizations `logo_url`, and image/document custom fields (`contacts.object.ts:56,142`; `organizations.object.ts:45`). - **Stored form**: `service://platform-file-uploads/v1/{tenant}/{folder}/{recordFolder}/{public-}{uuid}-{name}` in Rhythm 1.0's bucket (`asset-persistence/README.md:68-80`). - **External URL import** (the enrichment case): "Imported to permanent storage and stored as `service://…`. This changes Rhythm 1.0 behaviour, which stored external URLs as pointers" (`:137`); Rhythm 1.0 kept FullContact avatars as pointers (`:51`, citing rolodex-contacts `full-contact-enrichment-engine.js:39-41`). Debt note: "When FullContact enrichment is ported, the avatar URL it sets is imported through the same pipeline" (`docs/debt/image-files-and-enrichment.md:8`). - **Guards**: http/https only, no credentials, default port, no https→http redirect; every resolved address public (SSRF ranges incl. `169.254.169.254`), DNS pinned; ≤ 3 redirects; 3 s connect, 10 s per file, 20 s per save; 10 MiB image, 25 MiB document; bytes sniffed; JPEG/PNG/GIF/WebP only for images; SVG/HTML refused; streamed multipart, no partial object (`asset-persistence/README.md:172-197`; `services-platform:docs/assets.md:202-219`). - **Prepare, then commit**: `managed_assets` row `preparing` → copy/import → `ready`, all before the record transaction; the record commit moves rows to `attached` and replaced ones to `detached`; cleanup later (`:143-170`). The repository refuses a changed managed field that did not come through the pipeline (`requirePreparedAssets`, `:166-168`). - **Lifecycle table** `crm.managed_assets` (`managed-assets.table.ts:1-19`; columns in Appendix A.9 from `pkg:macroservice-assets@0.4.10 dist/managed-assets.js:78-113`): `source_kind` `staging | external`, `source_fingerprint` (the URL for an import, reused by a retry within an hour), `media_type`, `byte_size`, `state`. - **Unchanged value**: a value equal to the stored one is never fetched again (`asset-persistence/README.md:124-125`), so re-enriching with the same provider URL is free; a provider whose avatar URL changes on every call would re-import each time (Inferred). - Malware scanning: no (decision recorded, `:348`). --- ### 10. Write contracts and validation #### 10.1 Routes (Verified, `modules/entities/aws/api/contacts.routes.ts`) `POST /contacts/{tenantId}` (218-219), `GET /contacts/{tenantId}` (256-257), `GET /contacts/{tenantId}/{id}` (303-304), `PUT` (343-344), `PATCH` JSON Patch (360-361), `DELETE` (385-386), archive/unarchive (425-446), element actions `/contacts/{tenantId}/{id}/{kind}/{element_id}/{action}` for verify/relink/remove (191-192, 465-497), history routes, `reconcile_affiliation_dates` (399-400). Organizations mirror them (`organizations.routes.ts:144-401`). Duplicate checks `POST /duplicates/{tenantId}/{objectType}/checks` (`duplicates.routes.ts:98`). #### 10.2 Service functions and schemas (Verified) - `createContact(ctx, input, options)` (`contacts.ts:2096`) and `updateContact(ctx, id, input, expectedVersion, options)` (`:2245-2330`). The update: prepares automatic roles, locks the row `FOR UPDATE`, refuses an archived contact unless `allow_archived`, checks the version, refuses changed read-only properties, prepares affiliations, syncs job title, resolves linked sources, normalises, asserts rules, prepares custom fields, writes the row and collections, applies affiliation-type default links, writes custom fields, and **restores the stamps and writes no revision when nothing changed** (D-05 no-op) (`:2252-2311`). Files go through `entityAssets.save` first (`:2324`). - `ContactInput` = own writable fields + `email_addresses`, `phone_numbers`, `addresses`, `communication_preferences`, `affiliations`, `set_primary_organization`, `reapply_affiliation_defaults`, `custom_fields` (`:329-343`). - zod boundary derived from the tables (`createApiWritableBodyShape`), strict, with the server-owned columns accepted and ignored and the computed names refused (`aws/api/contacts.schemas.ts:1-17`). - Optimistic concurrency: `sys_version` on every field-changing write, missing 400 / stale 409 (Ruled D-05), and a stale version's refusal "says who moved it: a colleague or the system (M1-D6)" (`contacts.ts:2313-2321`). #### 10.3 Protected and system fields (Verified) - Server-owned on contacts: `tenant_id`, `number`, the three computed names, the region fingerprint, archive columns, the create digest (`contacts.table.ts:234-256`). - Read-only on replace (refused if changed, M1-D5): computed names, `number`, the four stamps, `sys_locked`, `origin_service`, `origin_service_record_id` (`contacts.ts:358-381`). Locks change only through the audited lock/unlock operations (E-14). - Read-only by metadata: civic districts (`contacts.object.ts:61-72`) — so the district lookup that enrichment would revive has **no writer path through the API at all** today (Inferred from `write: 'read_only'`; a server-side writer would be needed). - Organizations: `number`, `contact_role_compliance_status`, violations, fingerprint, archive columns (`organizations.table.ts:188-196`). #### 10.4 Execution context and non-user actors (Verified) - Tenant from the path and signed context, never the body; forced fail-closed RLS on `app.tenant_id` (`crm-arch:11-security-audit-and-operations.md:9-13`; `crm-arch:04-domain-and-data-model.md:10-13`). - Session settings every transaction sets: `app.tenant_id`, `app.user_id`, `app.customer_id`, `app.is_staff`, `app.correlation_id`, `app.source`, `app.operation`, `app.request_id`, `app.job_id`, `app.job_batch_id`, `app.originating_user_id` (`pkg:macroservice-db@0.15.0 dist/session-context.js:14-26`). - **Can a non-user write a contact today?** A CRM job (`origin === 'job'` or `system` caller) can, and is recorded as source `job` with an actor id; portal sign-up does it through SQL as `portal-sign-up`. A service principal from another macroservice cannot: there is no `contacts` service function and the revision source check excludes `service` (§7.2). An API client of the Integration Platform has no caller type (`integration` reserved). **Verified.** - Ruled W1 "system writers" take an equivalent version precondition inside their own transaction (`services-platform:docs/standards/record-write-contracts.md:96-103`); W7 is the only audited exception (migrations and repairs) and "An integration that finds versions inconvenient reads before it writes" (`:275-293`). **Ruled.** --- ### 11. Domain events #### 11.1 Business events (external) (Verified / Planned) - Published: **only `crm.contact.merged`** v1, subject `crm.contacts`, scope `tenant`, from the merge transaction (`services-crm:docs/business-events.md:12-16`; `aws/crm.events.ts:26-44`). Payload = Integration Platform catalogue `data`: `record` (id, status, names, primary e-mail as `email`, phone, `organization_id`, address) and `merged_ids` (`business-events.md:22-45`). - Transport: transactional outbox — ledger `crm.business_events` and queue `crm.business_event_queue` written in the transaction; `crm_business_events_dispatcher` (1-minute schedule + nudge) sends one SQS message per (event, destination) to `crm-business-events`; consumer invokes each subscriber's `_event_intake` or the Integration Platform router; 5 receives then DLQ; ledger kept 90 days (`business-events.md:71-91`). Envelope fields: `rhythm_event`, `id` (UUIDv7, dedupe key), `key`, `version`, `tenant_id`, `occurred_at`, `subject`, `scope`, `correlation_id`, `actor_id`, `replay`, `payload` (`:47-69`). - Planned: `crm.contact.created/.updated/.deleted`, `crm.organization.created/.updated/.deleted/.merged`, later `.archived/.unarchived`, "payloads carry ids and changed field names, no personal data" (`crm-arch:06-api-and-service-architecture.md:81`; `crm-arch:12-cross-application-contracts.md:53-58`); verification in CRM-M15 (`crm-arch:15-completion-roadmap.md:136-141`). #### 11.2 Change events (internal) (Verified) One `_change_events` table per entity, generic capture trigger, one event per record per save with the row and every changed child before/after and the changed fields; dispatcher to SQS; handlers by stream (`services-crm:docs/change-events.md:1-40`). Streams and handlers (`aws/crm.change-events.ts:11-35,66-99`): - `contact`: organization compliance recalculation, region assignment (+ override), Quick Search indexer, duplicate detection (batch). - `organization`: region assignment (+ override), linked-contact region assignment, Quick Search, duplicate detection. - `region_recalculation`, `compliance_reprocess`: runners; `activity`: history only. "Change capture remains CRM-internal except for named handlers CRM invokes (Quick Search, Membership)" (`crm-arch:12-cross-application-contracts.md:56-58`). An enrichment trigger inside CRM would be a new handler on the `contact`/`organization` stream (Inferred). --- ### 12. Query and semantic layer - **Registration**: `defineObject(contacts, …)` with fields **opt-out** ("every column of the table is a field, so a column added later is described and queryable without a second step"), `exclude` with reasons (`metadata/contacts.object.ts:1-17,132-140`); `defineQueryObject(contactsObject, …)` with permission, defaults (`DISCOVERABLE_RECORDS`), `hidden: ['image_url', 'id']`, `customFieldValues`, `source: contactsForQueries`, picks (primary organization, typed addresses/phones read from the effective views), collections (`affiliations`, `currentAffiliations`, `addresses`, `phoneNumbers`, as `exists` criteria) (`queries/contacts.query.ts:1-46,186-300`). Organizations read from `crm.organizations_for_queries` (with `contact_count`, `child_organization_count`) (`:457-480`). **Verified.** - **The view is an explicit grant**: `crm.contacts_for_queries` is "a select list that is the grant", `security_invoker`, read-only (`db/contacts-for-queries.view.ts:1-20`), so a new contact column reaches the query layer only when the view is recreated in a migration — whatever the metadata's opt-out says (Inferred from the view's definition). - **Keeping something out**: metadata `exclude` (as for `origin_service*`, `_ocd_id`); query `hidden`; Quick Search `neverIndexed` (`notes`, DOB, DOD, gender, pronouns, `:355-361`); history `exclude`/`redact` per stream (`change-event-installs.ts:44-55`; a unit test refuses secret-looking columns that are neither, `docs/record-history.md:92-94`). **Verified.** - **Exposing a related table**: the affiliations and addresses precedents — a table with its own metadata object (`crm.contact_addresses`, `crm.contact_organization_affiliations`), its own query object, and a collection on contacts for `exists` criteria (`queries/contacts.query.ts:281-300,365-446`). An enrichment table (e.g. per-field suggestions or provider snapshots) could follow it and stay out of the default contact list by not being a pick (Inferred). - Record-level/field-level security in queries is CRM-M23 (Open CRM-Q8; Platform capability, "CRM declares sensitivity in metadata (Recommended, later)") (`crm-arch:11-security-audit-and-operations.md:31-32`). **Planned.** --- ### 13. Consent ledger - **Ruled**: Marketing decision **D-M2** (Andrew 2026-10-20): "The consent and subscription ledger (subscription types, subscriptions, consent history, global unsubscribe incl. `email_opt_out`) lives in services-crm. Marketing reads and writes it through CRM capability functions; Marketing enforces it in its send gate; the Platform preferences module of chapter 24 is NOT built" (`marketing-arch:docs/proposals/marketing/decisions-2026-10-20.md:9`). - **Not built**: no consent, subscription-type or consent-history table in services-crm (searched schema and migrations; "consent" appears only in merge documents and compliance mapping). Marketing's milestones: MKT-M5 waits on "CRM consent-ledger milestone (after CRM-M1 is accepted)" (`services-marketing:docs/milestones/README.md:12`); until then the send gate honours `email_opt_out` only (`services-marketing:docs/milestones/MKT-M2.md:41`) and the subscription address rule is refused (`services-marketing:modules/resolution/services/resolution/address-rule.ts:4,23`). - **What exists** (Verified): `contacts.do_not_mail`, `do_not_call`, `email_opt_out`, `communication_preferences_last_confirmed` (`contacts.table.ts:133-136`) and `crm.contact_communication_preferences` (`communication_type_id` free text, `enabled`, `subscription_email_address`, `subscription_status` `subscribed | digest | unsubscribed`, `subscription_last_updated`; one per type per contact) (`contact-communication-preferences.table.ts:1-34`). Communication types are "**Missing** as a table (free text)" (`crm-arch:04-domain-and-data-model.md:101`); CRM-M13 plans them (`crm-arch:15-completion-roadmap.md:113-121`). - **Nothing about data-processing consent** (consent to be enriched, GDPR/CCPA basis, objection to profiling): **Absent** in CRM and not mentioned in D-M2's scope (Inferred from the D-M2 text, which is communication consent). --- ### 14. Portal profile - **Sign-up and claim**: Platform links a verified member to an unclaimed contact holding the exact e-mail, else asks `crm.portal_contact_candidates` for ≤ 3 candidates (score ≥ 50), else `crm.create_portal_contact` writes a new contact as the system actor `portal-sign-up` and queues a `portal_sign_up` duplicate pair for staff (`docs/merging/duplicate-detection.md:500-564`; migration `0077`). The member's contact is the Platform `cid` claim (`crm-arch:14-gap-matrix.md:243`; `rhythm-web:packages/web-runtime/src/eligibility/types.ts:59`). **Verified.** - **Own-profile editing**: **not in services-crm**. rhythm-web's member portal (`profile`, `profile-edit`, `profile-contact`, `profile-communication` routes) calls `/contacts/{tenantId}/portal/me`, `/me/addresses`, `/me/email_addresses`, `/me/phone_numbers`, `/me/organization` (`rhythm-web:packages/web-contacts/src/server/http-crm-client.ts:22-40,73,188,296`), and states "**Every route in this document is a request**" (`packages/web-contacts/API-ASSUMPTIONS.md:1-12`). No `/portal` route exists in `services-crm:modules/*/aws/api`. Planned as PO-01 / CRM-M22, "with name and credential locks", scheduled last (`crm-arch:14-gap-matrix.md:243`; `crm-arch:05-functional-architecture.md:351-358`). **Absent / Planned.** - **Are member-entered values flagged?** Only at the save level: a portal caller's write is `sys_source = 'portal'` in the revision (`execution-context.js:191-192`). There is no per-field "member-entered" or "member-confirmed" flag (Absent). The proposal mentions "name and credential locks" for the member's own profile (PO-01) but no field-level ownership model (Planned, undesigned). --- ### 15. Existing traces of enrichment, sources, verification, suggestions, review | Term | Where found | Label | | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------- | | FullContact | `docs/debt/image-files-and-enrichment.md:1-10` ("FullContact enriched social profiles; Google Civic filled the districts ... The fields are stored; nothing fills them"); `docs/migrations/asset-persistence/README.md:51` (1.0 kept FullContact avatars as pointers); `docs/migrations/postgres-indexing/README.md:712` (1.0 "social-profile caches" listed as enrichment caches, not carried) | Verified | | enrichment | `crm-arch:05-functional-architecture.md:61-62,349-350`; `14-gap-matrix.md:26,273` ("verify the Civic API's 2025 retirement before designing"); `15-completion-roadmap.md:217`; `docs/merging/experience/functional-requirements.md:90` and `v0-design-brief.md:113` (the duplicate-review destination is "intended to grow into a broader data quality center (enrichment, missing information, validation later)") | Planned | | source / origin | `origin_service`, `origin_service_record_id` (record-level); `source_mode` (addresses/phones); `sys_source` (revisions); `data_id_mappings.source_system` / `source_key` (imports) | Verified | | verified / needs_verification | disconnected rows only (`docs/disconnected-information.md`) | Verified | | confidence | duplicate `score`/`band`/`reasons` only | Verified (duplicates) | | suggestion | the contact editor's organization suggestion by domain (Recommended, `crm-arch:05-...:90-93`); duplicate "possible duplicates" hint | Planned / Verified | | pending change / review queue | duplicate candidates `review_queue` action; attention responsibility **`crm.data_quality_review`** ("Data quality review", expected holder `never`, today addressed only by `crm.organizations_out_of_compliance`) (`docs/attention.md:14-19,32-36`; `aws/crm.responsibilities.ts:12,38`) | Verified | | social profile last updated | `date_social_profile_last_updated` column; merge label "Social Profile Last Updated" | Verified (unused) | | pending/suggested values table | none | Absent | --- ### Appendix A. Quoted DDL and interfaces Quoted verbatim from Drizzle declarations at services-crm `e818ee48` unless stated. Drizzle names are the SQL column names. #### A.1 `crm.contacts` (`modules/entities/db/contacts.table.ts:52-173`) ```ts export const CONTACT_STATUSES = ['active', 'duplicate', 'deceased', 'inactive'] as const; export const GENDERS = ['male', 'female'] as const; export const contacts = crmSchema.table( 'contacts', { tenant_id: text('tenant_id').notNull(), id: text('id').notNull(), number: bigint('number', { mode: 'number' }).notNull(), status: text('status', { enum: CONTACT_STATUSES }).notNull().default('active'), prefix: text('prefix'), salutation: text('salutation'), first_name: text('first_name'), middle_name: text('middle_name'), last_name: text('last_name'), maiden_name: text('maiden_name'), nickname: text('nickname'), suffix: text('suffix'), credentials: text('credentials'), job_title: text('job_title'), name: text('name'), full_name: text('full_name'), name_last_first: text('name_last_first'), type_id: text('type_id'), region_id: text('region_id'), override_automatic_region_assignment: boolean('override_automatic_region_assignment'), sys_region_address_fingerprint: text('sys_region_address_fingerprint'), sys_create_digest: text('sys_create_digest'), national_district: text('national_district'), national_district_ocd_id: text('national_district_ocd_id'), state_upper_district: text('state_upper_district'), state_upper_district_ocd_id: text('state_upper_district_ocd_id'), state_lower_district: text('state_lower_district'), state_lower_district_ocd_id: text('state_lower_district_ocd_id'), spouse_id: text('spouse_id'), image_url: text('image_url'), gender: text('gender', { enum: GENDERS }), pronouns: text('pronouns'), date_of_birth: date('date_of_birth', { mode: 'string' }), date_of_death: date('date_of_death', { mode: 'string' }), notes: text('notes'), facebook: text('facebook'), twitter: text('twitter'), instagram: text('instagram'), linked_in: text('linked_in'), date_profile_last_updated: wireTimestamptz('date_profile_last_updated'), date_social_profile_last_updated: wireTimestamptz('date_social_profile_last_updated'), origin_service: text('origin_service'), origin_service_record_id: text('origin_service_record_id'), do_not_mail: boolean('do_not_mail').notNull().default(false), do_not_call: boolean('do_not_call').notNull().default(false), email_opt_out: boolean('email_opt_out').notNull().default(false), communication_preferences_last_confirmed: wireTimestamptz('communication_preferences_last_confirmed'), ...archiveColumns(), ...snakeSystemColumns(), }, (t) => [ primaryKey({ name: 'contacts_pk', columns: [t.tenant_id, t.id] }), unique('contacts_number_uq').on(t.tenant_id, t.number), foreignKey({ name: 'contacts_spouse_fk', columns: [t.tenant_id, t.spouse_id], foreignColumns: [t.tenant_id, t.id] }), index('contacts_name_idx').on(t.tenant_id, t.last_name, t.first_name), index('contacts_name_last_first_idx').on(t.tenant_id, t.name_last_first, t.id), index('contacts_spouse_idx').on(t.tenant_id, t.spouse_id), foreignKey({ name: 'contacts_region_fk', columns: [t.tenant_id, t.region_id], foreignColumns: [regions.tenant_id, regions.id] }), index('contacts_region_idx').on(t.tenant_id, t.region_id), foreignKey({ name: 'contacts_type_fk', columns: [t.tenant_id, t.type_id], foreignColumns: [contactTypes.tenant_id, contactTypes.id] }), index('contacts_type_idx').on(t.tenant_id, t.type_id), check('contacts_status_check', sql.raw(`"status" IN (${sqlList(CONTACT_STATUSES)})`)), check('contacts_gender_check', sql.raw(`"gender" IS NULL OR "gender" IN (${sqlList(GENDERS)})`)), check('contacts_spouse_not_self_check', sql`${t.spouse_id} IS NULL OR ${t.spouse_id} <> ${t.id}`), check('contacts_number_check', sql`${t.number} > 0`), check( 'contacts_first_or_last_name_check', sql`(${t.first_name} ~ '[^[:space:]]') IS TRUE OR (${t.last_name} ~ '[^[:space:]]') IS TRUE`, ), archiveCheck('contacts', t), unarchiveReasonCheck('contacts', t), lockedCheck('contacts', t.sys_locked), ], ); ``` (Comments removed; foreign key and check bodies re-wrapped onto fewer lines, otherwise verbatim.) `archiveColumns()` (`modules/entities/db/archive-columns.ts:32-37`): ```ts export const archiveColumns = () => ({ archived_at: wireTimestamptz('archived_at'), archived_by_id: text('archived_by_id'), archive_reason: text('archive_reason'), unarchive_reason: text('unarchive_reason'), }); ``` `snakeSystemColumns()` (`pkg:macroservice-records@0.5.1 dist/system-columns.js:63-71`, compiled JS): ```js const snakeSystemColumns = () => ({ sys_version: (0, pg_core_1.integer)('sys_version').notNull().default(1), sys_created_at: (0, exports.timestamptz)('sys_created_at').notNull().defaultNow(), sys_created_by_id: (0, pg_core_1.text)('sys_created_by_id').notNull(), sys_last_modified_at: (0, exports.timestamptz)('sys_last_modified_at').notNull().defaultNow(), sys_last_modified_by_id: (0, pg_core_1.text)('sys_last_modified_by_id').notNull(), sys_external_id: (0, pg_core_1.text)('sys_external_id'), sys_locked: (0, pg_core_1.text)('sys_locked', { enum: exports.LOCKED_VALUES }), }); ``` with `LOCKED_VALUES = ['deletes', 'updates & deletes']` (`:20`). Server-owned and read-only lists: ```ts // contacts.table.ts:247-256 export const SERVER_OWNED_CONTACT_COLUMNS: readonly ServerOwnedContactColumn[] = Object.freeze([ 'tenant_id', 'number', 'name', 'full_name', 'name_last_first', REGION_FINGERPRINT_PROPERTY, ...ARCHIVE_PROPERTIES, CREATE_DIGEST_PROPERTY, ]); // services/persistence/contacts.ts:358-372 export const CONTACT_READ_ONLY_PROPERTIES = Object.freeze([ 'name', 'full_name', 'name_last_first', 'number', 'sys_created_at', 'sys_created_by_id', 'sys_last_modified_at', 'sys_last_modified_by_id', 'sys_locked', 'origin_service', 'origin_service_record_id', ]); ``` #### A.2 `crm.organizations` (`modules/entities/db/organizations.table.ts:47-129`) ```ts export const ORGANIZATION_STATUSES = ['active', 'duplicate', 'inactive'] as const; export const CONTACT_ROLE_COMPLIANCE_STATUSES = ['in compliance', 'not in compliance'] as const; export const organizations = crmSchema.table( 'organizations', { tenant_id: text('tenant_id').notNull(), id: text('id').notNull(), number: bigint('number', { mode: 'number' }).notNull(), name: text('name').notNull(), type_id: text('type_id'), acronym: text('acronym'), description: text('description'), notes: text('notes'), status: text('status', { enum: ORGANIZATION_STATUSES }).notNull().default('active'), website: text('website'), domain_name: text('domain_name'), alternate_domain_names: text('alternate_domain_names').array(), parent_organization_id: text('parent_organization_id'), logo_url: text('logo_url'), flags: text('flags').array(), tax_exempt: boolean('tax_exempt'), region_id: text('region_id'), override_automatic_region_assignment: boolean('override_automatic_region_assignment'), sys_region_address_fingerprint: text('sys_region_address_fingerprint'), facebook: text('facebook'), twitter: text('twitter'), linked_in: text('linked_in'), date_profile_last_updated: wireTimestamptz('date_profile_last_updated'), contact_role_compliance_status: text('contact_role_compliance_status', { enum: CONTACT_ROLE_COMPLIANCE_STATUSES }), ...archiveColumns(), ...snakeSystemColumns(), }, // constraints: organizations_pk (tenant_id, id); organizations_number_uq; organizations_parent_fk; organizations_type_fk; // indexes on type, name, parent, region, compliance status; checks on status, parent_not_self, number > 0, // compliance status; archiveCheck; unarchiveReasonCheck; lockedCheck (lines 96-128) ); ``` #### A.3 `crm.contact_email_addresses` (`contact-email-addresses.table.ts:36-62`) ```ts export const MAX_EMAIL_ADDRESSES = 3; export const PRIMARY_EMAIL_UNIQUE_INDEX = 'contact_email_addresses_primary_email_address_uidx'; export const contactEmailAddresses = crmSchema.table( 'contact_email_addresses', { ...contactOwnedColumns(), // tenant_id, id, contact_id position: smallint('position').notNull(), email_address: emailAddress('email_address', CRM_SCHEMA_NAME).notNull(), ...snakeSystemColumns(), }, (t) => [ ...contactOwnership('contact_email_addresses', t, { owner_index: false }), unique('contact_email_addresses_position_uq').on(t.tenant_id, t.contact_id, t.position), index('contact_email_addresses_email_address_idx').on(t.tenant_id, t.email_address), index('contact_email_addresses_primary_email_address_idx') .on(t.tenant_id, t.email_address) .where(sql`${t.position} = 1`), check('contact_email_addresses_position_check', sql`${t.position} BETWEEN 1 AND ${sql.raw(String(MAX_EMAIL_ADDRESSES))}`), lockedCheck('contact_email_addresses', t.sys_locked), ], ); ``` `contactOwnedColumns()` (`contacts.table.ts:194-198`): `tenant_id text NOT NULL`, `id text NOT NULL`, `contact_id text NOT NULL`; owner FK `(tenant_id, contact_id)` → `crm.contacts` `ON DELETE CASCADE` (`:205-217`). #### A.4 `crm.contact_phone_numbers` (`contact-phone-numbers.table.ts:41-126`) ```ts export const contactPhoneNumbers = crmSchema.table( 'contact_phone_numbers', { ...contactOwnedColumns(), phone_number_type_id: text('phone_number_type_id').notNull(), phone_number: text('phone_number'), is_preferred: boolean('is_preferred').notNull().default(false), source_mode: text('source_mode', { enum: SOURCE_MODES }).notNull().default('owned'), source_affiliation_id: text('source_affiliation_id'), source_organization_id: text('source_organization_id'), source_organization_phone_number_id: text('source_organization_phone_number_id'), copied_from_organization_id: text('copied_from_organization_id'), copied_at: wireTimestamptz('copied_at'), ...disconnectedColumns(PHONE_NUMBER_DISCONNECTED_REASONS), ...snakeSystemColumns(), }, (t) => [ ...contactOwnership('contact_phone_numbers', t, { owner_index: false }), unique('contact_phone_numbers_one_per_type_uq').on(t.tenant_id, t.contact_id, t.phone_number_type_id), uniqueIndex('contact_phone_numbers_one_preferred_uidx') .on(t.tenant_id, t.contact_id) .where(sql`${t.is_preferred}`), // FKs: phone_number_type_fk (RESTRICT), source_affiliation_fk, source_organization_phone_number_fk; // checks: source_mode_check, source_check (CASE per mode), disconnected_reason_check, check( 'contact_phone_numbers_phone_number_check', sql`CASE WHEN ${t.source_mode} = 'linked' THEN ${t.phone_number} IS NULL ELSE ${t.phone_number} IS NOT NULL END`, ), lockedCheck('contact_phone_numbers', t.sys_locked), ], ); ``` Shared declarations (`contact-addresses.table.ts:63-87`): ```ts export const SOURCE_MODES = ['owned', 'copied', 'linked', 'disconnected'] as const; export const ADDRESS_DISCONNECTED_REASONS = ['organization_address_removed', 'organization_merged'] as const; export const PHONE_NUMBER_DISCONNECTED_REASONS = ['organization_phone_number_removed', 'organization_merged'] as const; export const disconnectedColumns = (reasons: TReasons) => ({ disconnected_from_organization_id: text('disconnected_from_organization_id'), disconnected_at: wireTimestamptz('disconnected_at'), disconnected_reason: text('disconnected_reason', { enum: reasons as unknown as [TReasons[number], ...TReasons[number][]] }), }); ``` #### A.5 `crm.contact_addresses` (`contact-addresses.table.ts:92-224`) ```ts export const contactAddresses = crmSchema.table( 'contact_addresses', { ...contactOwnedColumns(), address_type_id: text('address_type_id').notNull(), line1: text('line1'), line2: text('line2'), line3: text('line3'), city: text('city'), state: text('state'), postal_code: text('postal_code'), county: text('county'), country: text('country').default('US'), latitude: doublePrecision('latitude'), longitude: doublePrecision('longitude'), time_zone_id: text('time_zone_id'), seasonal_start: text('seasonal_start'), seasonal_end: text('seasonal_end'), seasonal_preferred: boolean('seasonal_preferred').notNull().default(false), is_preferred: boolean('is_preferred').notNull().default(false), source_mode: text('source_mode', { enum: SOURCE_MODES }).notNull().default('owned'), source_affiliation_id: text('source_affiliation_id'), source_organization_id: text('source_organization_id'), source_organization_address_id: text('source_organization_address_id'), copied_from_organization_id: text('copied_from_organization_id'), copied_at: wireTimestamptz('copied_at'), ...disconnectedColumns(ADDRESS_DISCONNECTED_REASONS), ...snakeSystemColumns(), }, // unique contact_addresses_one_per_type_uq (tenant_id, contact_id, address_type_id); one_preferred_uidx WHERE is_preferred; // address_type_fk RESTRICT; seasonal MM-DD checks; seasonal_preferred and seasonal_not_preferred checks; lat/long range // checks; source_affiliation_fk; source_organization_address_fk; source_mode/source/disconnected_reason checks; // contact_addresses_linked_values_check (a linked row holds no value); lockedCheck (lines 134-223) ); ``` Organization addresses (`organization-addresses.table.ts:30-57`): `tenant_id`, `id`, `organization_id`, `address_type_id`, `line1`, `line2`, `line3`, `city`, `state`, `postal_code`, `county`, `country` default `US`, `latitude`, `longitude`, `time_zone_id`, `is_preferred`, system columns; `organization_addresses_one_per_type_uq`, `_one_preferred_uidx`. Organization phone numbers (`organization-phone-numbers.table.ts:25-40`): `tenant_id`, `id`, `organization_id`, `phone_number_type_id`, `phone_number NOT NULL`, system columns; `organization_phone_numbers_one_per_type_uq`. Collection types (`collection-types.ts:37-44`): `tenant_id`, `id`, `name`, `applies_to` (default `contacts`), `display_order` (default 0). #### A.6 Affiliations, roles, types, custom field values `crm.contact_organization_affiliations` (`contact-organization-affiliations.table.ts:51-131`): ```ts export const contactOrganizationAffiliations = crmSchema.table( 'contact_organization_affiliations', { ...contactOwnedColumns(), organization_id: text('organization_id').notNull(), affiliation_type_id: text('affiliation_type_id').notNull(), is_primary: boolean('is_primary').notNull().default(false), job_title: text('job_title'), start_date: date('start_date', { mode: 'string' }), end_date: date('end_date', { mode: 'string' }), notes: text('notes'), sys_automatic_roles_due: boolean('sys_automatic_roles_due').notNull().default(false), sys_default_links_pending: jsonb('sys_default_links_pending').$type(), ...snakeSystemColumns(), }, (t) => [ ...contactOwnership('contact_organization_affiliations', t, { owner_index: false }), unique('contact_organization_affiliations_contact_uq').on(t.tenant_id, t.id, t.contact_id), unique('contact_organization_affiliations_contact_organization_uq').on(t.tenant_id, t.id, t.contact_id, t.organization_id), // organization_fk, affiliation_type_fk (no delete action) uniqueIndex('contact_organization_affiliations_one_primary_uidx') .on(t.tenant_id, t.contact_id) .where(sql`${t.is_primary}`), // indexes: contact_type, organization, organization_current (end_date IS NULL), type_organization, // automatic_roles_due, default_links_pending check('contact_organization_affiliations_primary_open_check', sql`NOT ${t.is_primary} OR ${t.end_date} IS NULL`), check( 'contact_organization_affiliations_dates_check', sql`${t.end_date} IS NULL OR ${t.start_date} IS NULL OR ${t.end_date} >= ${t.start_date}`, ), lockedCheck('contact_organization_affiliations', t.sys_locked), ], ); export const MAX_AFFILIATIONS = 100; ``` Plus the migration-only exclusion D4 over `daterange(start_date, end_date, '[]')` keyed by `crm.overlap_exclusion_key(...)` (`:26-29`). `crm.contact_organization_roles` (`contact-organization-roles.table.ts:41-54`): `tenant_id`, `id`, `affiliation_id`, `contact_id`, `contact_role_id`, `start_date`, `end_date`, system columns. `crm.affiliation_types` (`affiliation-types.table.ts:36-50`): `tenant_id`, `id`, `name NOT NULL`, `description`, `is_active NOT NULL DEFAULT true`, system columns; checks on configuration-id grammar and non-blank name. `crm.contact_types` (`contact-types.table.ts:30-53`): `tenant_id`, `id`, `name`, `description`, `is_active`, `display_order`, `...layoutAssignmentColumns()` (`console_page_layout_id`, `portal_page_layout_id`, `console_create_form_id`, `console_edit_form_id`, `portal_form_id`; `layout-slots.ts:128-133`), system columns. `crm.organization_types` (`organization-types.table.ts:52-117`): `tenant_id`, `id`, `name`, `is_active`, `display_order`, `restrict_contact_roles`, `applicable_contact_role_ids text[]`, `enable_contact_role_compliance`, `has_contact_role_compliance_options`, `notify_contacts_of_compliance_violation`, `notification_contact_role_ids text[]`, layout assignment columns, system columns. Custom field values (`pkg:macroservice-custom-fields@0.2.9 dist/custom-field-values.js:46-68`, compiled JS; CRM instances `crm.contact_custom_field_values`, `crm.organization_custom_field_values`): ```js function customFieldValuesTable(schema, tableName, owner) { return schema.table( tableName, { tenantId: (0, pg_core_1.text)('tenant_id').notNull(), recordId: (0, pg_core_1.text)('record_id').notNull(), customFieldId: (0, pg_core_1.text)('custom_field_id').notNull(), stringValue: (0, pg_core_1.text)('string_value'), integerValue: (0, pg_core_1.bigint)('integer_value', { mode: 'number' }), numericValue: (0, pg_core_1.numeric)('numeric_value'), booleanValue: (0, pg_core_1.boolean)('boolean_value'), fileUrlValue: (0, pg_core_1.text)('file_url_value'), listValue: (0, pg_core_1.text)('list_value').array(), dateValue: (0, pg_core_1.date)('date_value', { mode: 'string' }), dateTimeValue: (0, system_columns_1.timestamptz)('date_time_value'), referenceValue: (0, pg_core_1.text)('reference_value'), htmlValue: (0, pg_core_1.text)('html_value'), sysLastModifiedAt: (0, system_columns_1.timestamptz)('sys_last_modified_at').notNull().defaultNow(), sysLastModifiedById: (0, pg_core_1.text)('sys_last_modified_by_id').notNull(), }, (t) => [ (0, pg_core_1.primaryKey)({ name: `${tableName}_pk`, columns: [t.tenantId, t.recordId, t.customFieldId] }), (0, pg_core_1.foreignKey)({ name: owner.foreignKeyName, columns: [t.tenantId, t.recordId], foreignColumns: [owner.tenantId, owner.id], }).onDelete('cascade'), (0, pg_core_1.check)( `${tableName}_one_value_check`, (0, drizzle_orm_1.sql)`num_nonnulls(${t.stringValue}, ${t.integerValue}, ${t.numericValue}, ${t.booleanValue}, ${t.fileUrlValue}, ${t.listValue}, ${t.dateValue}, ${t.dateTimeValue}, ${t.referenceValue}, ${t.htmlValue}) = 1`, ), // list_check, custom_field_id_check, per-type partial indexes (lines 67-92) ], ); } ``` `crm.contact_communication_preferences` (`contact-communication-preferences.table.ts:19-34`): ```ts export const SUBSCRIPTION_STATUSES = ['subscribed', 'digest', 'unsubscribed'] as const; export const contactCommunicationPreferences = crmSchema.table( 'contact_communication_preferences', { ...contactOwnedColumns(), communication_type_id: text('communication_type_id').notNull(), enabled: boolean('enabled').notNull(), subscription_email_address: emailAddress('subscription_email_address', CRM_SCHEMA_NAME), subscription_status: text('subscription_status', { enum: SUBSCRIPTION_STATUSES }), subscription_last_updated: wireTimestamptz('subscription_last_updated'), ...snakeSystemColumns(), }, // unique contact_communication_preferences_type_uq (tenant_id, contact_id, communication_type_id) ); ``` #### A.7 Duplicate detection tables (`modules/duplicates/db/`) ```ts // match-keys.table.ts:28-33, 56-64, 81-88 export const CONTACT_MATCH_KEY_TYPES = ['email', 'phone', 'name_sound', 'name_sound_postal', 'name_dob', 'origin_id'] as const; export const ORGANIZATION_MATCH_KEY_TYPES = ['org_name', 'org_name_sound_postal', 'org_domain', 'org_phone', 'origin_id'] as const; // crm.contact_match_keys and crm.organization_match_keys: tenant_id: text('tenant_id').notNull(), record_id: text('record_id').notNull(), key_type: text('key_type', { enum: CONTACT_MATCH_KEY_TYPES }).notNull(), // ORGANIZATION_… for organizations key_value: text('key_value').notNull(), rule_set_version: smallint('rule_set_version').notNull().default(INITIAL_RULE_SET_VERSION), // duplicate-candidates.table.ts:46, 53, 75-105 (crm.contact_duplicate_candidates, crm.organization_duplicate_candidates) export const DUPLICATE_CANDIDATE_STATUSES = ['open', 'dismissed', 'confirmed'] as const; export const DUPLICATE_CANDIDATE_ORIGINS = ['detected', 'manual', 'portal_sign_up'] as const; tenant_id, id, low_record_id, high_record_id, score smallint, band, reasons jsonb (default []), evidence_fingerprint, rule_set_version, status (default 'open'), origin (default 'detected'), dismissed_by_id, dismissed_at, dismissed_fingerprint, dismissal_note, group_id, first_detected_at, last_evaluated_at // duplicate-rules.table.ts:40-82 (crm.duplicate_rules) export const DUPLICATE_RULE_RECORD_TYPES = ['contact', 'organization'] as const; export const DUPLICATE_RULE_MATCHES = ['email', 'name_zip', 'name_phone', 'org_name_domain'] as const; export const DUPLICATE_RULE_ACTIONS = ['warn', 'block', 'review_queue'] as const; tenant_id, id, name, status (default 'active'), record_type, match_on, action (default 'review_queue'), check_portal boolean (default false), position integer, minimum_band (default DEFAULT_MINIMUM_BAND) // duplicate-block-overrides.table.ts:42-59 (crm.duplicate_block_overrides) tenant_id, id, object_type, record_id, reason, rule_ids text[], overridden_record_ids text[], overridden_by_id, overridden_at ``` (The candidate and override column lists are condensed from the declarations at the cited lines; names and order are the declarations'.) Merge alias (`modules/merges/db/merged-record-aliases.table.ts:39-56`): `tenant_id`, `object_type`, `merged_id`, `survivor_id`, `merge_id`, `merged_origin_service`, `merged_origin_service_record_id`, `merged_sys_external_id`. Merge record (`record-merges.table.ts:64-108`): `tenant_id`, `id`, `plan_id`, `object_type`, `survivor_id`, `merged_ids text[]`, `participants jsonb`, `status`, `failure_code`, `failure_message`, `requested_by_id`, `attempts`, `original_values jsonb`, `final_values jsonb`, `resolutions jsonb`, `relationship_actions jsonb`, `identity_actions jsonb`, `validation jsonb`, `counts jsonb`, ... #### A.8 Record history and audit tables `crm.record_revisions` (`modules/entities/db/record-revisions.table.ts:74-81, 84, 110-166`): ```ts export const RECORD_REVISION_SOURCES = [ 'console', 'portal', 'public', 'integration', 'job', 'maintenance', ] as const satisfies readonly (typeof CHANGE_SOURCES)[number][]; export const RECORD_REVISION_OPERATIONS = ['created', 'updated', 'deleted'] as const; export const recordRevisions = crmSchema.table( 'record_revisions', { tenant_id: text('tenant_id').notNull(), id: text('id') .notNull() .default(sql`('rev_' || crm.change_events_uuid_v7()::text)`), sequence: bigint('sequence', { mode: 'number' }).notNull().generatedAlwaysAsIdentity(), record_type: text('record_type').notNull(), record_id: text('record_id').notNull(), record_label: text('record_label'), operation: text('operation', { enum: RECORD_REVISION_OPERATIONS }).notNull(), changes: jsonb('changes').notNull(), changed_fields: text('changed_fields') .array() .notNull() .default(sql`'{}'::text[]`), capture_version: smallint('capture_version').notNull(), transaction_id: xid8('transaction_id').notNull(), sys_recorded_at: timestamptz('sys_recorded_at').notNull(), sys_transaction_started_at: timestamptz('sys_transaction_started_at').notNull(), sys_actor_id: text('sys_actor_id'), sys_actor_label: text('sys_actor_label'), sys_source: text('sys_source', { enum: RECORD_REVISION_SOURCES }).notNull(), sys_operation_name: text('sys_operation_name'), sys_correlation_id: text('sys_correlation_id'), sys_request_id: text('sys_request_id'), sys_job_id: text('sys_job_id'), sys_job_batch_id: text('sys_job_batch_id'), }, (t) => [ primaryKey({ name: 'record_revisions_pk', columns: [t.tenant_id, t.id] }), index('record_revisions_record_idx').on(t.tenant_id, t.record_type, t.record_id, t.sequence.desc()), index('record_revisions_transaction_idx').on(t.tenant_id, t.transaction_id), index('record_revisions_job_idx').on(t.tenant_id, t.sys_job_id, t.record_id), check('record_revisions_operation_check', sql`${t.operation} IN (${sql.raw(list(RECORD_REVISION_OPERATIONS))})`), check('record_revisions_sys_source_check', sql`${t.sys_source} IN (${sql.raw(list(RECORD_REVISION_SOURCES))})`), check('record_revisions_changes_object_check', sql`jsonb_typeof(${t.changes}) = 'object'`), // two millisecond-precision checks ], ); ``` The library's sources (`pkg:macroservice-db@0.15.0 dist/session-context.js:35`): ```js exports.CHANGE_SOURCES = ['console', 'portal', 'public', 'integration', 'job', 'maintenance', 'service']; ``` Contact history configuration (`db/change-event-installs.ts:44-60`): ```ts export const CONTACT_HISTORY: ChangeEventsHistoryOptions = Object.freeze({ record_type: 'crm.contacts', label_column: 'name', exclude: [ 'origin_service', 'origin_service_record_id', 'national_district_ocd_id', 'state_upper_district_ocd_id', 'state_lower_district_ocd_id', ], redact: [], collections: [ { collection: 'email_addresses', parent_key: 'contact_id', identity: ['position'] }, { collection: 'phone_numbers', parent_key: 'contact_id', identity: ['phone_number_type_id'] }, { collection: 'addresses', parent_key: 'contact_id', identity: ['address_type_id'] }, { collection: 'communication_preferences', parent_key: 'contact_id', identity: ['communication_type_id'] }, { collection: 'affiliations', parent_key: 'contact_id', identity: ['organization_id', 'affiliation_type_id'] }, { collection: 'organization_roles', parent_key: 'contact_id', identity: ['contact_role_id'] }, { collection: 'custom_field_values', parent_key: 'record_id', identity: ['custom_field_id'] }, ], }); ``` `crm.record_lock_changes` (`record-lock-changes.table.ts:27-44`): `tenant_id`, `id` (`rlc_` + UUIDv7), `record_type`, `record_id`, `operation`, `locked_before`, `locked_after`, `reason NOT NULL`, `sys_actor_id NOT NULL`, `sys_recorded_at NOT NULL`. `crm.protected_records` (`protected-records.table.ts:17-26`): `tenant_id`, `record_type`, `record_id`, `reason NOT NULL`, `sys_created_at`. #### A.9 Assets and crosswalk `crm.managed_assets` (`pkg:macroservice-assets@0.4.10 dist/managed-assets.js:78-113`, SQL names): `tenant_id`, `id`, `operation_id`, `object_id`, `record_id`, `field_path`, `uri`, `bucket`, `object_key`, `version_id`, `source_kind` (`staging | external`), `source_fingerprint`, `media_type`, `byte_size`, `state` (`preparing, ready, attached, detached, abandoned, discarding, discarded, retired, failed`), `created_by_id`, `created_at`, `state_changed_at`, `cleanup_after`, `disposal` (`delete | retire`), `retired_key`, `cleanup_error`; unique `(bucket, object_key)`; `starts_with(object_key, tenant_id || '/')`. `crm.data_id_mappings` (migration `0068_data_job_engine_0_3.sql:146-159`, created in schema `rolodex`, renamed `crm` by `0075`): ```sql CREATE TABLE "rolodex"."data_id_mappings" ( "tenant_id" text NOT NULL, "source_system" text NOT NULL, "object_type" text NOT NULL, "source_key" text NOT NULL, "record_id" text NOT NULL, "last_source_revision" text, "sys_last_applied_version" integer, "sys_first_job_id" text NOT NULL, "sys_last_job_id" text NOT NULL, "sys_last_applied_at" timestamp with time zone DEFAULT now() NOT NULL, CONSTRAINT "data_id_mappings_pk" PRIMARY KEY("tenant_id","source_system","object_type","source_key"), CONSTRAINT "data_id_mappings_sys_last_applied_version_check" CHECK ("rolodex"."data_id_mappings"."sys_last_applied_version" IS NULL OR "rolodex"."data_id_mappings"."sys_last_applied_version" >= 1) ); ``` #### A.10 Interfaces ```ts // services/persistence/contacts.ts:329-343 export type ContactInput = ContactFieldsInput & { email_addresses?: EmailAddressInput[]; phone_numbers?: PhoneNumberInput[]; addresses?: AddressInput[]; communication_preferences?: CommunicationPreferenceInput[]; affiliations?: AffiliationInput[]; set_primary_organization?: SetPrimaryOrganization; reapply_affiliation_defaults?: { affiliation_id: string }; custom_fields?: Readonly>; }; // services/persistence/contacts.ts:2096-2100, 2245-2251 export async function createContact( ctx: ExecutionContext, input: ContactInput, options: HistoryWriteOptions & PreparedFilesOption & CreateReplayOption & DuplicateBlockOverrideOption = {}, ): Promise { export async function updateContact( ctx: ExecutionContext, id: string, input: ContactInput, expectedVersion: number, options: HistoryWriteOptions & PreparedFilesOption & AllowArchivedOption & ReadOnlyFieldsOption & KeepAutomaticRolesOption = {}, ): Promise { // aws/crm.service-functions.ts:21-27 — the only CRM service function export const CRM_SERVICE_FUNCTIONS = defineServiceFunctions(crmService, { tasks: { routes: tasksServiceRoutes, sensitive: false, description: 'tasks (create, complete, cancel, reassign and re-date a task for a workflow action)', }, }); // pkg:macroservice-execution@0.9.0 dist/execution-context.js:186-198 function changeSourceOf(ctx, isMaintenance) { if (isMaintenance) return 'maintenance'; if (ctx.requestOrigin.origin === 'job' || ctx.caller.accessType === 'system') return 'job'; if (ctx.caller.accessType === 'portal') return 'portal'; if (ctx.caller.accessType === 'none') return 'public'; if (ctx.caller.accessType === 'service') return 'service'; return 'console'; } ``` Planned relationship tables (households, **Ruled HC-D1, Andrew 2026-10-21**: "person-to-person relationships live in CRM-owned tables, not Platform associations", `households:10-data-model-and-service-boundaries.md:93-99`): `crm.relationship_types` (configuration id, `name`, `kind`, symmetric/directional labels, `may_authorize_access`, `system_code`) and `crm.contact_relationships` (UUIDv7 id, `relationship_type_id`, `contact_a_id`, `contact_b_id`, dates, `end_reason` in `ended, deceased, merged, correction`, overlap exclusion) (`:102-153`). Implementation deferred (session memory `households-architecture-standalone.md`: "implementation DEFERRED until a signed customer"). **Ruled / Planned.** --- ### What enrichment would reuse / what is missing (Inferred) Everything in this section is my inference from the Verified facts above. #### Reuse as is 1. **The contact and organization aggregates** (`createContact`/`updateContact` and the organization equivalents) as the only write path: normalisation, refusals, job-title sync, affiliation rules, custom fields, locks, archive refusal, no-op detection (an enrichment pass that finds nothing new writes nothing and moves no version), change events and revisions all come free. 2. **Columns for the common provider attributes** (photo, job title, socials, gender, pronouns, DOB, description, website, domain, logo) and **custom fields** for the rest (industry, size, revenue, other networks), on contacts and organizations. 3. **The asset pipeline** for avatars and logos, with its SSRF guards, limits, idempotent retries and per-item preparation in data jobs. 4. **Identity matching**: match keys, scorer, free-mail list, synchronous checks, `origin_id` key; the planned match-or-create function is exactly the "which record is this provider profile" question. 5. **Data Job Engine** operations for bulk application, with job-attributed revisions, per-row outcomes and scalar rollback; the crosswalk for provider ids if a provider is treated as a `source_system`. 6. **Record history** for after-the-fact provenance at save level ("this field changed in the enrichment job X on date Y"), queryable by `source`, `actor_id`, job. 7. **The D-25 disconnected pattern** (a value with provenance, `needs_verification`, explicit verify/relink/remove at the version read) as the model for staff review of an enriched or suggested value. 8. **Attention**: the registered but unused responsibility `crm.data_quality_review` is a ready audience for "enrichment suggestions to review" items, and the duplicate-review destination was designed to grow into a "data quality center (enrichment, missing information, validation later)". 9. **Merge** policies already treat socials/photo/profile timestamps as staff-chosen scalars and never widen consent. #### Missing or blocking 1. **A write path for a non-CRM caller**: the `contacts`/`organizations` service functions (CRM-M15) do not exist, and `crm.record_revisions_sys_source_check` refuses `service` — a migration and a console vocabulary change are needed before any service-principal write commits. The alternative is enrichment running as a CRM job (`source = job`). 2. **Field-level provenance**: no per-field source, provider, confidence, observed-at or last-verified for scalars or custom values; only addresses/phones have `source_mode`, and only for organization-sourced values. A provider-sourced value is indistinguishable from a staff-typed one except through revisions. 3. **A "do not overwrite human edits" rule**: nothing records that a value was entered or confirmed by staff or by the member (portal writes are only `sys_source = portal` per save; the portal profile routes are not built). The PATCH contract has no "fill if empty" or "only if unchanged since" per field, beyond `sys_version` for the whole record. 4. **Suggestion / pending-change storage**: no table or API for proposed values awaiting review; the only review queues are duplicate candidates and attention items. 5. **Semantics of the existing timestamps**: `date_profile_last_updated` and `date_social_profile_last_updated` are client-writable with no server meaning; an architecture must decide whether they become enrichment bookkeeping (server-owned) or are retired. 6. **Civic districts have no writer**: declared `read_only` in metadata, so the API cannot set them; a district provider (the proposal notes Google Civic's 2025 retirement must be verified) needs a server-side writer. 7. **Employer resolution**: a provider's employer must resolve to one existing organization (two or more → report, never guess); the domain lookup (OR-05) and organization match-or-create are not built; creating organizations from provider data hits duplicate `block` rules and affiliation side effects (automatic roles, compliance). 8. **E-mail model limits**: three untyped slots (CRM-Q6 open), primary uniqueness, no verification/bounce status; phones have no line-type/validity; addresses have no verification flag (postal verification out of scope, extension point only). 9. **Consent and eligibility**: the CRM consent ledger is ruled but not built; there is no data-processing/profiling consent concept at all; there is no member flag in CRM (membership lives in services-membership). 10. **Events for triggering**: only `crm.contact.merged` is published; `crm.contact.created/updated` and organization events are planned (CRM-M15). An enrichment-on-create trigger needs either those events or a CRM change-event handler. 11. **Rollback gaps**: collections (e-mails, phones, addresses, affiliations) and custom fields written by a bulk job cannot be rolled back; D-37/D-38 version capture and source-revision ordering are not built for CRM updates. 12. **Query exposure needs a migration**: the contacts query view is an explicit select list, so any new contact column for enrichment status must be added to `crm.contacts_for_queries` deliberately (or deliberately kept out); new sensitive provider fields need `neverIndexed`, metadata `exclude`, or history `redact` decisions. 13. **Merge coverage**: any new table holding contact or organization ids (suggestions, provider snapshots) must be classified in `modules/merges/references/crm-references.ts` or the coverage test fails; a cross-application copy would need a merge participant under CRM-ADR-12. ## 20. R3: Platform core, evidence for the Data Enrichment architecture ## R3: Platform core, evidence for the Data Enrichment architecture Read-only investigation of the Platform macroservice and its shared libraries, for the question "what does an enrichment capability (bulk and on-demand enrichment of contacts and organizations, AI-assisted research, a monthly budget, suggestions awaiting approval) get from Platform today, and what would it have to add?" | | | | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Repositories read | services-platform `origin/master` **43478ffda** (the enrichment worktree is at the same commit); macroservice-libraries `origin/master` **49728e6**; services-crm `origin/master` **e818ee48** (one file, for a seam) | | Not covered here | `modules/integrations` (investigator R1), `modules/data-jobs`, `modules/queries`, `modules/metadata` (other investigators). Each is named only where it is a seam | | Written | 2026-10-22, by a read-only agent; no source, build or deployment was touched | **Labels.** **Verified**: read at the cited path and line. **Planned**: written in an approved or proposed design but not built. **Decided (Andrew, date)**: a recorded decision of Andrew's, cited where it is recorded. **Inferred**: this report's reading of the evidence, not stated anywhere. **Absent**: looked for and not found (the search is stated). **Citation form.** `services-platform::` and `macroservice-libraries:packages//src/:`. Lines are those of the commits above. --- ### Summary 1. **One execution context, built only at a boundary** (an HTTP route, a job task, an event intake), carries the tenant, the caller, the dispatcher's access evaluation, the request origin, the logger, telemetry, the database executor and the tenant's time zone. Caller kinds are `console`, `portal`, `service` (another macroservice), `system` (a job acting as the macroservice) and `none`; a signed context may also speak for a `machine` principal (an API client). **Verified.** 2. **Provenance reaches the database on every transaction** as eleven transaction-local `app.*` settings, including `app.source` (`console`, `portal`, `public`, `integration`, `job`, `maintenance`, `service`), `app.operation`, `app.job_id` and `app.originating_user_id`. `integration` is reserved and nothing maps to it. There is **no slot for "which provider" or "what confidence"**. **Verified / Absent.** 3. **Permission keys** are `..`, declared in code with `definePermissions`, published in the service's registry manifest at deploy and reconciled into the catalog; tiers `read`/`user`/`admin` decide the generated built-in policies. **`alsoRequires` does not exist in any commit** (it is Andrew's 2026-10-21 decision for dataset queries, not landed). **Verified / Absent.** 4. **Two gates sit before every handler**: the application access key `.application.access` and the licence, both enforced everywhere since 2026-10-13. `platform` and `hq` are exempt and `crm` is always available, so enrichment keys under `platform.*` or `crm.*` would never be licence-gated; a separately sold enrichment needs its own application key. **Verified / Inferred.** 5. **Service-to-service calls are direct Lambda invokes authorized by IAM**, under a principal registry (`svc_`, integration adapters `svc_integration_`) that HQ can suspend without a deploy. **Verified.** 6. **Provider credentials are the Integration Platform's**: Secrets Manager under `/rhythm/integrations//...`, a dedicated KMS key, a broker that leases values for minutes; no secret byte in a table. **Verified.** 7. **There is no generic tenant-settings store.** Each capability owns a typed per-tenant table where "no row means the default". Two shapes exist: tenant-administered (forced RLS, `sys_version`, console routes) and staff-set (RLS-exempt, HQ `system` routes, an audit row per change). Defaults a workspace starts with are seed sets reconciled on every deploy. **Verified.** 8. **The only built metering is the LLM ledger**: credits (1/100 cent), a monthly allowance per tenant month, a single-statement authorize/settle/release, an in-flight cap, windowed overrides, 80 %/100 % crossings. The general entitlements and resource registry (licensing v2) is decided in direction and **not built**. **Verified / Decided / Absent.** 9. **The LLM module runs in Platform's process only**, Anthropic only (no Bedrock), one operation (`queries.generate_query`). How a second macroservice consumes it is decided (a library plus a Platform metering service function, built "when the first consumer exists") and not built. It never persists anything a model said. **Verified / Decided.** 10. **Audit is three things**: `platform.audit_events` (a closed list of event types, at most 1 KB of allow-listed metadata, append-only, no retention policy), `platform.security_audit_events` (`before`/`after` JSON, `reason`, `widening`) and the record change history (`record_revisions`: field-level changes, `sys_source`, `sys_job_id`, kept for the life of the tenant). None has a provenance or confidence field. **Verified / Absent.** 11. **Business events are a per-schema transactional outbox**, a dispatcher (every minute plus a post-commit nudge), SQS, and the subscriber's own `_event_intake` Lambda. No EventBridge bus. At least once, no ordering, deduplicate on `envelope.id`, 90-day ledger. **CRM publishes only `crm.contact.merged` today**, although the catalogue declares `crm.contact.created` and `.updated`. **Verified.** 12. **Background work has five sanctioned shapes**: a function, queued work (queue table, dispatcher, SQS consumer), a Durable Job (Step Functions plus the job engine, acting as the system), a Data Job (bulk record changes on a person's behalf, the person's authority re-checked per batch), and `macroservice-schedules` for tenant-defined recurrences. System cron is a fixed EventBridge rule. **Verified.** 13. **Workflow actions are registrable by any application**: `defineActionType` (params and outputs as JSON Schema, `inline`/`durable`/`host`, a service-function executor idempotent on the run id, a required permission), sent in a manifest at deploy. Tenant automation rules ("when event X, do actions") are raised by the event owner. An "Enrich contact" durable action fits the contract. **Verified / Inferred.** 14. **"Needs attention" is Attention**: items published in the application's own transaction through SECURITY DEFINER SQL, definitions registered at deploy, audiences by responsibility, role or user; visibility, never authority. **There is no generic approve/reject suggestion queue.** **Verified / Absent.** 15. **Contact photos by URL go through the asset pipeline inside the owning service's save** (SSRF guard, 10 MiB image cap, 20 s budget, byte sniffing, SVG refused). New tables follow forced RLS, configuration string ids versus UUIDv7 record ids, snake_case data and camelCase code, observed boundaries and two-dimension EMF metrics. R2-DEV-001 v1.6.2 fixes the plan template (including §7a Entity capabilities); **the next Platform milestone id is PLAT-M30**. **Verified.** --- ### 1. Execution context #### 1.1 Shape - **Verified.** "One execution enters the system, one execution context is created, and that context flows down through the application ... the execution context is always the first argument" (`services-platform:docs/execution-context.md:3-6`). - **Verified.** The type has `request`, `caller`, `access?`, `grants?`, `requestOrigin`, `tenantId?`, `dataJob?`, `timeZone`, `workload?`, `readerRouting?`, `onReader?`, `databaseUsage`, `deadlineAtMs?`, `remainingMs()`, `logging`, `telemetry`, `db`, `clock`, `now`, `moment()`, `isInTransaction`, `withTransaction()` (`macroservice-libraries:packages/execution/src/execution-context.ts:95-139`; quoted in Appendix A.1). The Platform doc's table of fields is `services-platform:docs/execution-context.md:12-25`. - **Verified.** What does not belong in it: "records, inputs, a capability's configuration, anything a function would otherwise take as an argument" (`services-platform:docs/execution-context.md:38-41`). An enrichment policy is therefore an argument or a read, never a context field. **Inferred.** #### 1.2 Who the caller can be - **Verified.** `CallerAccessType = AccessType | 'system' | 'none'`, where `AccessType = 'console' | 'portal' | 'service'`; "`service` for another macroservice whose IAM role invoked a service function, `system` for a job's own actor, `none` for an unidentified public request" (`macroservice-libraries:packages/authorization/src/security-context.ts:23-27`; `macroservice-libraries:packages/authorization/src/contracts.ts:36-44`). - **Verified.** A signed context names a **principal kind** `console | portal | machine`; `machine` is "an API client ... the identity an integration's credential is", ids `cli_` (`macroservice-libraries:packages/authorization/src/contracts.ts:46-57`). - **Verified.** The caller carries `actorId` (the id stamped as `sys_created_by_id`), `accessType`, `customerId?`, `tenantId?`, `identityId?`, `sessionRef?`, `isStaff`, `accessibleTenants`, `contactId?` (a portal user's linked CRM contact), `permissions?` and `originatingUserId?` (`macroservice-libraries:packages/authorization/src/security-context.ts:29-79`). - **Verified.** A service caller: `serviceCaller(principal, originatingUserId?)` reaches every tenant and holds every permission ("IAM has already decided what it may call"); `originatingUserId` is "recorded and never evaluated" (`macroservice-libraries:packages/authorization/src/security-context.ts:184-203`). - **Verified.** A job's caller: `SYSTEM_JOB_ACTOR = 'platform-system'`, `systemJobCaller()` builds `accessType: 'system', isStaff: true, accessibleTenants: '*'` (`macroservice-libraries:packages/execution/src/execution-context.ts:332-336`). - **Absent.** No caller kind named `integration`. `CHANGE_SOURCES` reserves the word: "`integration`, reserved for API-client callers (the platform has no such caller type yet, so nothing maps to it today)" (`macroservice-libraries:packages/db/src/session-context.ts:62-69`). Searched: `CallerAccessType`, `CHANGE_SOURCES`, `changeSourceOf`. #### 1.3 How it reaches persistence and audit - **Verified.** Every transaction sets, in one statement, `app.tenant_id`, `app.user_id`, `app.customer_id`, `app.is_staff`, `app.correlation_id`, `app.source`, `app.operation`, `app.request_id`, `app.job_id`, `app.job_batch_id`, `app.originating_user_id`, and switches to the application role (`macroservice-libraries:packages/db/src/session-context.ts:48-60`, `:135-160`). The last seven "are read by change-capture triggers only ... no policy reads [them], so they grant nothing" (`:43-47`). - **Verified.** The mapping from context to session: `dbSessionOf` takes the actor from `ctx.caller.actorId`, the operation from the request's `operationId` (or a maintenance reason), the job and batch only from `ctx.dataJob`, and the originating user only for a `service` caller (`macroservice-libraries:packages/execution/src/execution-context.ts:257-276`). `changeSourceOf`: maintenance, then `job` for a job origin or a system caller, then `portal`, `public`, `service`, else `console` (`:283-290`). - **Verified.** Row history reads these: record revisions carry `sys_actor_id`, `sys_source`, `sys_operation_name`, `sys_correlation_id`, `sys_request_id`, `sys_job_id`, `sys_job_batch_id` (`services-platform:modules/custom-objects/db/record-revisions.table.ts:22-24`, `:80-87`). A change with no `app.source` or `app.user_id` fails its COMMIT (`services-platform:docs/record-history.md:99-101`). - **Verified.** Security audit rows take the actor and `acting_for_user_id` from the execution's caller unless the event states them (`services-platform:docs/audit-events.md:54-58`; `services-platform:docs/security/service-calls.md:20-23`). #### 1.4 How a job, an event handler or an integration callback gets one - **Verified (job).** `createJobExecutionContext({ operationId, jobId, tenantId?, requestId?, correlationId?, requestedBy?, executionId?, ... })`: the system caller, origin `job` with no address, a logger bound to `jobId`/`requestedBy`/`executionId`, the tenant's zone. "A Step Functions task rebuilds it from its payload on every invocation; nothing of a context is serialized into an execution" (`macroservice-libraries:packages/execution/src/execution-context.factory.ts:446-494`). - **Verified (data job).** A data-job batch "does not act as the macroservice": it resolves the submitter's grants fresh, evaluates them with the dispatcher's evaluator under the same application-access and licensing modes, stamps every row with the submitter and sets `app.job_id`/`app.job_batch_id`; a person disabled or unlicensed "is refused at the next batch" (`macroservice-libraries:packages/execution/src/data-job-context.ts:1-25`). - **Verified (event subscriber).** The intake "runs `handle` in the event's tenant (`withTenant`), using a job-style context with origin `internal` and the publishing request's correlation id" (`macroservice-libraries:packages/events/README.md:130-133`). `RequestOrigin.origin` is `'request' | 'internal' | 'job'` (`macroservice-libraries:packages/authorization/src/contracts.ts:257-262`). - **Seam (R1).** An integration adapter acts as principal `svc_integration_` (`macroservice-libraries:packages/authorization/src/security-context.ts:149-169`); webhook ingress is in `modules/integrations` and is R1's. #### 1.5 The "source" concept - **Verified.** `app.source` / `sys_source` is the only "source" concept, and it describes the **channel** of the write (console, portal, job, service...), not the origin of a value (`macroservice-libraries:packages/db/src/session-context.ts:62-72`). - **Verified, discrepancy.** Platform's `record_revisions` admits `REVISION_SOURCES = ['console', 'portal', 'public', 'integration', 'job', 'maintenance']`, without `service` (`services-platform:modules/custom-objects/db/record-revisions.table.ts:50`, check at `:92`), while the library can set `app.source = 'service'` (`macroservice-libraries:packages/db/src/session-context.ts:69`). **Inferred:** a service caller writing a custom-object record would fail the revision's check constraint; CRM's own table was not read here. --- ### 2. Security and authorization #### 2.1 Permission keys - **Verified.** "A capability defines the permissions of one resource once — `definePermissions('platform', 'customers')` — and hands routes the resulting object; no route spells a key as a string" (`macroservice-libraries:packages/authorization/src/permission-catalog.ts:6-8`). Tiers: `read` is read; create/update/delete are user; `bulk` is admin; on `category: 'configuration'` writes are admin; "a `sensitive` key must state its tier explicitly" (`:14-16`). The `Permission` shape: `key`, `app`, `resource`, `action`, `description`, `tier`, `sensitive?`, `staffOnly?`, `resourceType?`, `aliases?` (`:49-65`). - **Verified.** Key grammar in the manifest validator: `/^[a-z0-9]+(?:[-_][a-z0-9]+)*(\.[a-z0-9]+(?:[-_][a-z0-9]+)*){2}$/` (`services-platform:modules/security/authorizer/registry/registry-manifest.ts:27`); the naming standard spells keys `.` (`services-platform:docs/naming-conventions.md:36`). Platform's own declarations use kebab-case resources (`platform.llm-usage.read`, `services-platform:aws/platform.service.ts:1508-1511`). - **Verified.** A route's requirement is `{ access, all, any, resource?, evaluates, impersonation }`; `evaluates` are "further keys the handler checks itself; they ride in the slice" (`macroservice-libraries:packages/authorization/src/contracts.ts:70-80`). - **Verified.** A handler that needs a key its route does not require reads `ctx.caller.permissions` through `hasPermission` (`macroservice-libraries:packages/authorization/src/security-context.ts:65-73`); a repository asks the dispatcher's evaluation with `requireAccess(ctx, key, id)` or narrows lists with `accessFilter` (`services-platform:docs/execution-context.md:17`). #### 2.2 How an application registers its keys - **Verified.** The registry manifest carries `routes`, `permissions: PermissionDefinition[]` and `builtInPolicies?` (`macroservice-libraries:packages/authorization/src/contracts.ts:464-478`). Publishing prepares the catalog reconciliation, then commits it, then flushes the authorizer cache (`:495-504`). - **Verified.** Generated tier policies `policy::read-only|user|administrator` hold every key of that tier; declared built-ins are `defineBuiltInPolicy({ key: 'policy::', app, rules, formerKeys })`, shipped by each service in its own manifest; "A service may declare built-ins only for its own apps" (`services-platform:docs/security/security-administration.md:146-171`). - **Verified.** "Shipping a built-in never grants it to anyone. Defaults exist only through built-in roles" (`services-platform:docs/security/security-administration.md:189-192`). - **Verified.** Each application has `.application.access`; "a person may use that application's permissions in a tenant only while they hold that key there" (Decided, Andrew, 2026-10-11), enforced everywhere since 2026-10-13; `platform` and `hq` are exempt (`services-platform:docs/security/application-access.md:3-6`, `:14-15`, `:28-33`). - **Verified.** The licence step answers 403 `app_unlicensed` before the handler; "`platform`, `hq` and `crm` are exempt in every mode" (`services-platform:docs/licensing.md:86-90`). #### 2.3 `alsoRequires` - **Absent.** `git grep alsoRequires origin/master` in services-platform finds nothing, and `git log --all -S alsoRequires` finds no commit; the libraries' `authorization` package has no such field. - **Decided (Andrew, 2026-10-21, recorded in the session memory `dataset-query-needs-read-too`, not in the repository).** A dataset query needs both the dataset's `.query` key and the records' `.read` key, to be enforced by `QUERY_DATASETS` entries gaining `alsoRequires`. A fleet job was briefed; it has not landed. The queries investigator owns the detail. #### 2.4 Role holders - **Verified.** `platform.role_holders(p_tenant_id, p_role_id, p_resource_type?, p_resource_id?) RETURNS TABLE (user_id, contact_id, display_name, email_address)`, called inside the caller's tenant transaction (`services-platform:docs/security/role-holders.md:12-32`). Counts active, in-force, unconditioned assignments at tenant, customer or the named resource scope, never platform scope (`:51-69`). EXECUTE is granted to `rhythm_table_reader`, which every macroservice role is a member of (`:117-122`). #### 2.5 Service principals and service calls - **Decided (Andrew, 2026-10-17).** A service calls another "by invoking its Lambda function directly, never by HTTP"; "the called function trusts the invoker"; "Authorization is IAM"; sensitive capabilities "live in separate functions so that the invoke permission is the permission"; routes only a service may call have access level `service`; the person acted for travels as provenance (`services-platform:docs/proposals/service-principals.md:11-20`). - **Verified.** Registry `platform.service_principals` (`id = 'svc_' || service`, `kind` `macroservice|integration`, `status` `active|suspended`); nothing deletes a principal; HQ suspends and resumes through `system` routes; a suspension reaches every callee within one minute (`services-platform:docs/security/service-calls.md:25-36`, `:84-102`). - **Verified.** The fixed list is `svc_platform, svc_crm, svc_accounting, svc_commerce, svc_events, svc_membership, svc_certifications, svc_learning, svc_marketing, svc_websites, svc_submissions` (`macroservice-libraries:packages/authorization/src/security-context.ts:130-142`). An enrichment capability hosted in a new macroservice would need a new entry here and a library release. **Inferred.** - **Verified, doc stale.** `service-calls.md` §7 says `serviceCaller` "refuses any principal outside the constant" (`services-platform:docs/security/service-calls.md:150-152`), but the library now also admits `svc_integration_` (`macroservice-libraries:packages/authorization/src/security-context.ts:171-177`, `:189-194`). - **Verified.** Registry rules enforced at manifest generation: a `service` route must be served by a service function; a route whose key is `sensitive` only through a `sensitive` function (`services-platform:docs/security/service-calls.md:120-129`). Platform's service functions today: `workflow_actions` and `eligibility`; "Quick Search indexing, audit events, number seeds" are named candidates (`:131-143`). #### 2.6 Integration credentials - **Verified.** Tenant credential secrets `/rhythm/integrations////`; platform-scoped `/rhythm/integrations///`; "No secret byte is stored in a Platform table" (`services-platform:docs/security/integration-credentials.md:12-21`). - **Verified.** A dedicated KMS key `alias/platform-integrations-credentials-`; only Platform's service role holds the grants; "Adapters' roles receive nothing" (`:25-55`). - **Verified.** The broker leases values in memory for five minutes by default, only for `active`/`degraded` connections, and audits every lease in `integration_audit_events` (`:123-146`). Rhythm's own provider keys (an enrichment vendor's master key would be one) are written by staff through `PUT /hq/integrations/{key}/secrets/{purpose}` (`:103-121`). - **Verified.** Extension handler secrets are a separate prefix, `/rhythm/extensions//`, write-only for the service role (`services-platform:docs/extensions.md:108-112`). --- ### 3. Tenant configuration #### 3.1 What exists - **Absent.** No generic tenant-settings or feature-flag table. Searched `modules/*/db/*.table.ts` for setting, policy, config, allowance, budget, limit, preference; `docs/*.md`, `docs/tenants`, `docs/standards` for "feature flag". The `tenants` row holds identity and lifecycle columns only (`services-platform:modules/tenants/db/tenants.table.ts:130-179`). - **Verified.** Per-capability tables, each with its own typed shape. Two patterns: | Pattern | Example | Key, RLS, writer | Citation | | ------------------------------------------------------------ | ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | Tenant-administered configuration of a registered definition | `platform.attention_settings` | `(tenant_id, definition_key)`; forced tenant policy; `values` JSON "inside the declared bounds" of the definition's `settings_schema`; writes carry `sys_version`; "A missing row is the definition's declared behaviour" | `services-platform:modules/attention/db/attention-settings.table.ts:1-23`, `:34-49` | | Tenant-administered on/off per channel | `platform.notification_channel_settings` | `(tenant_id, id)`, id is the channel; "A missing row means the channel's default"; `settings` JSON per channel (SMS `monthly_segment_ceiling`) | `services-platform:modules/notifications/db/notification-channel-settings.table.ts:1-12` | | Tenant-administered per record type | `platform.quick_search_configurations` | `(tenant_id, record_type)`; a configuration `version` separate from `sys_version`; bounded list | `services-platform:modules/quick-search/db/quick-search-configurations.table.ts:1-15` | | Staff-set limits | `platform.llm_tenant_limits` | primary key `tenant_id`; "no row means every platform default applies"; typed columns, not JSON ("Every Platform setting is a typed column"); **no RLS** because staff write from no tenant | `services-platform:modules/llm/db/llm-tenant-limits.table.ts:1-23`, `:31-48` | | Staff-set eligibility override | `platform.snapshot_tenant_policies` | one row per tenant and type or `'*'`; precedence type row, `'*'` row, default; written only through HQ | `services-platform:modules/snapshots/db/snapshot-tenant-policies.table.ts:1-13` | | Staff decision, revoke-never-edit | `platform.licence_overrides`, `platform.data_export_entitlements` | one row per decision with a required reason; revoked, never updated; in force by window | `services-platform:docs/licensing.md:97-110`; `services-platform:modules/licensing/db/data-export-entitlements.table.ts:1-22` | - **Verified, discrepancy.** The `llm_tenant_limits` header still says "Nothing reads it yet" (`services-platform:modules/llm/db/llm-tenant-limits.table.ts:5`), while `docs/llm.md` says the limits are "written and enforced" (`services-platform:docs/llm.md:352-354`) and `modules/llm/services/persistence/limits.ts` reads it. #### 3.2 Defaults a workspace is given - **Decided (Andrew, 2026-10-19, D-48).** Baseline configuration "is declared in code, versioned, and reconciled idempotently into every existing workspace on every deployment"; never overwrites a customization (`services-platform:docs/standards/baseline-configuration.md:3-7`). Mechanism: a participant's `SeedSet`s with stable keys, `ON CONFLICT DO NOTHING`, the seed ledger `.tenant_seed_applications`; "a migration never inserts tenant configuration" (`:22-47`). #### 3.3 Application access and entitlements per tenant - **Verified.** `platform.tenant_applications`, one row per tenant and application (`provisioning`, `provisioning_failed`, `enabled`, `suspended`); available = always-available or licensed and enabled (`services-platform:docs/licensing.md:9-33`). The registry of applications is the `APPLICATIONS` constant reconciled into `platform.applications`; "Adding an application is therefore a pull request" (`services-platform:modules/licensing/catalog/applications.ts:1-8`). #### 3.4 An enrichment policy, following the pattern (Inferred) - A tenant-editable policy (which fields, which providers, auto-apply versus review, schedule) follows `attention_settings` / `notification_channel_settings`: a table owned by the enrichment module, keyed `(tenant_id, )`, forced RLS, `snakeSystemColumns()`, "no row means the declared default", writes carrying `sys_version`, the defaults as a seed set if a starting row is wanted. - A Rhythm-set ceiling (the monthly budget a plan includes, an off switch) follows `llm_tenant_limits`: one row per tenant, typed columns, RLS-exempt with its reason in `db/row-level-security.ts`, written only by an HQ `system` route with an audit row and a `reason`, `widening` true when it raises spend (`services-platform:docs/llm.md:861-864`). --- ### 4. Licensing, entitlements, metering, credits #### 4.1 Built - **Verified.** Licence states (`always_available`, `licensed_by_override`, `licensed_by_contract`, `in_grace`, `expired`, `withheld_by_override`, `not_licensed`) and the operational half (`services-platform:docs/licensing.md:17-33`). - **Verified.** The grant slice carries `licensedApps`; the dispatcher judges licences without a database read (`services-platform:docs/licensing.md:322-336`). - **Verified.** The HubSpot source "is built but has no key" (`services-platform:docs/licensing.md:92-95`). - **Verified.** A quantitative entitlement exists for one capability: `platform.data_export_entitlements` with `executions_per_period`, `period` (`contract_year`, `calendar_year`), `minimum_interval_days`, `max_concurrent_executions`, `asset_budget_bytes`, defaults when no row (`services-platform:modules/licensing/db/data-export-entitlements.table.ts:1-16`, `:40-55`), described as "the licensing shim ... until the entitlements platform exists" (`:2-4`). - **Verified.** The LLM credit ledger (section 9.4) is the only consumption meter with enforcement. #### 4.2 Decided, not built - **Verified status.** "Status: version 2, 2026-10-19. Nothing in this proposal is built" (`services-platform:docs/proposals/licensing-entitlements-billing/README.md:5`). Rhythm 2.0 is greenfield, no grandfathering (Decided, Andrew, 2026-10-19 evening; `:32`). - **Decided (Andrew, 2026-10-19).** E-01 to E-27 answered the same day (`services-platform:docs/proposals/licensing-entitlements-billing/17-decision-register.md:83`), among them: the catalog in code (E-01), Rhythm as the authoritative ledger (E-02), every SKU may carry quantitative limits with the count the owning application's (E-04), soft thresholds versus enforced limits versus ceilings (E-05) (`:21-25`); the first enforced resources list, everything else warns (E-14, `:34`); licensing notices go to a responsibility through notifications (E-25, `:45`). - **Planned.** A resource definition each application registers in `licensing-registration.ts`: `key` `..`, `kind` `gauge|counter|high_water`, `scope` `tenant|customer`, `period` (`calendar_month`, `contract_month`, ...), `enforcement` `soft|enforced|metered`, `thresholds`, `safetyCeiling`, `reservable`, `measurement` (`services-platform:docs/proposals/licensing-entitlements-billing/04-quotas-and-resource-limits.md:5-29`). "Metered": usage above the allowance is billable only if the agreement permits (`:97-99`). Periods are computed in the tenant's zone (`:101-113`). - **Planned.** The runtime SDK `@rhythmsoftware/macroservice-licensing` with a reservation call (`services-platform:docs/proposals/licensing-entitlements-billing/07-runtime-enforcement-and-metering.md:19`, `:62`). **Absent** as a package: `macroservice-libraries:packages/` has no `licensing`. - **Absent.** The candidate list has no enrichment or AI resource (`services-platform:docs/proposals/licensing-entitlements-billing/04-quotas-and-resource-limits.md:41-61`). #### 4.3 What is reusable for "monthly enrichment budget" (Inferred) - The LLM ledger's single-statement reservation (Appendix A.6) is the only proven pattern for a hard per-tenant monthly budget under concurrency; the licensing v2 `counter` + `enforced` + `reservable` resource is the general home once built. - If enrichment spend is model spend, it can be the LLM allowance itself (one number per organization, Decided D-5 below). If it is vendor lookups, it is a second meter: either a licensing v2 resource (`.enrichment.lookups.monthly`) or, before that exists, a capability-owned ledger copied from `llm_allowance_periods`. --- ### 5. Audit events and record history #### 5.1 `platform.audit_events` - **Verified.** Nine event types: `record.viewed`, `record.merged`, `commerce.intervention`, `eligibility.change`, `custom_record.change`, `attention.acknowledged|snoozed|dismissed|escalated` (`services-platform:docs/audit-events.md:3-12`). - **Verified.** Columns: `id` (UUIDv7 by the database), `tenant_id`, `actor_user_id` (no foreign key), `event_type` (enum by check), `record_type` (the object id, also the dataset id), `record_id` (no foreign key), `occurred_at` (millisecond), `idempotency_key` (unique per tenant and actor), `metadata` (allow-listed keys per event type, at most 1,024 bytes) (`services-platform:docs/audit-events.md:62-81`). There is **no before/after, no field diff and no source column**. - **Verified.** Append-only by revoke (`:105-112`). **Retention: "There is none yet, and this is a known follow-up ... is Andrew's to make"** (`:126-132`). - **Verified.** Other services write through Platform-owned SQL functions in their own transaction (`record.merged` by `platform.merge_apply_*`; `commerce.intervention` "through a Platform-owned function") (`:160-166`, `:192-201`). Free text ("the reason and the evidence") is deliberately kept out: "this store's metadata is a kilobyte of short tokens" (`:212-215`). - **Verified.** "Adding a new event type is a larger change and a deliberate one": extend `AUDIT_EVENT_TYPES`, the metadata allow list, and both checks in a migration (`:465-469`). #### 5.2 `platform.security_audit_events` - **Verified.** Columns include `actor_user_id`, `actor_access_type` (`console`, `portal`, `service`), `acting_for_user_id`, `customer_id`, `tenant_id`, `entity_type` (enum), `entity_id`, `entity_version`, `action`, `widening`, `affected_keys`, `before` jsonb, `after` jsonb, `affected_count`, `request_id`, `outcome` (`succeeded|refused|failed`), `reason` (`services-platform:modules/security/db/security-audit-events.table.ts:110-155`). Used for configuration changes by staff and administrators (LLM limits, `services-platform:docs/llm.md:861-864`; extension handlers and bindings, `services-platform:docs/extensions.md:57-59`). #### 5.3 Record change history - **Decided (Andrew, 2026-10-11, accepted 2026-10-12).** The record change history design (`services-platform:docs/proposals/record-change-history.md:3`); retention "kept for the life of the tenant" (decision D5, `:800-803`); `app.source` reserved `integration` "for API-client callers" (`:272-277`). - **Verified.** Platform adopted it for custom-object records and associations; capture is CRM's SQL copied with `crm.` replaced by `platform.` (`services-platform:docs/record-history.md:3-9`, `:27-44`). - **Verified.** A revision row: `record_type`, `record_id`, `object_id`, `record_label`, `operation` (`created|updated|deleted`), `changes` jsonb, `changed_fields`, `capture_version`, `transaction_id`, `sys_recorded_at`, `sys_transaction_started_at`, `sys_actor_id`, `sys_actor_label`, `sys_source`, `sys_operation_name`, `sys_correlation_id`, `sys_request_id`, `sys_job_id`, `sys_job_batch_id` (`services-platform:modules/custom-objects/db/record-revisions.table.ts:58-87`). "No limit, deliberately ... retention is the life of the tenant" (`:26-28`). - **Verified.** "Fail closed. A change with no `app.source` or no `app.user_id` raises in the deferred settle"; the one way round is `app.change_events = off`, used by purge and reset (`services-platform:docs/record-history.md:99-105`). - **Seam.** Contact and organization history is CRM's `crm.record_revisions`; Platform's table holds association revisions of native records that "are kept and not shown" until CRM reads both (`services-platform:docs/record-history.md:147-151`). #### 5.4 Can an event carry "source: integration X, confidence Y"? (Absent / Inferred) - **Absent.** No provider, confidence, or provenance column in `audit_events`, `security_audit_events` or `record_revisions`; `audit_events.metadata` could carry short tokens only after a migration adds a new event type and its allow list. - **Inferred.** The attributable facts available on every enrichment write are: `sys_source` (`job`, `service`, or `console`), `sys_operation_name` (an operation id such as `enrichContact`), `sys_actor_id` (`platform-system`, `svc_`, or the person), `sys_job_id` (only for a data job). "Which vendor, which confidence" has to live in the enrichment capability's own table, joinable to the revision by `sys_correlation_id` or `sys_request_id`. --- ### 6. Business events - **Verified.** "A business event is a fact Platform states for anyone to hear ... The publisher knows no subscriber, and a subscriber's failure never reaches the transaction that committed the fact" (`services-platform:docs/business-events.md:3-6`). - **Verified.** Declared once with `defineEvent`, refused at load if it disagrees with the Integration Platform's catalogue `DOMAIN_EVENT_CATALOG` (`:13-15`). Naming `..`; payloads are "identifiers plus declared facts, snake_case, never a `sys_*` column and never the actor (the envelope carries both)" (`:30-31`). - **Verified.** Envelope: `rhythm_event: 1`, `id` (UUIDv7, the dedupe key), `key`, `version`, `tenant_id`, `occurred_at`, `subject { type, id }`, `scope`, `correlation_id`, `actor_id`, `replay`, `payload` (`:43-45`; `macroservice-libraries:packages/events/src/envelope.ts:34-47`, Appendix A.3). - **Verified (bus).** Outbox tables `.business_events` (ledger, 90 days) and `.business_event_queue` written in the business transaction; a dispatcher every minute plus a post-commit nudge; one SQS message per event and destination; a consumer invoking each destination (`macroservice-libraries:packages/events/README.md:23-35`; `services-platform:docs/business-events.md:49-67`, `:104-113`). "There is no internal event bus either (README 7.7)" and no EventBridge (`macroservice-libraries:packages/events/README.md:216-221`). - **Verified (subscribing).** `defineEventSubscriber({ event, versions, intake_function_parameter, handle })` and `runEventIntake([...], { service })`; the subscriber is declared in the service's manifest and registered into `platform.event_subscribers` (`macroservice-libraries:packages/events/README.md:114-141`; `services-platform:docs/business-events.md:124-129`). - **Verified (guarantees).** Atomic with the business change; at least once; "No ordering, not even per subject"; "Idempotency is the subscriber's job, done on `envelope.id`"; "Gone is success" (`macroservice-libraries:packages/events/README.md:134`, `:143-160`). - **Verified.** External delivery (webhooks) belongs to the Integration Platform's router; the router destination is added only when `/rhythm/platform//integrations/event-router-arn` exists (`services-platform:docs/business-events.md:110-113`). - **Verified.** Platform publishes `objects.record.*` and `objects.association.*`; it subscribes to nothing yet (`services-platform:docs/business-events.md:17-28`, `:126-128`). - **Verified (the trigger an enrichment needs).** The catalogue declares `crm.contact.created|updated|deleted|merged` and the organization equivalents (`macroservice-libraries:packages/integration-contracts/src/domain-events.ts:56-63`), but CRM declares only `ContactMerged` (`crm.contact.merged`) in `services-crm:aws/crm.events.ts:26-27`. **Absent:** a published `crm.contact.created`. An enrichment triggered on contact creation needs CRM to publish it first. **Inferred.** - **Verified, doc stale.** `docs/attention.md:373-374` says "Platform publishes no business events yet"; `docs/business-events.md` shows it does since migration `20261020011722`. - **Verified (change events, a different thing).** A change event is the row delta a trigger captures, "reacted to by the owning service (or handed to another service's Lambda by a remote handler)" (`macroservice-libraries:packages/events/README.md:191-198`). --- ### 7. Jobs and background work #### 7.1 The five sanctioned shapes | Shape | When | Mechanism | Citation | | ----------------------------------- | -------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------- | | A function the request calls | finishes in a second or two, idempotent, recovery is a person asking again | none | `services-platform:docs/jobs.md:29-33` | | Queued work | one piece of work a save must not lose and must not wait for | queue row in the save's transaction, dispatcher (nudge + every minute) to SQS, deleted on acceptance; Lambda consumer does the work; SQS owns retries and the dead-letter queue | `services-platform:docs/queued-work.md:3-8`, `:13-18`, `:48-59` | | Durable Job | multi-step, outlives the request, must be resumed or settled unattended | Step Functions per job type plus `@rhythmsoftware/macroservice-jobs` (submission, steps, events, settlement, reconciler) | `services-platform:docs/jobs.md:3-5`, `:13-18` | | Data Job | bulk record changes a person asked for; the domain owns the write | `@rhythmsoftware/macroservice-data-jobs`; interactive lane ≤ 500 items, bulk lane with Step Functions; receipts in the same transaction; submitter's authority per batch; rollback by compensating job | `services-platform:docs/proposals/rhythm-data-job-engine-plan.md:42-60`; `macroservice-libraries:packages/execution/src/data-job-context.ts:5-12` | | Recurring schedule (tenant-defined) | a rule the tenant sets ("every Monday") | `macroservice-schedules`: rows with `sys_next_run_at`, one EventBridge rule a minute per service, exactly-one claim per occurrence, queued run | `macroservice-libraries:docs/schedules.md:16-34` | - **Verified.** A system schedule is a fixed EventBridge rule on a function: e.g. `createScheduledInvocation(scope, alias, { schedule: events.Schedule.cron({ minute: '20' }), ... })` for attention evaluation (`services-platform:modules/attention/aws/logic/attention-evaluation.lambda.ts:78-79`). "No EventBridge Scheduler, and no AWS schedule per record" (`macroservice-libraries:docs/schedules.md:18-21`). - **Verified.** "Starting a Step Functions job is outside this standard, explicitly": `createJob` in the transaction, `startJob` (or `startJobOrLeaveToReconciler`) after the commit (`services-platform:docs/queued-work.md:20-27`; `services-platform:docs/jobs.md:111-121`). #### 7.2 The job model - **Verified.** Adding a job: tables from `jobRootColumns`/`jobStepColumns`/`jobEventColumns`, the kind in the cross-job views, a step catalog and `defineJobWorkflow`, `registerJobType` with `onSettledFailure` (mandatory), idempotent task operations taking `ctx` first, `JobTaskRefusal` for non-retryable failures, `createJobTaskHandler`, `createJobStateMachine`, submitters declared in `aws/cdk/platform.job-starters.ts` (`services-platform:docs/jobs.md:37-124`). - **Verified.** Long backlogs: a task may declare `repeat: { maxIterations }`, save its cursor on the job row and answer `{ more: true }` (`services-platform:docs/jobs.md:60-65`). - **Verified.** Retries and timeouts are Step Functions': "Nothing else in the system loops, waits or retries a job" (`services-platform:docs/jobs.md:15`). Execution input carries identifiers only (`:175-183`). - **Verified.** A person who submitted a job is told how it ended exactly once through `notifyRequester`; a system-submitted job declares nothing (`services-platform:docs/jobs.md:138-146`). - **Verified.** Monitors: `GET /jobs` (console, `platform.jobs.read`), `/hq/jobs` (staff) and `/tenant_jobs/{tenantId}`, across every macroservice through the cross-job views (`services-platform:docs/jobs.md:126-136`). #### 7.3 Concurrency and load - **Verified.** Admission permits per tenant (`db-interactive`, 16) and per cluster (`db-heavy`); `report` and `bulk` routes are throttled to 50/s burst 20 (`services-platform:docs/workload-governance.md:19-25`). - **Verified.** The LLM in-flight cap, 8 per tenant by default, inside the reservation statement (`services-platform:docs/llm.md:406-412`). - **Status.** The Data Job Engine plan header still says "the build is on hold" (`services-platform:docs/proposals/rhythm-data-job-engine-plan.md:3`), but the milestones README shows PLAT-M19 changes requested and PLAT-M20 In Progress (`services-platform:docs/milestones/README.md:72-75`). The data-jobs investigator owns the detail. #### 7.4 Which mechanism for which enrichment work (Inferred) - **Bulk enrichment a person starts from a list** writes many CRM records on that person's behalf: the Data Job Engine is the sanctioned path (receipts, per-record rejections, revision attribution by `sys_job_id`, rollback). The vendor or model calls before the write are the open question: a data operation calls them inside its batch, or a Durable Job gathers results first and then submits a data job. - **A nightly refresh nobody submitted** is a fixed EventBridge rule that submits one Durable Job per tenant (as the responsibilities and derived-assignment reconciles do, `services-platform:docs/responsibilities.md:28`; `services-platform:docs/row-level-security.md:61`). A per-tenant cadence the tenant chooses is a `macroservice-schedules` row. - **Enrich-on-create** is an event subscriber (once CRM publishes the event) that writes a queue row; queued work does the lookup. --- ### 8. Workflows and workflow actions - **Verified, naming trap.** `docs/workflows/` is the CI pipeline ("Workflows: the CI pipeline files, ready to copy", `services-platform:docs/workflows/README.md:1-5`). Business workflow actions are `docs/workflow-actions/`. - **Decided (Andrew, 2026-10-17).** Application processes and workflow actions: "all twelve decisions D-01 to D-12 taken by Andrew the same day as recommended" (`services-platform:docs/proposals/application-processes-and-workflow-actions/README.md:5`); D-01 is revised by the Submissions architecture of 2026-10-20 (`:1`). - **Verified.** Platform's module is the registry of trigger points and action types, templates, business calendars, automation rules and cross-service views of runs; "nothing here executes an action"; the engine is the library, running "inside each host's own transaction and schema" (`services-platform:docs/workflow-actions/README.md:3-14`). - **Verified (registration).** An application declares `defineTriggerPoint`, `defineActionType`, `defineOutcomeKind`, builds a manifest and invokes `PUT /workflow_actions_service/registry/{app}` on `platform_workflow_actions_service` at its deploy; the principal must be the application (`svc_crm` registers `crm`); validation, conflicts (409), retirement of undeclared entries (`services-platform:docs/workflow-actions/README.md:67-92`). - **Verified (contract).** `ActionTypeDefinition`: `key`, `app`, `label`, `category`, `params` (zod; JSON Schema in the manifest), `execution` (`inline|durable|host`), `executor` (`service_function` with `function_name`, `in_process`, or `host`), `idempotency` (`action_run|host_lock|last_write_wins`), `outputs`, `reversal?`, `hosts`, `host?`, `allowed_triggers`, `contract_version`, `accepted_contract_versions`, `permission?`, `mappable_paths?` (`macroservice-libraries:packages/workflow-actions/src/contracts.ts:30-31`, `:87-88`, `:571-616`; Appendix A.5). - **Verified (executor).** "`run.id` is the key it dedupes on"; `durable` runs "by the worker after commit, with the tenant's execution context; it may call another service. A failure it knows is final throws `ActionRefusal`; anything else is retried by the queue" (`macroservice-libraries:packages/workflow-actions/src/executors.ts:1-15`, `:81-97`). - **Verified (triggers).** Platform declares the automation host's `event` point and the library's timer points `timer.elapsed`, `timer.deadline`, `timer.sla` (`services-platform:modules/workflow-actions/contracts/platform-declarations.ts:170-172`); a business event becomes a trigger for automation rules only when its owner registers it as a trigger point of host `platform.automation_rule` (`services-platform:docs/workflow-actions/automation-rules.md:16-20`). - **Verified (automation rules).** "When this business event happens, do these actions"; the event owner's consumer calls `raiseForEvent`, which reads `enabled` rules' current published version (at most 200 per event) and calls `raise` in the owner's schema; publishing requires `platform.workflow-actions.automation-manage` and every action's `permission` (`services-platform:docs/workflow-actions/automation-rules.md:1-60`). - **Verified.** Platform-owned action types: `communications.send`, `notifications.notify` (declared and off), `workflow.set_context`, `sla.breach`, `audit.note` (`services-platform:docs/workflow-actions/README.md:94-100`). Example declaration: `services-platform:modules/workflow-actions/contracts/platform-declarations.ts:46-65`. - **Inferred.** "Enrich contact" is a `durable` action type with a `service_function` executor, idempotency `action_run`, `hosts: ['*']` or the CRM and automation hosts, `permission` an enrichment key. For it to fire on "contact created", CRM must publish `crm.contact.created` and register it as a trigger point of `platform.automation_rule`. --- ### 9. LLM module #### 9.1 What it is - **Verified.** "Every model call Rhythm makes goes through one function, `run_operation` ... one file in the repository may speak to a provider ... A unit-tier guard enforces the first" (`services-platform:docs/llm.md:3-7`). - **Verified.** Contract `runOperation(ctx, operationKey, { promptVariables, initialMessages, tools, handlers, finalToolNames, acceptFinal, purposeOf })` returns `{ invocationId, outcome, finalToolCall?, usage, credits, attempts }`; "A consumer never names a model, never holds an API key, never prices anything and never writes an invocation row" (`:14-28`). Closed refusal set (`:34-42`, Appendix A.6). - **Verified (prompt registry).** Operations declared in `modules/llm/catalog/llm-operations.ts`: provider, model, generation, loop limits, `prompt_ref`, `prompt_version`, preflight estimate; rates in `llm-rate-definitions.ts`; both published at deploy into append-only version histories (`services-platform:docs/llm.md:179-199`; `services-platform:modules/llm/catalog/llm-operations.ts:26-40`). One operation today, `queries.generate_query` (`:66`). - **Verified (provider).** `LLM_PROVIDERS = ['anthropic', 'stub']`; "only `anthropic` reaches a network" (`services-platform:modules/llm/runtime/provider.ts:22-24`). `openai` "may not be imported by any file at all" (`services-platform:docs/llm.md:1051-1052`). **Absent:** Bedrock (searched `docs/llm.md`, `modules/llm/runtime`). - **Verified (no state from answers).** "`run_operation` hands the final tool call back to its caller and persists nothing from a response but usage counts and ids. What is done with an answer is the consumer's, and so is validating it" (`services-platform:docs/llm.md:1053-1054`). - **Verified (no web search tool in use).** The rate model knows a `web_search_requests` dimension only in unit tests (`services-platform:modules/llm/tests/unit/catalog/classify-usage-dimensions.spec.ts:20`); no operation declares one. #### 9.2 Observability - **Verified.** `llm_invocations`, `llm_invocation_attempts`, `llm_executions` (one AI operation of many calls), daily rollup, threshold crossings (`services-platform:docs/llm.md:211-224`). Attribution columns from the context: `application_key`, `feature_key`, `actor_access_type`, `acting_for_user_id`, `request_id`, `correlation_id`, `execution_id` (`:235-248`). Metrics `llm.*` (`:340-348`). Request and response bodies captured for ninety days (Decided, Andrew, 2026-10-18, D-1; `services-platform:docs/proposals/llm-observability/11-decisions.md:7`). #### 9.3 How another app invokes it - **Verified.** Only Platform's queries module calls `runOperation` (`modules/queries/services/generate/*`); "No other macroservice imports the module or a provider SDK" (`services-platform:docs/proposals/llm-observability/01-current-state.md:36`). The module serves no route that calls a model (`services-platform:docs/llm.md:1049-1050`). Runtime access is given per function by `apply_llm_runtime_access`, today only to the queries API function (`:999-1003`). - **Decided (Andrew, 2026-10-18, D-5).** "library plus Platform metering service, built when the first consumer exists"; the service function would offer `open_invocation`, `record_attempt`, `settle_invocation` with deploy-time operation registration (`services-platform:docs/proposals/llm-observability/11-decisions.md:11`, `:70-76`). **Absent:** neither is built. #### 9.4 Credits and the ledger (precedent for metered consumption) - **Verified.** "A credit is one hundredth of a cent. A tenant has a monthly allowance (100,000 credits by default), plus the overrides in force, less any adjustments; the period key is the first day of the tenant's own month" (`services-platform:docs/llm.md:362-364`). Authorize holds the maximum, settle charges what was reported, release returns the hold (`:366-370`). The authorize statement is the whole decision (`:374-391`, Appendix A.6). Ceilings $1,000 effective, $100 per override, 92-day windows (Decided, Andrew, 2026-10-18, D-3; `11-decisions.md:9`). - **Verified.** Limits per tenant (`llm_enabled`, `monthly_allowance_credits`, `disabled_operation_keys`, `in_flight_cap`), each change one transaction with an audit row and a required reason (`services-platform:docs/llm.md:831-864`). - **Decided (Andrew, 2026-10-18, D-4).** 80 % and 100 % notices to the Administrator role holders "for now", later a responsibility (`services-platform:docs/proposals/llm-observability/11-decisions.md:10`). - **Status.** PLAT-M1 is "Awaiting Plan Approval" and PLAT-M2 to M4 "Proposed" (`services-platform:docs/milestones/README.md:12-15`) while `docs/llm.md` describes the limits and overrides as live: the code ran ahead of the milestone records. --- ### 10. Attention and responsibilities (where suggestions awaiting approval could surface) - **Verified.** Attention: "one place where every application publishes the conditions that matter ... Applications write items at source, inside their own transactions, through SQL functions, and run sweeps; Platform stores the items, resolves who sees them ... An audience is visibility, never authority" (`services-platform:docs/attention.md:8-14`). - **Verified.** An item is "One occurrence of a condition about one record": `definition`, `dedupe_key`, `generation`, class (`required|advisory|opportunity`), `about_*`, `scope_refs`, declared `facts`, `due_at`, `expires_at`, `destination_params`, status `open|resolved|expired`, rank (`services-platform:docs/attention.md:28-30`). Audiences: responsibility, user, role, contact, organization, eligibility (`:32`, `:99-100`). - **Verified.** Write path: `platform.publish_attention_item(...)`, `resolve_attention_item`, `resolve_unconfirmed_attention_items`, `attention_sweep_completed`, SECURITY DEFINER, granted to capability role `rhythm_attention_writer`; the source service must be the caller's application role (`services-platform:docs/attention.md:66-90`; Appendix A.7). - **Verified.** Definitions registered at deploy through `platform_attention_registration` with facts, templates, destination, actions each with a permission, settings schema, lifecycle and reminders (`services-platform:docs/attention.md:119-160`). - **Verified.** Not built: business events of items, portal read (held by Andrew 2026-10-18), tenant-authored definitions, HQ routes, teams (`services-platform:docs/attention.md:16-18`, `:371-380`). - **Verified.** Responsibilities: "A responsibility definition is a well-known operational area an application registers at deploy"; "A responsibility grants no permission" (`services-platform:docs/responsibilities.md:9-17`). - **Absent.** A generic "proposed change awaiting approval" record (suggested value, approve, reject, apply) in Platform. Searched `modules/*` table names and `docs/*.md` for suggestion, proposal, pending change, review queue. The nearest built things are attention items (a pointer to work, not the work) and the Submissions review engine (another repository, not read). - **Inferred.** An enrichment review queue is the enrichment capability's own table of suggestions; Attention carries "N enrichment suggestions await review" per record or per tenant, with an action whose permission is the enrichment review key; Notifications (`services-platform:docs/notifications.md:8-12`) carries "your bulk enrichment finished". --- ### 11. Files and assets (contact photo by URL) - **Verified.** The convention: "a save that assigns a new file to such a field succeeds only once that file is permanently stored, validated and owned by the caller's tenant ... Every macroservice does it through one shared pipeline ... and never writes a downloader or a copier of its own" (`services-platform:docs/assets.md:3-9`). - **Verified.** An external `http(s)` URL is classified "External resource" and imported to a new key (`services-platform:docs/assets.md:56-66`). - **Verified (import rules).** https/http only, no credentials, default port; every resolved address public unicast, lookup pinned against DNS rebinding; at most 3 redirects; headers in 3 s, body in 10 s, preparation in 20 s; 10 MiB image, 25 MiB document; no compressed body; bytes sniffed; JPEG, PNG, GIF, WebP for images; SVG and HTML refused; no malware scanning (`services-platform:docs/assets.md:202-219`). - **Verified (verification and storage).** Prepare outside any transaction (`preparing` → `ready`), then the capability's short transaction writes the record and attaches the rows; the repository guard `requirePreparedAssets` refuses a reference written without the pipeline (`services-platform:docs/assets.md:85-106`). Bucket and keys are Rhythm 1.0's, the key begins with the tenant (`:22-29`). Lifecycle `managed_assets` table per macroservice (`:221-233`). - **Verified.** The front-half upload route is `POST /files/{tenantId}/uploads` (`platform.files.upload`) (`services-platform:docs/files.md:11-19`). - **Inferred.** Because the guard sits in CRM's repository operations, an enrichment that sets a contact photo must hand the URL to CRM's save (a service call or a data operation), not write the field. --- ### 12. Addressing a record generically; extension points - **Verified.** The metadata registry holds "the registry of objects other macroservices own (CRM's Contacts and Organizations first), with their standard fields as the owner declared them" (`services-platform:docs/metadata.md:3-16`). The object id (`crm.contacts`) is the authoritative entity id (`services-platform:docs/standards/custom-field-capability.md:67`). - **Verified.** The same object id is used as the generic record address elsewhere: `audit_events.record_type` "The record's object id, which is also the query dataset id (`crm.contacts`)" (`services-platform:docs/audit-events.md:70`); attention `about.object_id` must be "a live registered object of the application" (`services-platform:docs/attention.md:135`); revisions carry `object_id` (`services-platform:modules/custom-objects/db/record-revisions.table.ts:66-67`). - **Verified.** Polymorphic references (a link to one of several record types) were built in metadata 0.22.0 (`services-platform:docs/proposals/entity-aware-queries-polymorphic-references.md:3`, `:9-15`). Metadata detail is another investigator's. - **Verified, naming trap.** `docs/well-known-objects.md` is not about record types: it is fixed-id development records such as the FAA tenant (`services-platform:docs/well-known-objects.md:3-9`). - **Verified (extensions).** Registry of extension points (`resolver`, `validator`, `transformer`, `action`; cardinality; scope kinds; modes; failure policies), event types and subscribers, written at deploy by `platform_extensions_registration`; tenant handlers (`lambda`, `https` refused, `rule`, `builtin`) and bindings per scope; every invocation recorded through capability role `rhythm_extension_invocation_writer` (`services-platform:docs/extensions.md:3-9`, `:17-27`, `:50-56`, `:61-81`). - **Seam (R1).** The Integration Platform's capability catalogue already declares `crm.contact.enrich`: "Fields a provider knows about an address (districts, representatives, geocodes), for CRM to write through its own persistence", `mode: 'sync'`, `personal_data: true`, `permission: invoke('crm')` (`macroservice-libraries:packages/integration-contracts/src/registry.ts:1081-1099`). - **Inferred.** "Enrich record of kind X id Y" can be addressed as `(object_id, record_id)` without CRM internals; writing the result cannot, because the owner's persistence (and its asset guard, history capture and events) must do the write. --- ### 13. Snapshots - **Verified.** "Monthly, versioned, point-in-time snapshots of tenant data", one run ledger for every type of every service, each owner writing its runs in the same transaction as its rows (`services-platform:docs/snapshots/README.md:1-17`). - **Verified.** A snapshot's `basis` is one of `captured`, `finalized`, `reconstructed`, `estimated`, `imported` (`macroservice-libraries:packages/snapshots/src/snapshot-states.ts:31-34`); Platform's own type "is not a report: its basis is `captured` from current state, so it is never history" (`services-platform:docs/snapshots/README.md:16`). - **Inferred.** Snapshots are monthly aggregates for reporting, not a per-record "what we knew at time T". Per-record history is the record revisions (section 5.3); "which provider said this, when" needs the enrichment capability's own append-only rows. --- ### 14. Conventions any new table must follow - **Verified (RLS).** Every transaction declares tenant and caller and drops to `rhythm__app` (NOBYPASSRLS); each tenant-owned table has one forced policy `USING/WITH CHECK (tenant_id = NULLIF(current_setting('app.tenant_id', true), ''))`; it fails closed; repositories still filter on `tenantOf(ctx)` (`services-platform:docs/row-level-security.md:10-42`). Every exemption is listed with its reason in `db/row-level-security.ts` (job roots, queue tables, registries, staff-written tables) (`:44-74`). - **Verified (partitions).** Partition by month only "operational history that expires as a whole month"; "A record that lives until somebody deletes it is never partitioned" (`services-platform:docs/partitions.md:11-16`). A partitioned table's uniqueness needs a companion key table (`services-platform:docs/attention.md:49-56`). - **Decided (Andrew, 2026-10-19, A4 of INTG-M1).** "a configuration table is keyed by an immutable string id, only a record by a UUIDv7" (`services-platform:docs/integrations/intg-m1-string-ids-notes.md:3-4`). Grammar `^[a-z](?:[a-z0-9]|_(?=[a-z0-9])){0,62}$`, derived from the display name, `sys_` reserved for shipped rows, keyed `(tenant_id, id)` (`macroservice-libraries:packages/records/src/configuration-ids.ts:1-22`, `:28-41`). - **Decided (Andrew, 2026-10-16).** "snake_case is ONLY for database tables and columns, domain record properties ... Code names are camelCase"; tables plural snake_case; API JSON, paths and event payloads snake_case; path variables camelCase; permission keys and object ids `.` (`services-platform:docs/naming-conventions.md:12-36`). - **Verified.** System columns start `sys_` (Decided, Andrew, DJE §21.4; `services-platform:docs/proposals/rhythm-data-job-engine-plan.md:3-5`); tables spread `snakeSystemColumns()` (e.g. `services-platform:modules/llm/db/llm-tenant-limits.table.ts:48`). - **Verified.** Platform's AGENTS.md sections that bind a new module: Columns Are Defined Once, Strings Trimmed Empty Is Null, Every Non-Aggregate List Has a Limit, Tables and Persistence Belong to Their Module, Execution Context First, Tenant Data Is Isolated by RLS, Every Query Has an Index Decision, Time Is the Tenant's, Files Go Through the Asset Pipeline, Background Jobs, Observability, Every Query Has Integration Coverage (`services-platform:AGENTS.md:153-606`, headings). --- ### 15. Observability - **Verified.** Every boundary opens a telemetry context; every log line and metric carries `correlationId`, `traceId`, `spanId`, service, environment, request, caller, tenant, and job ids in a job (`services-platform:docs/observability.md:9-27`). Correlation crosses Step Functions (`:32-35`). - **Verified.** Metrics through `ctx.telemetry.recordMetric(name, value, attributes)`, EMF, namespace `Rhythm`, dimensions `service` and `environment` only; ".duration" is milliseconds (`services-platform:docs/observability.md:64-70`). - **Verified.** Alarms: per function errors, throttles, duration near timeout; REST API 4xx/5xx rate; correlation discontinuity; job failure and timeout; all to SNS `rhythm-macroservice-alerts` (`services-platform:docs/observability.md:87-108`). - **Verified.** Adding to it: a new boundary is wrapped by `observeInvocation`; a new function by `createMicroserviceFunction` under `observeFunctions`; a new metric `'..count'` and "never a new dimension" (`services-platform:docs/observability.md:131-138`). Every module serves `GET //health` (`services-platform:AGENTS.md:550-551`). --- ### 16. R2-DEV-001 milestone standard and Platform milestone ids #### 16.1 The standard - **Decided (Andrew).** Version 1.6.2, approved 2026-10-20 ~11:00 ET, in force for all Rhythm 2.0 work (`services-platform:docs/standards/milestone-development-and-acceptance.md:5-9`). Four rules: no implementation without an approved plan; no handoff without a passing cumulative automated run; no acceptance without hands-on verification; one milestone In Progress and at most two unaccepted per application (`:13-18`). - **Verified.** Ids `-M`, never reused; app codes `ACCT, CRM, MEMB, EVNT, SUBM, CMTE, COMM, WEB, INTG, PLAT, LIB` (`:28`). Kinds: user-facing (live walkthrough), platform capability with a consumer, backend-only (guided verification) (`:30-36`). Architecture documents include a Milestone plan chapter (`:38`). `RPT` is used in the Platform README but is not in the standard's code list (`services-platform:docs/milestones/README.md:77-86`). - **Verified.** Stage A plan contents: objective; scope in and out; how it works for the user; assumptions each marked approved with a citation or not; decisions with example, options, consequences, recommendation; approach; deliverables; dependencies; **entity capabilities**; acceptance scenarios with priorities; walkthrough preview; risks (`services-platform:docs/standards/milestone-development-and-acceptance.md:72-85`). Short form allowed when no decision is open (`:90`). "An assumption is approved only if a recorded decision explicitly says so" (`:91`). - **Verified (the entity capabilities requirement).** "for every entity the milestone touches, whether it has custom fields, configurable forms and configurable page layouts (copied from the registry under R2-ARCH-001, never decided by the agent), what configurable console and portal behaviour this milestone delivers, and what is deferred and where it is recorded" (`:82`). The template's §7a columns: Entity, Custom fields, Console forms/layouts this milestone, Portal forms/layouts, Semantic layer, Shared capabilities reused/missing, Deferred and where recorded (`services-platform:docs/standards/templates/milestone-implementation-plan.md:50-56`). CRM contacts and organizations are the CRM entities classified with custom fields (`services-platform:docs/standards/custom-field-capability.md:34`). - **Verified (template sections).** 1 Objective, 2 Scope, 3 How it works for the user, 4 Assumptions, 5 Decisions for Andrew, 6 Approach, 7 Deliverables (screens, API routes and services, data, events and integrations), 7a Entity capabilities, 8 Dependencies, 9 Acceptance scenarios (`-M-S01`, cross-milestone `X01`, priority, layer, tier), 10 Walkthrough preview, 11 Pipeline, 12 Risks, then the approval line (`services-platform:docs/standards/templates/milestone-implementation-plan.md:1-90`). - **Verified.** Cumulative layers L1 (this milestone's P0/P1), L2 (every earlier milestone's P0/P1), L3 (cross-milestone journeys), L4 (everything, weekly); priorities P0/P1/P2; prove each scenario in the fastest conclusive tier (`services-platform:docs/standards/milestone-development-and-acceptance.md:174-198`). - **Verified.** "Each Platform capability (its own section in `services-platform/docs/milestones/README.md`, its own owning session) counts as an application of its own" (Decided, Andrew, 2026-10-19, v1.4) (`:202`). Developer-executed Chrome walkthrough before any handoff (Decided, Andrew, 2026-10-20) (`:109-115`). #### 16.2 Platform milestone ids in use - **Verified.** "PLAT-M1 to PLAT-M4 ... AI usage ... PLAT-M5 to PLAT-M8 ... custom objects; PLAT-M9 ... custom field capability registry; PLAT-M10 ... image and file fields; PLAT-M11 to PLAT-M14 ... Eligibility Engine; PLAT-M15 to PLAT-M18 ... responsibilities and attention; PLAT-M19 to PLAT-M22 ... Data Job Engine; PLAT-M23 to PLAT-M29 ... Notifications; **the next Platform milestone takes PLAT-M30**" (`services-platform:docs/milestones/README.md:5`). - **Verified, stale reference.** `docs/proposals/responsibilities-and-attention/11-integration-architecture.md:101` still says attention's notification adoption is "milestone PLAT-M30 of that package"; the notifications package was renumbered to PLAT-M23..M29 (commit `70d44a446`), so that adoption is now PLAT-M29. An enrichment milestone taking PLAT-M30 should note it. - **Verified.** Other ranges in the same README: `RPT-M1..M4` (`:81-86`), `INTG-M1..M8` (`:92-101`). - **Verified.** No `enrich` milestone or branch exists other than this worktree's `enrichment-architecture`. --- ### Appendix A. Quoted interfaces #### A.1 Execution context (`macroservice-libraries:packages/execution/src/execution-context.ts:95-139`) ```ts export interface ExecutionContext { readonly request: RequestContext; readonly caller: Caller; /** The dispatcher's decision for this request; absent for an execution that had none (a job, a script, a test). */ readonly access?: Evaluation; readonly grants?: GrantSlice; /** When and from where this execution arrived; what conditions are evaluated against. */ readonly requestOrigin: RequestOrigin; readonly tenantId?: string; readonly dataJob?: DataJobAttribution; readonly timeZone: string; readonly workload?: ResolvedWorkloadPolicy; readonly readerRouting?: ReaderRouting; readonly onReader?: boolean; readonly databaseUsage: DatabaseUsage; readonly deadlineAtMs?: number; remainingMs(): number | undefined; readonly logging: RhythmLogger; readonly telemetry: RhythmTelemetry; readonly db: Executor; readonly clock: Clock; readonly now: Moment; moment(value: InstantInput): Moment; readonly isInTransaction: boolean; withTransaction(callback: (transactionContext: ExecutionContext) => Promise): Promise; } ``` Caller kinds (`macroservice-libraries:packages/authorization/src/security-context.ts:27`; `contracts.ts:36`, `:44`, `:52`): ```ts export type UserAccessType = 'console' | 'portal'; export type AccessType = UserAccessType | 'service'; export type CallerAccessType = AccessType | 'system' | 'none'; export type PrincipalKind = 'console' | 'portal' | 'machine'; ``` Session provenance (`macroservice-libraries:packages/db/src/session-context.ts:48-69`): ```ts export const SESSION_CONTEXT_SETTINGS = [ 'app.tenant_id', 'app.user_id', 'app.customer_id', 'app.is_staff', 'app.correlation_id', 'app.source', 'app.operation', 'app.request_id', 'app.job_id', 'app.job_batch_id', 'app.originating_user_id', ] as const; export const CHANGE_SOURCES = ['console', 'portal', 'public', 'integration', 'job', 'maintenance', 'service'] as const; ``` (The settings array is one entry per line in the source; joined here.) Job boundary (`macroservice-libraries:packages/execution/src/execution-context.factory.ts:447-465`): ```ts export interface JobBoundary { readonly operationId: string; readonly jobId?: string; readonly tenantId?: string; readonly requestId?: string; readonly correlationId?: string; readonly handlerName?: string; readonly service?: string; /** Who asked for the job (the submitting request's actor): bound to every line, never the caller the job acts as. */ readonly requestedBy?: string; readonly executionId?: string; readonly db?: Executor; readonly remainingTimeMs?: number; } ``` #### A.2 Audit event shapes `platform.audit_events` (`services-platform:docs/audit-events.md:64-74`, summarised column by column): | Column | Type | Rule | | ----------------- | ----------- | ----------------------------------------------- | | `id` | text | UUIDv7 generated by the database | | `tenant_id` | text | forced policy | | `actor_user_id` | text | `ctx.caller.actorId`, no foreign key | | `event_type` | text | one of `AUDIT_EVENT_TYPES` (nine), by check | | `record_type` | text | object id = dataset id | | `record_id` | text | no foreign key | | `occurred_at` | timestamptz | millisecond precision by check | | `idempotency_key` | text | unique per `(tenant_id, actor_user_id)` | | `metadata` | jsonb | allow-listed keys per event type, ≤ 1,024 bytes | `platform.security_audit_events` (`services-platform:modules/security/db/security-audit-events.table.ts:122-155`, columns as declared): ```ts occurred_at: timestamptz('occurred_at').notNull().defaultNow(), actor_user_id: text('actor_user_id').notNull(), actor_access_type: text('actor_access_type', { enum: ACTOR_ACCESS_TYPES }), actor_identity_id: text('actor_identity_id'), session_ref: text('session_ref'), acting_for_user_id: text('acting_for_user_id'), acting_session_ref: text('acting_session_ref'), customer_id: text('customer_id'), tenant_id: text('tenant_id'), entity_type: text('entity_type', { enum: AUDIT_ENTITY_TYPES }).notNull(), entity_id: text('entity_id').notNull(), entity_version: integer('entity_version'), action: text('action').notNull(), widening: boolean('widening').notNull().default(false), affected_keys: text('affected_keys') /* array */, before: jsonb('before'), after: jsonb('after'), affected_count: integer('affected_count'), request_id: text('request_id'), outcome: text('outcome', { enum: AUDIT_OUTCOMES }).notNull().default('succeeded'), reason: text('reason'), ``` Record revision (`services-platform:modules/custom-objects/db/record-revisions.table.ts:49-50`, `:58-87`): ```ts export const REVISION_OPERATIONS = ['created', 'updated', 'deleted'] as const; export const REVISION_SOURCES = ['console', 'portal', 'public', 'integration', 'job', 'maintenance'] as const; // columns: tenant_id, id ('rev_' || uuidv7), sequence, record_type, record_id, object_id, record_label, operation, // changes (jsonb), changed_fields (text[]), capture_version, transaction_id (xid8), sys_recorded_at, // sys_transaction_started_at, sys_actor_id, sys_actor_label, sys_source, sys_operation_name, sys_correlation_id, // sys_request_id, sys_job_id, sys_job_batch_id ``` #### A.3 Business event envelope (`macroservice-libraries:packages/events/src/envelope.ts:34-47`) ```ts export interface BusinessEventEnvelope

{ readonly rhythm_event: typeof ENVELOPE_VERSION; readonly id: string; readonly key: string; readonly version: number; readonly tenant_id: string; readonly occurred_at: string; readonly subject: { readonly type: string; readonly id: string }; readonly scope: Readonly>; readonly correlation_id: string | null; readonly actor_id: string | null; readonly replay: boolean; readonly payload: P; } ``` Subscriber (`macroservice-libraries:packages/events/README.md:116-127`): ```ts export const grantCategoryOnCredential = defineEventSubscriber({ event: CredentialIssued, versions: [1], intake_function_parameter: '/rhythm/membership/event-intake-function-arn', handle: async (ctx, envelope) => { /* ctx is in envelope.tenant_id; deduplicate on envelope.id or reconcile against current state */ }, }); export const handler = runEventIntake([grantCategoryOnCredential], { service: 'membership' }); ``` #### A.4 Job definition (`services-platform:docs/jobs.md:42-121`, steps in order) 1. Tables: root `jobRootColumns({ jobKind })` + `jobRootConstraints(...)` + domain columns; step and event tables `jobStepColumns(root)` / `jobEventColumns(root)`. 2. The kind in `modules/jobs/db/cross-job-query.view.ts` and a migration of the three cross-job views. 3. Step catalog and workflow: `defineJobWorkflow` (publication task, domain tasks with retry and timeout, failure task, execution timeout); `repeat: { maxIterations }` for backlogs. 4. `registerJobType` with the catalog, `repositoryFor`, the state machine ARN variable, `retryable`, `cancellable`, the submit permission, `constraintNames`, `runLocally`, and `onSettledFailure` (or `NO_JOB_FAILURE_CONSEQUENCE` with a reason). 5. Task operations: `ctx` first, read by `jobId`, idempotent; `JobTaskRefusal` for final failures; `{ step: 'skipped' }`. 6. `createJobTaskHandler({ service, kinds, operations, prepare })`. 7. `createJobStateMachine(scope, { stateMachineName: 'platform-…', workflow, taskFunction, ... })`; submitters declared in `aws/cdk/platform.job-starters.ts`. 8. `createJob(ctx, { kind, submissionId, domainFields })` in the transaction; `startJob` / `startJobOrLeaveToReconciler` after commit. Recurring schedule (`macroservice-libraries:docs/schedules.md:30-34`): ```text EventBridge rate(1 minute) ─▶ dispatcher ─ claim (one transaction) ─▶ schedule_runs (queued) │ ▲ └─ publish, no transaction open ─▶ SQS ─▶ consumer ─ start ─ handler ─ finish ``` #### A.5 Workflow action contract (`macroservice-libraries:packages/workflow-actions/src/contracts.ts:576-616`; `executors.ts:22-45`, `:81-85`) ```ts export type ExecutorReference = { readonly kind: 'service_function'; readonly function_name: string } | { readonly kind: 'in_process' } | { readonly kind: 'host' }; export interface ActionTypeDefinition { readonly key: string; readonly app: string; readonly label: string; readonly category: string; readonly params: TParams; readonly execution: ExecutionKind; // 'inline' | 'durable' | 'host' readonly executor: ExecutorReference; readonly idempotency: IdempotencyMode; // 'action_run' | 'host_lock' | 'last_write_wins' readonly outputs: TOutputs; readonly reversal?: string; readonly hosts: readonly string[]; readonly host?: string; readonly allowed_triggers: readonly string[]; readonly contract_version: number; readonly accepted_contract_versions: readonly number[]; readonly permission?: string; readonly mappable_paths?: readonly string[]; } export interface ActionRunView { readonly id: string; readonly tenant_id: string; readonly host_kind: string; readonly host_instance_id: string; readonly host_definition_key: string | null; readonly trigger_point: string; readonly trigger_occurrence_id: string; readonly batch_id: string; readonly action_key: string; readonly action_type_key: string; readonly contract_version: number; readonly occurrence: number; readonly execution: ExecutionKind; readonly params: Readonly>; readonly occurrence_facts: Readonly>; readonly on_failure: OnFailure; readonly attempt_count: number; readonly provenance: Provenance; } export interface DurableActionExecutor { readonly key: string; readonly execution: 'durable'; execute(ctx: ExecutionContext, run: ActionRunView, facts: ActionFacts): Promise; } ``` (The two inline comments on `execution` and `idempotency` are this report's, from `contracts.ts:30` and `:87`.) Registration call (`services-platform:docs/workflow-actions/README.md:74-81`): ```ts await invokeService({ functionArn: process.env.PLATFORM_WORKFLOW_ACTIONS_SERVICE_FUNCTION_ARN!, method: 'PUT', path: `/workflow_actions_service/registry/${app}`, body: buildManifest({ app, trigger_points, action_types }), }); ``` #### A.6 LLM invoke contract and ledger `services-platform:docs/llm.md:14-23`: ```ts runOperation(ctx: ExecutionContext, operationKey: string, { promptVariables, initialMessages, tools, handlers, finalToolNames, acceptFinal, purposeOf, }): Promise<{ invocationId, outcome, finalToolCall?, usage, credits, attempts }> ``` (One property per line in the source, each with a comment; joined here.) Thrown refusals (`services-platform:docs/llm.md:34-42`): `LlmAllowanceExceededError` 429 `llm_allowance_exceeded`; `LlmProviderTimeoutError` 503; `LlmProviderUnavailableError` 503; `LlmProviderRefusedError` 503; `LlmBudgetExhaustedError` 500; `LlmOperationDisabledError` 409; `LlmInFlightExceededError` 429 with `Retry-After: 2`. Operation declaration (`services-platform:modules/llm/catalog/llm-operations.ts:26-40`): ```ts export interface LlmOperationDefinition { readonly operation_key: LlmOperationKey; readonly application_key: string; readonly feature_key: string; readonly provider: LlmProviderName; readonly model: string; readonly generation: LlmRoutingGeneration & { readonly effort: LlmReasoningEffort }; readonly limits: LlmRoutingLimits; readonly prompt_ref: string; readonly prompt_version: string; readonly preflight_estimate: LlmPreflightEstimate; } ``` The reservation (`services-platform:docs/llm.md:377-388`): ```sql UPDATE platform.llm_allowance_periods SET reserved_credits = reserved_credits + :estimate, in_flight_count = in_flight_count + 1 WHERE tenant_id = :tenant AND period_key = :period AND settled_credits + reserved_credits + :estimate <= LEAST(:platform_ceiling, recurring_base_credits + COALESCE((SELECT SUM(credits) FROM platform.llm_allowance_overrides WHERE tenant_id = :tenant AND kind = 'additional_credits' AND revoked_at IS NULL AND starts_at <= :now AND :now < ends_at), 0)) AND in_flight_count < :in_flight_cap ``` #### A.7 Licensing and entitlement tables `platform.llm_tenant_limits` (`services-platform:modules/llm/db/llm-tenant-limits.table.ts:31-48`): ```ts tenant_id: text('tenant_id').notNull(), llm_enabled: boolean('llm_enabled').notNull().default(true), monthly_allowance_credits: bigint('monthly_allowance_credits', { mode: 'number' }), // null = platform default disabled_operation_keys: text('disabled_operation_keys').array().notNull().default(sql`'{}'::text[]`), in_flight_cap: integer('in_flight_cap'), notes: text('notes'), ...snakeSystemColumns(), ``` `platform.data_export_entitlements` defaults (`services-platform:modules/licensing/db/data-export-entitlements.table.ts:47-55`): ```ts export const DEFAULT_DATA_EXPORT_ENTITLEMENT = Object.freeze({ enabled: true, executions_per_period: 30, period: 'contract_year', minimum_interval_days: 28, max_retention_days: 30, max_concurrent_executions: 1, asset_budget_bytes: 200 * 1024 ** 3, } as const); ``` Planned resource definition (`services-platform:docs/proposals/licensing-entitlements-billing/04-quotas-and-resource-limits.md:11-29`): `key` `..`; `owner`; `name`, `description`; `unit`; `kind` `gauge|counter|high_water`; `scope` `tenant|customer`; `period` `none|calendar_month|contract_month|contract_year|calendar_year|rolling_days:N`; `combine` `additive|max|replace`; `unlimitedAllowed`; `enforcement` `soft|enforced|metered`; `thresholds`; `safetyCeiling`; `reservable`; `commerciallyExposed`; `counting`; `measurement`; `reconciliation`. Attention publish (`services-platform:docs/attention.md:72-76`): ```sql platform.publish_attention_item(p_tenant_id text, p_definition_key text, p_dedupe_key text, p_about_object_id text, p_about_record_id text, p_about_label text, p_subject_kind text, p_subject_id text, p_scope_refs jsonb, p_audiences jsonb, p_facts jsonb, p_due_at timestamptz, p_expires_at timestamptz, p_destination_params jsonb, p_source_service text, p_correlation_id text) RETURNS TABLE (id text, generation integer, created boolean) ``` --- ### Inferred: extension points an enrichment capability would use, and what is missing Everything in this section is **Inferred** from the evidence above unless it says otherwise. #### What it would use as it is | Need | Existing extension point | Evidence | | ------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | -------------- | | Who asked, in which tenant, through which channel | `ExecutionContext`; `app.source`, `app.operation`, `app.job_id` stamped on every write | §1 | | Its own permission keys and a reviewer role | `definePermissions` in its service manifest; built-in policy and role default | §2.1, §2.2 | | Calling CRM to write enriched values | a CRM service function over IAM (Service Calls), or a CRM data operation through the Data Job Engine | §2.5, §7.1 | | A vendor's API key and per-tenant connections | Integration Platform credentials and broker; the `crm.contact.enrich` capability already declared | §2.6, §12 (R1) | | A tenant's enrichment policy | a capability-owned configuration table, "no row = default", forced RLS, configuration string ids | §3 | | A Rhythm-set monthly budget | a copy of the LLM ledger pattern, or the LLM allowance itself for model spend | §4.3, §9.4 | | AI-assisted research | `runOperation` with a new operation key, prompt file and preflight estimate, if it runs inside Platform | §9.1 | | Bulk enrichment of a list | Data Job Engine (a person's authority per batch, receipts, rollback) | §7.1, §7.4 | | Nightly refresh | an EventBridge rule submitting one Durable Job per tenant; a tenant-set cadence through `macroservice-schedules` | §7.1 | | Enrich on create | a business-event subscriber plus queued work | §6, §7 | | "Enrich contact" in automation | a `durable` workflow action type with a service-function executor | §8 | | Telling staff | Attention items for pending reviews; Notifications for finished runs | §10 | | Contact photo from a URL | CRM's save through the asset pipeline (external URL import) | §11 | | Addressing a record generically | `(object_id, record_id)` as audit, attention and revisions do | §12 | | Logs, metrics, alarms | observed boundaries, `recordMetric('enrichment..count')`, `observeFunctions` | §15 | #### What is missing or blocks the obvious design 1. **A provenance model for a value.** No table records "this field's value came from provider X with confidence Y at time T" (§5.4). `app.source` describes the write channel only, and `integration` is reserved with nothing mapping to it (§1.5). The enrichment capability has to own an append-only provenance or suggestion table, keyed to the record by `(object_id, record_id)` and to the revision by correlation or request id. 2. **A review queue for suggested changes.** Platform has no approve/reject/apply record (§10). Attention can point at the work but is not the work. 3. **A published `crm.contact.created` / `crm.organization.created`.** Declared in the catalogue, not published by CRM (§6). Enrich-on-create and an automation trigger both depend on it, and on CRM registering it as an automation trigger point (§8). 4. **Cross-service LLM consumption.** Decided as a library plus a Platform metering service function, not built (§9.3). If research runs outside Platform, this has to be built first (it would be the "first consumer" D-5 waits for), or the research operation has to live in Platform. 5. **A general metering and entitlement runtime.** The licensing v2 resource registry and SDK are decided in direction and not built (§4.2). A "monthly enrichment budget" before that exists is a capability-owned ledger copied from the LLM one, with a later migration into a licensing resource. 6. **A sellable gate.** Keys under `platform.*` or `crm.*` are never licence-gated (`platform` exempt, `crm` always available) (§2.2). If enrichment is sold separately, it needs its own application key in `APPLICATIONS`, its own `.application.access`, and a licence source (HubSpot product or HQ override). 7. **A service principal for a new macroservice.** `SERVICE_PRINCIPAL_IDS` is a fixed constant in the authorization library (§2.5). A new `svc_enrichment` means a library release, a registry seed, and IAM grants on each callee function. 8. **Audit event type for enrichment actions.** `audit_events` has a closed event-type list and a 1 KB short-token metadata; adding `enrichment.*` events is a migration and a deliberate decision (§5.1). Configuration changes fit `security_audit_events` (entity type added by migration, as extensions did). 9. **No vendor-side or web-search tooling in the LLM module.** One provider (Anthropic), one operation, no web-search tool in use (§9.1). "Research" with live web access is new ground for the foundation's boundary guard and rate model. 10. **Retention for any high-volume enrichment log.** `audit_events` has no retention yet and Andrew still owes that decision (§5.1); an enrichment call log is operational history that should follow the monthly-partition convention (§14), as the Integration Platform's execution tables do. 11. **Documentation drift to be aware of** (Verified): `service-calls.md` §7 versus the library on integration principals (§2.5); `llm_tenant_limits` header versus `docs/llm.md` (§3.1); `attention.md` §9 versus business events (§6); the Data Job Engine plan's "on hold" header versus the milestones README (§7.3); `11-integration-architecture.md:101` naming PLAT-M30 for a milestone now numbered PLAT-M29 (§16.2); Platform's `REVISION_SOURCES` lacking `service` (§1.5). ## 21. R4. Data operations (Data Job Engine) and the query layer: evidence for Rhythm 2.0 Data Enrichment ## R4. Data operations (Data Job Engine) and the query layer: evidence for Rhythm 2.0 Data Enrichment Read-only evidence report for the enrichment architecture proposal. Nothing in any repository was modified except this file. No build, test or database was run; everything below is read from source and documents. ### Evidence base and conventions | Short name | What was read | Commit | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------ | | `services-platform` | `origin/master`, read from the worktree `/Users/andrew/rhythm2/worktrees/services-platform/enrichment-architecture`, which sits exactly on that commit (the primary checkout `/Users/andrew/rhythm2/services-platform` was 277 commits behind it and was not used) | `43478ffda` (2026-10-22 10:12 ET, "PLAT-M20: stream F done") | | `macroservice-libraries` | `origin/master` extracted with `git archive` (the local checkout was at 0.10.2 and stale); `packages/data-jobs` is 0.12.1, which is the version Platform pins (`services-platform:package.json:108`) | `49728e6ba` | | `services-crm` | `origin/master` extracted with `git archive`; not in the brief, read only where Question 2 and Question 11 cannot be answered from Platform (contacts are CRM's) | `e818ee48` | `react-console` was not read or touched; statements about console behaviour come from Platform and CRM documents and from the PLAT-M19..M22 milestone records. Citation form: `repo:path:line`. Library paths are `macroservice-libraries:packages//src/:line`. Labels used on every claim: **Verified** (read in code or a document that describes built behaviour), **Planned** (a proposal, plan or milestone not yet built), **Decided (Andrew, date)** (a recorded ruling), **Inferred** (my reading, not stated in a source), **Absent** (searched for and not found). A proposal that Andrew has not approved is labelled Planned even when the text says "Recommended". ### Summary (12 bullets) 1. **The Data Job Engine (DJE) is a shared library plus per-service tables, not a service.** The engine is `@rhythmsoftware/macroservice-data-jobs` (0.12.1 in Platform); each macroservice keeps eight tables of its own built from the library's column contracts and declares one `defineDataOperation` per object type and kind (`insert`, `upsert`, `update`, `delete`). Verified: `macroservice-libraries:packages/data-jobs/README.md:3-21`, `:460-480`; `services-platform:modules/data-jobs/db/data-jobs.table.ts:1-49`. 2. **Platform's own operations are three: custom field definitions, custom object records, associations.** Contacts, organizations, activities and five configuration catalogues are CRM's. Platform has no data-job queue or worker; a Platform job runs inside the submitting request, capped at 500 items. Verified: `services-platform:modules/data-jobs/services/operations/data-operations.ts:32-36`, `services-platform:docs/data-jobs/platform-data-operations.md:195-262`. 3. **The model is validate, then apply, with a receipt per item.** Items are frozen at submission, batched (25 by default), and each item writes a receipt in the same transaction as the change; a refused item becomes a rejection (code, field paths, safe message, never values). 13 job states, 7 item outcomes. Verified: `macroservice-libraries:packages/data-jobs/src/data-job-states.ts:12-26`, `:79-89`; `data-job-columns.ts:224-273`. 4. **Rollback exists only where the owning service declares it, and enrichment-shaped writes are the hard case.** A rollback is another data job planned from the receipts and the record revisions; CRM declares it for contact and organization updates only; custom field values and collections (e-mail, phone, address) are `not_reversible_operation`; Platform's operations declare none, so there is no `/rollback`. Verified: `macroservice-libraries:packages/data-jobs/README.md:188-232`; `services-crm:docs/debt/rollback-of-collections.md:3-8`; `services-platform:docs/data-jobs/platform-data-operations.md:150-157`. 5. **Upsert matches by `id` or by the producer's `sourceKey` within a `source.system`, never by e-mail or name.** CRM contacts declare `matchingKeys: ['id', 'sourceKey']`; the match is the crosswalk `data_id_mappings (tenant, source_system, object_type, source_key)`. A natural-key match and an `ambiguous_match` code are named in the contract but implemented nowhere. Imports never merge; duplicate detection is a separate CRM service. Verified: `services-crm:modules/data-jobs/services/operations/contact-import.ts:91`; `macroservice-libraries:packages/data-jobs/src/data-id-mapping-columns.ts:33-53`. Absent: any e-mail matching in a job. 6. **Provenance today is per record, not per field.** A job writes revisions with `sys_source = 'job'`, the submitter as actor, and `sys_job_id`/`sys_job_batch_id` (CRM); the crosswalk row names the source system, key, first and last job; contacts also keep `sys_external_id`, `origin_service` and `origin_service_record_id`. No value table carries a source, confidence, retrieval time or licence. Verified: `services-crm:docs/data-jobs.md:489-503`, `:884-890`; `macroservice-libraries:packages/custom-fields/src/custom-field-values.ts:98-113`. 7. **Uploads are generic and large; Data Runs are not available for ongoing work.** Multipart direct to S3 (1 byte to 5 TiB, five content types, 16 MiB minimum part, 30/14/7-day prefix retention); a job reads a pinned object version. A Data Run is registered only for `initial_import` into an `implementation` tenant in `loading`, so it cannot carry recurring enrichment. Verified: `services-platform:docs/data-jobs/uploads.md:74-95`, `:311-334`; `services-platform:modules/data-jobs/db/data-runs.table.ts:17-19`, `:62`. 8. **A run's status model is reusable as is.** Counts (applied, skipped, rejected, blocked, conflict, not attempted), per-item results, rejections, pause, resume, cancel, retry of blocked rows, revalidation of conflicts, correction jobs (`correctsJobId`) and rollback are engine behaviours, not CRM behaviours. Not built: schedules, an export lane, frozen server-side selections, a served capability registry, an events table for interactive jobs. Verified/Absent: `services-platform:docs/proposals/data-engine-architecture/03-existing-architecture-assessment.md:165-170`. 9. **A new table becomes queryable through two registrations plus a Platform key list.** The owner's metadata registration and query registration run at every deploy; Platform also carries a build-time list of dataset query keys (`modules/queries/datasets.ts`) that routes must name in `evaluates`. Both manifest schemas are `.strict()`, so a library bump with a new key must reach Platform before the owner's deploy. A dataset's query permission is not enough on its own where its records have a read permission (Decided (Andrew, 2026-10-21, CRM-ACC-0064): `.query` AND `.read`). Verified: `services-platform:docs/queries/dataset-registry-audit.md:3-4`, `:90`; `services-platform:docs/queries/semantic-query-layer.md:602-611`. 10. **Custom fields are opt-in per entity and only Andrew decides which; their values may be written only by the record's owner.** Contacts and organizations are approved; values live in `_custom_field_values` (ten typed columns, one row per record and field). "Every macroservice reads any table; only its owner writes it." An integration contract already exists, `crm.contact.enrich@1`, whose description says CRM writes the answer through its own persistence with `app.source = 'integration'`. Verified: `services-platform:docs/standards/custom-field-capability.md:19-24`, `:32-34`; `services-platform:AGENTS.md:309-322`; `macroservice-libraries:packages/integration-contracts/schemas/capabilities/crm/crm.contact.enrich@1.json:5`. 11. **Quick Search is a browser type-ahead over Algolia, not an identity-matching API.** Closed list of two record types (contacts, organizations), a tenant-chosen field list in three ranked tiers (e-mail is tier 2 by default), and an "Include inactive records" checkbox that is the browser's half of a filter. For a server-side match by e-mail the supported path is the query layer. Verified: `services-platform:docs/quick-search.md:3-6`, `:77-98`, `:187-193`; `services-platform:modules/quick-search/registry/record-types.ts:142-165`. 12. **Any table keyed by contact id must join three standing mechanisms, and nothing today classifies licensed data.** Merge (a declared policy per column, or the repository's coverage test fails), the tenant lifecycle (purge, reset, export declaration) and the query description stamp. The tenant data export has seven classes and per-dataset `withheldFields` with reasons, but no class for third-party licensed data; query exports and Explorer carry no classification at all. Verified/Absent: `macroservice-libraries:packages/record-merge/src/merge-coverage.ts:6-9`; `services-platform:modules/data-exports/export-coverage/contract.ts:28-43`. --- ### 1. The Data Job Engine (PLAT-M19..M22) #### 1.1 What it is and where it stands - **Verified.** "Platform's half of the Data Job Engine": how a file gets in, how a package is loaded into a tenant, what a tenant reset removes, and which records Platform loads itself; the approved design is `docs/proposals/rhythm-data-job-engine-plan.md`, and "the engine itself is `@rhythmsoftware/macroservice-data-jobs`" (`services-platform:docs/data-jobs/README.md:3-5`). - **Decided (Andrew, 2026-10-12)** plan approved, build held (D1-D6) and **Decided (Andrew, 2026-10-21 ~14:00 ET)** "Complete the Data Job Engine and Bound the Scope" instruction approving the PLAT-M19..M22 plan (`services-platform:docs/proposals/rhythm-data-job-engine-plan.md:1428`; `services-platform:docs/milestones/PLAT-M19-data-job-engine-foundations.md:11`). - **Milestone state at `43478ffda`** (Verified, the records are "the only source of this milestone's status"): | Milestone | Status | Cite | | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ | | PLAT-M19 Engine foundations verified | Changes requested (Engineering Verified 2026-10-21 20:22 ET, run djem19r3, L1 24/24; Andrew 2026-10-22 09:19 ET: back end stands, three console changes through v0) | `PLAT-M19-data-job-engine-foundations.md:7`, `:13`, `:51-57` | | PLAT-M20 Small bulk operations from lists | In Progress (CRM side landed on services-crm master `e818ee48`; console landing branch waits) | `PLAT-M20-small-bulk-operations.md:7`, `:49` | | PLAT-M21 Full job lifecycle in the console | Proposed | `PLAT-M21-data-job-lifecycle-console.md:7` | | PLAT-M22 Representative migration proof | Proposed | `PLAT-M22-migration-proof-and-close-out.md:7` | - **Planned, not approved for build.** The "next-generation" proposal `docs/proposals/data-engine-architecture/` (set-based L3 load lane, capability registry, export lane, schedules, frozen selections) "is not approved for build; only the decisions Andrew gave on it on 2026-10-19 (Q-01..Q-07) bind, and none of them changes what this plan builds" (`services-platform:docs/proposals/rhythm-data-job-engine/milestone-plan.md:9-11`). The completion plan lists as stopped (Andrew, 2026-10-21): "no set-based L3 loading, no Migration Workbench ... no schedules, no export lane, no capability registry, no frozen server-side selections" (`milestone-plan.md:70-73`). #### 1.2 Where a job and a run live (tables) Verified. Eight engine tables per service, all tenant-owned with forced row-level security (`services-platform:docs/data-jobs/platform-data-operations.md:274-278`): `data_jobs`, `data_job_items`, `data_job_batches`, `data_job_receipts`, `data_job_rejections`, `data_job_submissions`, the batch queue (`queued_data_job_batches` in Platform and CRM, `data_job_batch_queue` in the library), and the crosswalk `data_id_mappings` (`services-platform:modules/data-jobs/db/data-jobs.table.ts:2-8`, `:32-49`; `macroservice-libraries:packages/data-jobs/README.md:466-475`). | Table | Holds | Notable rules (all Verified) | Cite | | ----------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | | `data_jobs` | One row per submitted job: object type, operation, contract version, mode, purpose, selection, patch, source, conflict policy, `corrects_job_id`, `rolls_back_job_id`, `data_run_id`, state, lane, request hash, impact, impact digest, validated versions, suppressed effects, 8 counters, pause code, failure code and summary, bulk cursor, timestamps | `sys_` prefix rule (Decided, Andrew 2026-10-11/12); `sys_version` is the concurrency token; `sys_created_by_id` is the submitter every batch re-resolves | `data-job-columns.ts:77-142` | | `data_job_items` | Frozen input of an interactive job, one row per position | unique `(tenant, job, record_id)` and `(tenant, job, source_key)`: no duplicate target in one job | `data-job-columns.ts:145-166`, `:336-356` | | `data_job_batches` | Range of items, state, attempt, transaction count, applied/rejected/skipped, failure code, `chunk_ref` for bulk | one row per position in the job | `data-job-columns.ts:168-199`, `:358-381` | | `data_job_receipts` | Proof the change committed, written with the change: outcome, detail (created/updated/unchanged/deleted), `sys_record_version`, suppressed effects | PK `(tenant, job, item)`; values never stored | `data-job-columns.ts:224-244`, `:406-424` | | `data_job_rejections` | Why an item was not applied: outcome, code, field paths, message, constraint, dependency item, source key, group key, `source_ref` | "never record values"; PK includes attempt so a retry's rejection never overwrites the first | `data-job-columns.ts:246-273`, `:426-443` | | `data_job_submissions` | Idempotency ledger `(tenant, submitter, key)` | 30-day retention; `purgeDataJobSubmissions` | `data-job-columns.ts:275-286`; `README.md:526-536` | | `data_id_mappings` | Crosswalk `(tenant, source_system, object_type, source_key)` to `record_id`, `last_source_revision`, `sys_last_applied_version`, first and last job | outlives jobs; no FK to jobs; only data-job table the app role may DELETE from | `data-id-mapping-columns.ts:33-53`; `data-job-tables.ts` | | `platform.data_job_uploads` | One upload session (Section 3) | | `data-job-uploads.table.ts:112-173` | | `platform.data_runs`, `platform.data_run_steps` | A registered migration package and one row per dataset | Section 1.9 and 3 | `data-runs.table.ts:81-146`; `data-run-steps.table.ts:72-159` | - **Verified, retention.** Receipts and rejections "expire with [the job] (90 days)" (`data-id-mapping-columns.ts:6-7`); Decided (Andrew, 2026-10-12) D6 "90 days after the job ends; also the rollback window" (`rhythm-data-job-engine-plan.md:1426`; `PLAT-M19-...md:30`). - **Absent.** The enforcement of that retention is not built: "No receipt or rejection retention sweep; 90-day retention described only" (Inferred in the architecture assessment, `03-existing-architecture-assessment.md:173`). The plan's `data_job_events` table for interactive jobs (`rhythm-data-job-engine-plan.md:1274-1275`) is not among the engine's eight tables (`README.md:466-475`). - **Verified.** Partitioning of receipts and rejections is deliberately deferred (`README.md:512-524`). #### 1.3 Job types Verified, `DataOperationKind = update | upsert | insert | delete` (`data-job-states.ts:59-61`); mode `validate | apply`; purpose `operational | initial_import` (`data-job-states.ts:49-56`). What exists per service: | Service | Object type / kinds | Cite | | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Platform | `platform.custom_fields` `insert`, `upsert`; `platform.custom_object_records` `upsert`; `platform.associations` `upsert` ("That is the whole list, deliberately") | `data-operations.ts:6-12`, `:32-36` | | CRM | `crm.contacts` `update`, `insert`, `upsert`, `delete`; `crm.organizations` `update`, `insert`, `upsert`, `delete`; `crm.activities` `insert`, `upsert`, `update`, `delete`; five configuration catalogues (`insert`/`upsert`/`update`, status patch, `delete`); `crm.regions`, `crm.activity_types` | `services-crm:docs/data-jobs.md:215-225`, `:745` (contact delete); files `contact-delete.ts`, `organization-delete.ts` in `services-crm:modules/data-jobs/services/operations/` | Not data jobs, but adjacent (Verified): the tenant **reset** is a durable job `tenant_reset` on the state machine `platform-tenant-reset` (`services-platform:docs/data-jobs/tenant-reset.md:58-60`); the CRM bulk lane's orchestration is a durable job `bulk_data_job` (`services-crm:docs/data-jobs.md:617-622`); a **Data Run** is a parent over several data jobs (Section 1.9). There is no "tenant reset" operation type in the engine's `DataOperationKind`. #### 1.4 How a job declares its operations Verified. One `defineDataOperation` per object type and kind (`define-data-operation.ts:365-432`); the declaration is checked at load, "a service that will not start" rather than a job that rejects every item (`README.md:68-69`). Its members, quoted in Appendix A: `objectType`, `operation`, `contractVersions`, `permissions {submit, perRecord}`, `patchableFields`, `matchingKeys`, `dependsOn`, optional `groupBy`, `itemDependencies`, `importBehaviour.suppress`, `collectionMapping`, `prevalidate`, `prepare`, `apply`, `create`, `update`, `bulkApply`, `rollback`, `classifyError` (`define-data-operation.ts:266-343`). Rules enforced at declaration (`:365-432`): `update` and `delete` must list `id` as a matching key, `insert` and `upsert` must list `sourceKey`, a delete declares no patchable fields and no rollback, and `create`/`update` paths exist only for the kinds that have them. "The engine never reads domain tables": it hands the domain receipts and takes back items; the domain never sees queues, batches or receipts (`define-data-operation.ts:7-9`; `README.md:206`). #### 1.5 How a run is started and executed **Submission (Verified).** A request names `objectType`, `operation`, `contractVersion`, `mode`, optional `purpose`, a `selection {ids}` with a common `patch`, or `records[]` of prepared per-record values, an optional `source`, `conflictPolicy` and `correctsJobId` (`data-job-request.ts:36-69`, Appendix B). Refusals made before anything is written: unknown contract version, a patch touching a non-patchable field, more items than the interactive cap, a repeated id or source key, an initial import for a tenant that may not take one, and a delete submitted as `apply` (`validate-data-job-request.ts:7-19`). Idempotency: `Idempotency-Key` header, unique per `(tenant, submitter, key)`, with a canonical request hash (`data-job-request.ts:6-12`; `rhythm-data-job-engine-plan.md:271-273`). **Interactive lane (Verified, with the plan's SQS picture Planned in detail).** At most 500 items (`DATA_JOB_LIMITS.maxItems: 500`, batch size 25, validation expires after 24 hours: `data-job-config.ts:18-23`). One transaction writes the job, items, batches and queue rows; after the commit a dispatcher claims queue rows and publishes one SQS message per batch; one worker invocation runs one batch (`README.md:7-11`, `:80-96`; `rhythm-data-job-engine-plan.md:544-556`, `:558-586`). Each batch: read the job (paused or cancelled does no work), re-resolve the submitter's authority, `prevalidate` the batch, `prepare` with no transaction open, open one transaction, per item insert the receipt key, call the domain's own `apply`, complete the receipt; one failing object is isolated by halving the transaction (`README.md:86-96`; `define-data-operation.ts:319-326`). **Bulk lane (Verified for CRM).** The same URL serves both: "a body that names `source` names a file" (`services-crm:docs/data-jobs.md:585-591`). At submission the job freezes each object's bucket, key, version id, size and hash. Four task operations drive it from a Step Functions state machine: `chunkSource` (chunk files of at most 100 items or 2 MB), `planBatches` (one batch per chunk), `admitBatches` (hands batches to the bulk queue up to a budget) and `settleIfDone` (`README.md:352-364`; `bulk-data-job-config.ts:24-28`). No `data_job_items` rows exist for a bulk job (`services-crm:docs/data-jobs.md:615-616`). Admission ceilings: per job 4, per tenant 4, global 8 by default (`bulk-data-job-config.ts:59`); the cluster budget is `BULK_DATA_JOB_CLUSTER_BATCHES = { dev: 4, prod: 12 }` with worker shares CRM 2 and 6, Accounting 2 and 6 (`bulk-batch-budget.ts:24-36`; production values are "Proposed"). A tenant's ceiling never exceeds its service's share less one (`README.md:408-411`). Fair order across tenants (`planAdmission`): a job waiting more than ten minutes first, then round-robin across tenants (`README.md:391-399`). A malformed-line ratio over 10% after 100 lines fails the job `malformed_source` (`bulk-data-job-config.ts:42`; `README.md:371-374`). A job stuck behind a wedged ceiling for 15 minutes fails `admission_stalled` (`bulk-data-job-config.ts:120`). Sized for "about ten million records" (`bulk-data-job-config.ts:66-72`). **Platform runs its jobs in the submitting request (Verified).** "Platform has no data-job queue and no worker". `submit_custom_field_data_job` writes the job, commits, then calls `dispatchDataJobBatches` with a publisher that keeps the messages and runs each claimed batch with `runDataJobBatch` in process, with `redelivers: false`; a transient failure that exhausts its attempts pauses the job with code `failed_transient` (`platform-data-operations.md:195-223`). A dataset by upload is read at submission as JSON Lines and "the interactive cap of 500 items ... is the documented ceiling" (`platform-data-operations.md:252-262`). Adopting a queue later "would change ... Nothing else: no table, no route, no response and no operation" (`platform-data-operations.md:247-250`). **Concurrency and conflict (Verified).** Validate captures each record's version (`sys_validated_versions`); apply verifies it and an item that moved is a `conflict` (`changed_since_validation`); a conflict is "revalidated, never retried" (`README.md:123-157`; Decided (Andrew, 2026-10-19) D-37). Under the default policy `reject`, a keyed write compares with the version the crosswalk last wrote, so a person's later edit stands; `overwrite` is the audited exception (`README.md:268-291`; `rhythm-data-job-engine-plan.md:337-346`; `services-crm:docs/data-jobs.md:292-302`). #### 1.6 Progress, per-row results, errors - **Counters on the job row** (Verified): items, applied, skipped, rejected, blocked, conflict, not attempted, plus `batches {planned, finished}` so a client can say "batch 7 of 20" (`data-job-reads.ts:20-41`; `README.md:497-510`). Each counter "is a cache of the evidence beside it" (`README.md:501-503`). - **States** (Verified): `accepted, validating, validated, queued, running, paused, cancelling, cancelled, completing, succeeded, completed_with_rejections, failed, expired`; final states are never left except `completed_with_rejections -> queued` for a retry of blocked rows (`data-job-states.ts:12-26`, `:133-157`). - **Final state** (Verified): `succeeded` when every item applied or was skipped, `completed_with_rejections` when at least one was rejected, blocked or in conflict, `failed` on a systemic stop, `cancelled` when a person cancelled (`data-job-states.ts:184-201`). - **Per-item outcomes** (Verified): `applied` (detail `created`, `updated`, `unchanged`, `deleted`), `rejected`, `blocked`, `conflict`, `skipped_duplicate`, `skipped_stale`, `not_attempted` (`data-job-states.ts:79-93`). - **Reads** (Verified): status (one row), `GET .../results` (every item in position order with its outcome), `GET .../rejections`, keyset paged on `item_seq`, 100 per page (`data-job-reads.ts:123-155`; `README.md:98-99`). The results route is the interactive lane's; a bulk job has no item rows (`rhythm-data-job-engine-plan.md:1000`; `services-crm:docs/data-jobs.md:615-616`). PLAT-M19 R3 records that Results and Rejections "must work at five million rows" and that "the server-side results API is PLAT-M21 scope" (`PLAT-M19-...md:57`), so results at scale is Planned. - **Errors** (Verified): `classifyError` answers either one item's outcome (`invalid_value`, `not_found`, `unique_violation`, `conflict`, a domain code) or `systemic`, which stops the job; a transient systemic error is redelivered and then pauses the job (`README.md:49-54`; `define-data-operation.ts:179-198`). Rejection messages never carry values and a database error is reported as class and SQLSTATE only (`platform-data-operations.md:92-94`). A rejection-rate guard pauses a job at 20% after 1,000 items or 50% in the first 100 (`data-job-config.ts:54-59`), pause code `rejection_rate` (`data-job-columns.ts:124-125`). - **Groups** (Verified): items sharing a `groupKey` succeed or fail together (`README.md:101-121`). - **Dependencies** (Verified): an item whose parent source key is not in the crosswalk is `blocked` `dependency_missing`; `retry` takes up only blocked rows once the dependency has loaded (`README.md:298-303`, `:245-248`). #### 1.7 Rollback: what is reversible and how Verified mechanism. A rollback "is another data job, planned by the domain and run by the same engine" (`README.md:188-192`). The domain declares `rollback { compensatingOperation, contractVersion, plan }`; `plan` receives the receipts of the original job (record id, outcome, detail, version written) and returns ordinary items of its own `update` (before-values plus `expectedVersion`) or `delete` (for records the job created) and a `notReversible` list with codes `changed_since`, `deleted_since`, `has_dependants`, `not_reversible_operation` (`README.md:193-207`; `define-data-operation.ts:207-264`). `planDataJobRollback` creates the rollback job already validated, with an impact `{ reversible, notReversible, byCode }`; the irreversible receipts become that job's rejections (`README.md:209-215`). Refusals: `hard_delete_not_reversible`, `rollback_not_declared`, `rollback_job_not_final`, `rollback_window_passed`, `already_rolled_back` (`README.md:219-229`). A rollback never overwrites a change made after the job (`README.md:239-244`); "Retry" of a rolled-back job is refused `retry_after_rollback` (`README.md:245-248`). What is and is not reversible today: | Operation | Reversible? | Evidence | | ---------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `crm.contacts` `update` (a patch) | Yes, from the revisions the job left (`crm.record_revisions`, found by `sys_job_id`), field by field, only where no later revision changed the field and the contact still holds the job's value | `services-crm:docs/data-jobs.md:867-872`; `rhythm-data-job-engine-plan.md:934-947` | | A contact job that set a **custom field** or changed a collection (e-mail, phone, address, organization history) | **No**: receipt is `not_reversible_operation` naming the collection; "custom field values would have to be converted back out of the value column" | `services-crm:docs/data-jobs.md:880-882`; `services-crm:docs/debt/rollback-of-collections.md:3-8`, `:16-22` | | `crm.contacts` `insert`/`upsert` | **No `rollback` declared** (the declaration has no such member); the plan says a created record would be deleted "only if no later revision exists" | `contact-import.ts:83-163` (Absent); plan `:938-940` (Planned). Only `contact-update.ts:103` and `organization-import.ts:140` (its `update` kind only) declare one (grep of `rollback:` over CRM's operations) | | `crm.*` configuration imports, activities | No: "a configuration import declares no compensating operation ... nor does an activity import or delete" | `services-crm:docs/data-jobs.md:105-107` | | Hard delete | Never; protection is before: validation impact, digest echo, typed confirmation (`DELETE 37 contacts`) | `README.md:159-186`; Decided (Andrew, 2026-10-19) CRM-Q10 `PLAT-M19-...md:35` | | Platform operations (`platform.custom_fields`, `platform.custom_object_records`, `platform.associations`) | No: "The operation declares no compensating operation, deliberately"; the way back is a tenant **reset** | `platform-data-operations.md:150-157` | | Initial import into a `loading` tenant | By tenant reset and reload (reset empties domain data, keeps configuration) | `tenant-reset.md:54-98`; plan `:959-967` | Window: "rollback is offered while the original job's receipts exist (90 days, D6)" (`rhythm-data-job-engine-plan.md:946-947`). #### 1.8 Audit linkage - **Verified (CRM).** Every revision a job writes carries `sys_source = 'job'`, the submitter as actor, `sys_job_id` and `sys_job_batch_id` (migration 0057; the index `record_revisions_job_idx (tenant_id, sys_job_id, record_id)`), set from the session keys `app.job_id` and `app.job_batch_id` that "only a data job's batch sets" (`services-crm:docs/data-jobs.md:884-890`). Decided (Andrew, 2026-10-12) D4 (`rhythm-data-job-engine-plan.md:1424`). - **Verified.** The submitter's identity is held with no foreign key so a deleted user still resolves (`rhythm-data-job-engine-plan.md:1244-1248`); each batch re-resolves the submitter's grant slice and stops the job `access_revoked` when the grant has gone (`rhythm-data-job-engine-plan.md:1232-1241`; `README.md:549-551`). - **Verified.** Platform's operations write through the owning module's persistence ("never a statement of this module's"), so "a loaded record is numbered, named, held unique, given its files and audited as a person's write is" (`platform-data-operations.md:76-81`). Record history exists in Platform only for custom object records and associations (`services-platform:docs/record-history.md:3-9`). - **Absent / dated.** The 2026-10-18 assessment states "Change capture exists only in CRM; Platform and Accounting write no change events or revisions" (`03-existing-architecture-assessment.md:177`, a proposal document, later partly overtaken by `docs/record-history.md`). - **Verified, attempted work.** Rejections, conflicts and cancelled items are "not a change" and live in the job's own tables, never in revisions (`rhythm-data-job-engine-plan.md:1273-1274`). - **Planned.** `app.source = 'integration'` is "the unused slot in `CHANGE_SOURCES`" for Integration Platform writes (`services-platform:docs/proposals/integration-platform/17-synchronization.md:91`); the Integration Platform sync engine (INTG-M5) is `Proposed`, "Plan: not yet written" (`docs/milestones/INTG-M5-data-synchronization.md:7-10`). #### 1.9 The "platform data operations" contract other services implement **What a service must publish for Platform (and a Data Run) to load into it generically (Verified):** 1. **A `defineDataOperation` per object type and kind**, with the object type `.` (regex in `define-data-operation.ts:358`) registered in the service's operation registry (Platform's is `data-operations.ts`; CRM's is `services-crm:modules/data-jobs/services/operations/data-operations.ts`), "a new object type or a new operation is an entry here and a file beside the others, and nothing else changes - not the routes, not the capability" (`data-operations.ts:2-4`; `services-crm:docs/data-jobs.md:231-232`). 2. **The same set of job routes under the service's own record path**, "the same ten routes CRM publishes ... A producer that can load a CRM dataset loads a Platform one without learning a second shape" (`platform-data-operations.md:110-112`); CRM says "eleven, declared once and parameterised by the object type" (`services-crm:docs/data-jobs.md:100-102`). The Platform surface table: submit and list `POST/GET .../data_jobs`, `GET .../{jobId}`, `.../results`, `.../rejections`, `POST .../records_by_source_key`, `.../apply`, `.../pause`, `.../resume`, `.../cancel`, `.../retry` (`custom-field-data-jobs.routes.ts:146-346`). Each surface "serves its own object types and no other" (`platform-data-operations.md:125-127`). 3. **Permissions**: a submit set and one per-record key, both re-checked per batch as the submitter. Platform's custom field routes require `platform.custom-fields.bulk` and one of `.create`/`.update` (`anyOf`) (`platform-data-operations.md:117-121`). The convention is `.bulk` plus the operation's own key, default tier `admin` (`rhythm-data-job-engine-plan.md:1229-1231`). 4. **Job visibility**: holders of the surface's every-job key across the tenant see every job; others see only theirs; another submitter's job answers 404, never 403 (`platform-data-operations.md:129-135`). 5. **A reconciliation read**: `POST .../records_by_source_key` (at most 100 keys), served while the tenant is `loading`, read-only (`platform-data-operations.md:159-193`; `services-crm:docs/data-jobs.md:419-457`). 6. **Reset participation** for a service that holds a tenant's records: the lifecycle manifest declares `"operations": [..., "reset"]`; without it Platform "refuses the reset ... before anything is emptied" (`tenant-reset.md:103-132`). 7. **For a Data Run only**: an entry in `DATA_RUN_SERVICES` (service name plus the first label of its API host) and the service's `_API_BASE_URL` (`data-run-services.ts:37-48`; `platform-data-operations.md:291-298`). `route_of_object_type` derives service and path from the object type's prefix alone (`data-job-port.ts:218`; `data-runs.md:167-171`). **How a Data Run starts and advances (Verified).** "Platform orders the work; the owning macroservice does it" (`data-runs.md:15`). Registration reads the manifest from a completed upload, verifies each dataset's size, SHA-256, upload ownership, `destination.tenantId` and environment, and orders datasets into waves with Kahn's algorithm (`data-runs.md:77-117`). `POST .../start` polls every step with a job, stops at the first failed step, submits every ready step, and settles `succeeded` when every step is `succeeded` or `blocked`; "Nothing advances a run in the background ... a run that advanced under nobody's authority is precisely what this engine must not have", so a run moves only when someone holding a token starts, resumes or reads it (`data-runs.md:35-39`, `:128-143`). A step is submitted at most twice (`MAX_STEP_ATTEMPTS = 2`, `data-run-steps.table.ts:65`; `data-runs.md:145-148`). Each call has a 10-second timeout (`DATA_JOB_CALL_TIMEOUT_MS`, `data-job-port.ts:122`); a step unreadable for 30 minutes fails (`DATA_JOB_UNREADABLE_GRACE_MS`, `data-job-port.ts:147`). "Nothing is rolled back" by a run (`data-runs.md:150-151`). The 2026-10-18 assessment calls "A Data Run advances only while someone polls it" a gap (`03-existing-architecture-assessment.md:175`; Inferred consequence there). --- ### 2. Upsert matching, duplicates, external identifiers, provenance #### 2.1 How an import matches an existing record - **Declared by the owning service (Verified).** `matchingKeys` is a member of the operation declaration: "how an item may name a record: `id`, a producer's `sourceKey`, a natural key the domain declares" (`README.md:35`; `define-data-operation.ts:40-41`). For contacts the declaration is `matchingKeys: ['id', 'sourceKey']` (`services-crm:modules/data-jobs/services/operations/contact-import.ts:91`; the update operation the same, `services-crm:modules/data-jobs/services/operations/contact-update.ts:95`). Platform's custom object records are keyed `:`, with the bare key as `sys_external_id` (`platform-data-operations.md:13-15`). - **The match itself (Verified).** For an `insert`, `upsert` or keyed `update` the engine, inside the item's transaction, reads the crosswalk row `FOR UPDATE` and decides: no row, create; a row, update that record; a lower `sourceRevision` is `skipped_stale`, equal `skipped_duplicate`, higher or absent applies (`README.md:268-279`; `keyed-item-decision.ts`). "The row is written in the same transaction as the record's own change ... a rejected item writes no row at all, so the corrected resubmission of that key is a create and not a suppressed duplicate" (`README.md:264-266`). - **An `insert` of a known key is refused by name**; an `upsert` of a known key replaces the record whole, "exactly as that record's own `PUT` replaces it" (`services-crm:modules/data-jobs/services/operations/contact-import.ts:204-206`; `services-crm:docs/data-jobs.md:272-275`). - **Natural-key matching: Planned in the plan, Absent in the code.** The plan says "optionally an alternate natural key such as a membership number" and that a match finding more than one record "is `ambiguous` (rejected, never resolved silently)" (`rhythm-data-job-engine-plan.md:238-241`, failure table `:483`). The string `ambiguous_match` appears in the engine only in a comment on `DataItemRejection.code` (`define-data-operation.ts:96`) and nowhere in CRM's data-jobs module or Platform's code (search of the three trees); no operation matches by e-mail or name. Absent. - **A patch can be keyed too (Verified, Decided (Andrew, 2026-10-19) D-38).** An `update` item naming a `sourceKey` goes through the crosswalk; a key the crosswalk has no row for is `blocked`, "a patch never creates and never writes a row for a key that names nothing" (`README.md:281-287`). - **Merged-away ids (Planned/Verified).** A merged record is hard-deleted and aliased in `crm.merged_record_aliases`; an item keyed by an old id resolves through `resolveMergedId` or is rejected `record_merged`, "never recreated" (`rhythm-data-job-engine-plan.md:247-250`; `macroservice-libraries:packages/record-merge/src/merged-ids.ts:1-23`). #### 2.2 Duplicate handling - **Within a delivery (Verified).** `skipped_duplicate` means the producer's key was already loaded and the delivery would do nothing new; a repeated id or source key inside one job is refused whole (`services-crm:docs/data-jobs.md:199-205`; `validate-data-job-request.ts:14-15`). - **Look-alike people (Verified).** "Imports never merge: look-alike rows are inserted and duplicate detection raises candidates" (`rhythm-data-job-engine-plan.md:250`). CRM's duplicate detection computes match keys per contact: `email` ("strongest common evidence"), `phone`, `name_sound`, `name_sound_postal`, `name_dob`, `origin_id` (`origin_service` + `origin_service_record_id`, "same source record imported twice") and organization keys; a key shared by more than 200 records is skipped (`services-crm:docs/merging/duplicate-detection.md:39-57`). Detection "never merges anything; it proposes pairs" (`duplicate-detection.md:3-4`). The detection design is marked "draft for review (2026-10-11)" in its header, though its tables are referenced as built elsewhere. - **A contact's primary e-mail is unique per tenant among live contacts (Verified).** A plain index plus triggers that raise the old unique index's violation; "an inactive or deceased contact does not hold its address" (Decided (Andrew, 2026-10-19) M1-D1) (`services-crm:docs/debt/duplicate-email-prevention.md:7-12`). #### 2.3 "External identifiers" and "source record ids" | Concept | Where | Shape | Cite | | -------------------------------------------- | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Crosswalk key | every service's `data_id_mappings` | `(tenant_id, source_system, object_type, source_key)` to `record_id`; `last_source_revision`, `sys_last_applied_version`, `sys_first_job_id`, `sys_last_job_id`, `sys_last_applied_at` | `data-id-mapping-columns.ts:33-53` | | `sys_external_id` | a system column of every record | `text`, "the identity a record had in the system it came from"; the caller writes it (`sysExternalId` defaults to editable), unique per object for custom object records | `macroservice-libraries:packages/records/src/system-columns.ts:43`; `services-platform:docs/metadata.md:254-258`; `services-platform:docs/custom-objects.md:211` | | CRM import identity | `contacts`, `organizations` | an import stores the item's own `sys_external_id` or else its source key (`withProducerIdentity`); contact number stays the tenant's own | `services-crm:modules/data-jobs/services/imports/import-items.ts:45-61`; `services-crm:docs/data-jobs.md:489-497` | | `origin_service`, `origin_service_record_id` | `crm.contacts` columns | "External references (no foreign key here)"; feed the duplicate key `origin_id` | `services-crm:modules/entities/db/contacts.table.ts:29`, `:131-132`; `duplicate-detection.md:48` | | Social profile columns | `crm.contacts` | `facebook`, `twitter`, `instagram`, `linked_in`, `date_profile_last_updated`, `date_social_profile_last_updated` | `contacts.table.ts:125-130` | | Integration identity map | Integration Platform sync | `integration_external_ids (profile, object_id, record_id) <-> external_key` plus versions and digests | **Planned**: `17-synchronization.md:93-99`; no such table in Platform code (search of `modules`, `lib`, `db` finds none) | #### 2.4 Does a run record provenance on each written record? - **Per record, Verified.** (a) The revision: `sys_source = 'job'`, submitter, `sys_job_id`, `sys_job_batch_id` (CRM; Platform for custom records); (b) the crosswalk row: `source_system`, `source_key`, first and last job and the version written; (c) the receipt: record id, outcome, detail, version, suppressed effects; (d) the job row: `source {system, deliveryId, objects[]}` (`data-job-request.ts:19-29`; `data-job-columns.ts:91-92`); (e) `sys_external_id` on contacts and organizations. "No history of the other system ... The producer's own audit trail stays with the producer" (`services-crm:docs/data-jobs.md:501-503`). - **Per field, Absent.** The custom field value row holds only `sys_last_modified_at` and `sys_last_modified_by_id` beside its value (`custom-field-values.ts:111-112`); the contact's scalar columns hold nothing about which job set them except through the revision. - **Suppression is recorded.** Live side effects an initial import suppressed are written on the job and on every receipt (`data-job-columns.ts:113-114`, `:238-239`). --- ### 3. Uploads and data runs #### 3.1 Upload lifecycle (Verified, `uploads.md`) 1. `POST /data_jobs/{tenantId}/uploads` opens a session (`fileName`, `contentType`, exact `size`, optional `sha256`); the server builds the key `inbound/{tenantId}/{uploadId}/{fileName}` and starts an S3 multipart upload (`uploads.md:56-79`, `:322`). 2. `POST .../uploads/{uploadId}/parts` signs up to 100 part URLs, each good for one hour and one part (`uploads.md:127-155`). 3. The client PUTs parts straight to S3; "No byte of the file passes through Rhythm" (`uploads.md:13-22`, `:157-176`). 4. `POST .../complete` names every part and ETag; Rhythm completes the multipart upload, `HEAD`s the object and records size, ETag and **version id**; the job reads that version, "so overwriting the key afterwards changes nothing the job sees" (`uploads.md:178-195`). A wrong size or digest aborts the session. 5. `GET .../uploads/{id}` reads the session and the parts S3 holds (resume); `POST .../abort`; `GET .../uploads?state=` lists (`uploads.md:220-268`). 6. States `open -> completed | aborted | expired`, never back (`uploads.md:270-283`; `data-job-uploads.table.ts:58-62`). Permission: one key `platform.data-jobs.upload`, administrator tier, covering every operation including reads (`uploads.md:43-49`). #### 3.2 Limits, layout, retention | Item | Value | Cite | | ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | | Size | exactly declared; 1 byte to **5 TiB** | `uploads.md:78`; `data-job-uploads.table.ts:86` | | Content types | `application/x-ndjson`, `application/json`, `text/csv`, `application/gzip`, `application/zip` | `uploads.md:81-89`; `data-job-uploads.table.ts:76` | | Part size | `roundUpToWholeMiB(max(16 MiB, ceil(size/9000)))`; at most 9,000 parts | `uploads.md:99-101`; `data-job-uploads.table.ts:89-95`, `:192-195` | | Digest | verified only up to **256 MiB** (30-second function); larger files record `declared_sha256` with `sys_sha256` null; an open decision | `uploads.md:300-309` | | Bucket | `platform-data-jobs-`, private, versioned, encrypted, HTTP refused | `uploads.md:313`; `data-jobs.settings.ts:11` | | Prefixes and retention | `inbound/` 30 days (re-tag `retain=true` to keep), `chunks/` 14 days, `exports/` 7 days; incomplete multipart abandoned after 2 days; classification "personal" | `uploads.md:315-334`; `data-jobs.settings.ts:33-35`, `:68-77` | | Archives | `application/zip` is accepted but "Nothing unpacks one yet"; rules for a later step are written (entry cap, uncompressed size, ratio, no path escape, no nesting) | `uploads.md:343-365`; `data-job-uploads.table.ts:73-74` | | Which formats a bulk job reads | CRM: JSON Lines, CSV, gzip of either; "a single JSON document, a zip archive or a gzipped file whose name says neither is refused at submission" | `services-crm:docs/data-jobs.md:602-605` | #### 3.3 Data runs (Verified) A Data Run is registered from a manifest plus one upload per dataset (at most 100 datasets), ordered by `dependsOn`, and run as one bulk data job per dataset (`data-runs.md:3-8`, `:60-117`). It is staff-only (`hq.data-runs.manage`, `system` access) and not gated on the tenant's availability (`data-runs.md:41-49`). **It is not usable for recurring work:** `DATA_RUN_PURPOSES = ['initial_import']`; registering one is refused unless the tenant's status is `implementation` and lifecycle `loading` (`TenantNotInImplementation`, `TenantNotLoading`) (`data-runs.md:51-58`; `data-runs.table.ts:17-19`, `:61-63`: "An `operational` package into a live tenant is a later decision and is deliberately absent rather than half-built"). Decided (Andrew, 2026-10-11) "initial data imports go only into implementation tenants" (`tenant-reset.md:21`). #### 3.4 How a file of results from a provider could be ingested (what exists) - **Verified path.** Upload the file (ndjson/csv, optionally gzip) -> submit a job naming `source: { system, uploadId }` to the service that owns the target object type -> bulk lane reads the pinned version a line at a time; a line that cannot be read is a rejection naming its line number, never a job failure (`README.md:366-375`). The first part is generic; the second needs an operation for the target object type. - **Platform constraint, Verified.** A Platform operation reads the whole file at submission and refuses a dataset over 500 lines "with the number it carried" (`platform-data-operations.md:254-262`). A provider result file larger than 500 lines needs a service that has a bulk lane (CRM does; Platform does not). - **No provider-specific operation exists.** The only data operations over contact data are CRM's `crm.contacts` kinds above; there is no enrichment object type, no suggestion object type and no "apply suggestion" operation. Absent. --- ### 4. What a run keeps that an "enrichment run" could reuse as is All Verified in the engine (`macroservice-libraries:packages/data-jobs`): | Capability | Where | Notes | | ------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | | Counts per outcome and per batch; per-item results; per-item rejections with code, field paths, safe message | `data-job-reads.ts:20-41`; `data-job-columns.ts:246-273` | rejections never hold values | | Validate, impact, expiry, digest | `README.md:159-186`; `data-job-config.ts:14`; `data-job-request.ts:107-110` | impact digest is mandatory on every apply of a validated Platform job (`platform-data-operations.md:225-228`) | | Idempotent submission and replay | `data-job-request.ts:6-12`; `README.md:526-536` | 30-day ledger | | Pause, resume, cancel at a batch boundary | `rhythm-data-job-engine-plan.md:348-352`; `data-job-states.ts:133-157`; `data-runs.md:252-262` (run level) | "Nothing committed is undone" by a cancel | | Retry of blocked rows only; revalidation of conflicts; correction jobs (`correctsJobId`) | `README.md:149-157`, `:245-248`; `data-job-request.ts:65-66` | revalidate refused for bulk jobs and rollbacks (`revalidation_not_interactive`, `rollback_revalidation_refused`) | | Transient-failure handling: batch attempts, `failed_transient`, pause code | `platform-data-operations.md:218-223`; `data-job-columns.ts:124-129` | | | Rejection-rate guard | `data-job-config.ts:54-59` | | | Crosswalk and `records_by_source_key` | Section 2; `platform-data-operations.md:159-193` | | | Admission and fairness (bulk only) | Section 1.5 | | | Actions function for consoles | `data-job-actions.ts` (`retry`, `rollback`, `revalidateConflicts` with `requires` keys) | the console renders controls from it (`data-job-actions.ts:1-8`) | | Data Jobs list, detail, results pages | console, built; PLAT-M19 R3 asks for server-side results at scale | `milestone-plan.md:36`; `PLAT-M19-...md:57` | What does **not** carry over: `data_runs`/`data_run_steps` (initial import only, one step per migration dataset, polled by a caller) and the Data Run readiness verdict (a judgement about activating a tenant: `data-runs.md:284-299`). --- ### 5. Queries and the semantic layer #### 5.1 How a service registers datasets and fields - **Fields are opt-out (Decided (Andrew, 2026-10-08)).** "if I add a field, it should be queryable... I'm going to forget to do that"; "Every column of a registered object's table is a field unless its owner excludes it in `defineObject` with a reason" (`semantic-query-layer.md:52-55`; `docs/metadata.md:239-244`). `exclude` carries a reason that "review reads", the contract's own example reasons being "personal data waiting for field-level security, internal bookkeeping, a reference to an object not registered yet" (`macroservice-libraries:packages/metadata/src/object-registration.ts:434-439`). A column no data type describes fails the owner's build (`docs/metadata.md:243-244`). - **Two registrations at every deploy (Verified).** The owner builds a metadata manifest (`defineObject`; CRM `aws/crm.metadata.ts`) and a query manifest (`defineQueryObject`; `aws/.queries.ts`); the owner's deploy workflow invokes `platform_metadata_registration` then the query registration "right after its stacks deploy ... so a new column never depends on someone remembering a step" (`docs/metadata.md:279-290`; `semantic-query-layer.md:451-468`; Decided (Andrew, 2026-10-08)). Platform validates and replaces the application's registration in one transaction; an object, field or collection missing from the new manifest is **retired**, never deleted, so tenants' custom fields and overrides survive (`docs/metadata.md:286-290`; `semantic-query-layer.md:892-898`). - **A Platform-side key list (Verified).** `modules/queries/datasets.ts` `QUERY_DATASETS` names each dataset's query permission and scope "because the authorizer resolves a caller's grant slice only for keys a route names in `evaluates`" (`docs/queries/dataset-registry-audit.md:3-5`); "Only `datasets.ts` carries a build-time key list; the routes' `evaluates` and the unit pin derive from it" (`dataset-registry-audit.md:90`). So a new service dataset needs an entry in Platform and a Platform release, in addition to the owner's registration. Example entries: `datasets.ts:71-75`, `:79-86`. - **Library bumps (Verified).** `queryObjectSchema` and `queryRegistrationManifestSchema` are `.strict()` (`macroservice-libraries:packages/metadata/src/query-registration.ts:823`, `:944`), so a manifest member an older Platform does not know is refused. The metadata changelog shows additive optional members arriving as minor versions: 0.23.0 plain property keys, rules, `discriminator`; 0.24.0 capabilities; **0.25.0 `discovery.archivedField`**; 0.26.0 `recipientRelationships`, `businessEvents`, `cancellation`; 0.27.0 `roleOptions` (`packages/metadata/CHANGELOG.md:3-35`). `docs/metadata.md:271-277` records the earlier failure ("a Platform still on the older contract accepts the deploy - the failure metadata 0.4.0 caused on 2026-10-10") and the rule "a manifest states one of these only where it differs from the default a reader applies". Platform pins metadata 0.27.0 (`package.json:117`). Inferred operational consequence: bump and deploy Platform first (Andrew's session memory of 2026-10-20, "CRM declared discovery.archivedField before Platform took metadata 0.25.0; CRM deploy refused at registration", is not a repository document). - **Stamps (Verified).** Every table a description reads carries four statement triggers `platform.attach_query_description_stamp(table, tenant_column)` that bump `platform.query_description_stamps` in the changing transaction; "no macroservice creates a stamps table, sequence or trigger function of its own"; a guard spec fails any table a cold build reads without the triggers (`semantic-query-layer.md:1516-1534`, `:1592-1603`; Decided (Andrew, 2026-10-15) one stamps table). - **Reader role grants (Planned in 10.1, Decided (Andrew, 2026-10-08) in decision 6/9).** The reader role's grants "follow the registered columns" (`semantic-query-layer.md:54-55`) and queries run as `rhythm_query_reader` under row-level security with `app.tenant_id` set (`:67-69`); the detail of `queries grants` generation is in a section marked [Proposed] (`:2569-2585`). #### 5.2 Relationships and joins - **To-one** relationships come from `reference` fields; a query traverses `contactId.`; to-many are opt-in by the target's owner (`reverse(...)` in the registration); tenant equality is added to every join (`semantic-query-layer.md:867-879`). Decision 13 restores a one-hop `.` read as a `LEFT JOIN` (`:83-91`). - **Joins in a request** (`model` with a base dataset and joined datasets, inner or left; "a one-to-many join repeats rows as the console warns") (`:59-66`). - **Collections and `exists`** (Verified, Built 2026-10-13): a to-many child declared on the dataset (`table`, `via`, `fields`, one nested level) is tested by `exists` criteria "without the query ever joining them, so a record is answered once however many of its rows match" (`:1071-1100`). Limit: 20 collections per object (`query-registration.ts:819`). - **Picks** (proven-unique to-one joins on key plus `where`), `computedFields` over a `source` view, `configuredRelationships` generated from a tenant's configuration rows and custom object datasets with generated families from association types (`semantic-query-layer.md:910-1070`, `:2999-3024`; `query-registration.ts:808-814`). - **Query description stamps and `source`**: a dataset may be read from a security-invoker view over the table that adds computed columns (`query-registration.ts:788-796`). #### 5.3 How a new table from another module becomes queryable Verified steps (no step is optional): (1) the table has `tenant_id`, the forced tenant policy and the owner's text-normalization trigger; (2) `defineObject` for the table (fields opt-out, `exclude` with reason, `recordIdField`, labels); (3) `defineQueryObject` with `permission` `..`, `defaultFields`, `defaultSort`, `hiddenFields`, `discovery`, `collections`, `picks`, `quickSearch` as needed (`query-registration.ts:723-822`); (4) the stamp triggers on every table a description will read; (5) an entry in Platform's `QUERY_DATASETS` and the routes' `evaluates` (Section 5.1); (6) a deploy that runs the metadata and then the query registration. A relationship from `crm.contacts` **to** the new table is either a to-one reference field on the new table plus a `reverse` declared by the contacts owner, a `collections` entry on contacts, or a pick; a reference field on the new table to `crm.contacts` gives joins "from" the new dataset only (`semantic-query-layer.md:867-875`). #### 5.4 Registered but excluded from defaults - **`defaultFields`** (max 255) and **`defaultSort`** decide what a new query starts with; **`hiddenFields`** (max 1,000): "`hidden: true` (query registration, UI hint): not offered in the picker by default but usable by anyone authorised" (`query-registration.ts:760-770`; `semantic-query-layer.md:595`; flagged at `resolve-query-model.ts:839-840`). - **`discovery`** (one field's discoverable option ids and an optional `archivedField`) is a report, not an injected filter: "nothing injects them into a query that runs" (`semantic-query-layer.md:1477-1495`). - **Excluded from being a field at all**: `exclude` in `defineObject` (above). Verified: there are **no field-level permissions** in Rhythm 2.0 today: Decided (Andrew, 2026-10-08) decision 11 "there are no field-level rules (sections 6.10 and 10.3's field rules are withdrawn)" (`semantic-query-layer.md:74-76`) and `rhythm-data-job-engine-plan.md:1230-1231`. `restrict` and `maskUnless` appear in the design table as "pending the security design" (`semantic-query-layer.md:886-888`). #### 5.5 Performance of adding related tables to contact queries - **Verified.** "Only attachments some referenced field needs are emitted. No to-many join ever reaches the outer `FROM`" (`semantic-query-layer.md:1892-1894`); a to-many relation is an `EXISTS` only; a custom field is a `LEFT JOIN` on the full primary key; limits: 2 to-one hops, 40 joins "beyond that custom fields switch to scalar subqueries", page 100 default and 1,000 max, 15 s statement timeout (`:2652-2662`). An unused related dataset therefore adds no per-query join cost. - **What it does cost (Verified).** The description. "Contacts describes eight objects and reads the registry, the picks, the relationships and the tenant's configuration rows (measured in dev on 2026-10-14: about 500 ms of a 640 ms query)", so descriptions are cached keyed by stamps, with an in-process layer and a shared Valkey store (`:1508-1514`, `:1535-1551`). A new related object or relationship on contacts enlarges every contact description build (Inferred), and the cache entry is rebuilt on any stamp move for the tenant. - **Index rules under row-level security (Verified).** Only leakproof conditions (`=`, ranges, `= ANY`) use an index; `lower()`, `ILIKE`, `numeric` comparisons and list operators are filters; keyset paging seeks on the first sort key (`:2669-2680`). A new table's filters and sorts need plain-column indexes leading with `tenant_id` (`:2681-2685`). The custom field value table's text index "holds the field, never the value" (`custom-field-values.ts:131-138`). - **Scale evidence (Verified).** CRM's duplicate-detection backfill reached a 797,186-row `contact_match_keys` table (`services-crm:docs/merging/duplicate-detection.md:306`); the query layer measured keyset pages at 0.3-0.6 ms on a 1M-contact tenant with an index on the first sort key (`semantic-query-layer.md:2679-2680`). #### 5.6 ".query AND records .read" (Decided (Andrew, 2026-10-21, CRM-ACC-0064, option (a) "Need both")) "Searching, exploring, reporting on or joining into `crm.organizations` needs `crm.organizations.query` AND `crm.organizations.read`; a role that denies the read loses every query surface over those records, while the query key stays a separate grant" (`semantic-query-layer.md:602-605`). Implemented as `also_requires` per dataset in `QUERY_DATASETS` (contacts family `crm.contacts.read`; organizations, tasks, activities, memberships, submissions), evaluated in one place (`mayQueryDataset`) so describe, run, a relation into the dataset, the catalog's `authorized` flag, a custom table's required permissions and a report's visibility all pick it up; "No route names an `also_requires` key: the service resolves them with the dataset keys" (`semantic-query-layer.md:606-611`; `datasets.ts:71-75`, `:236-253`). Inferred for enrichment: a new dataset over contact-keyed rows needs its own query key and, if its rows are read one at a time anywhere, its own read key, plus the contact's `crm.contacts.read` where the dataset joins into contacts (a relation into a dataset the caller cannot query "is not described and is refused as an unknown field", `semantic-query-layer.md:89-90`). --- ### 6. The metadata module: custom fields and the capability registry #### 6.1 Tables (Verified, `docs/metadata.md:82-104`) `platform.registered_objects`, `registered_fields`, `registered_collections` (written by the owner's registration only); `custom_fields` (tenant, through the API, PK `(tenant_id, object_id, key)`), `standard_field_label_overrides`, `custom_field_rules` (one row per kind). Definition columns are typed (`fieldDefinitionColumns`); "there is no JSON" (`:101-103`). DDL in Appendix E. #### 6.2 Custom field definitions - **Key**: lowercase words joined by `_` or `-` ending `__c` (`member_since__c`); a custom object's property has a plain key (Decided (Andrew, 2026-10-08) and D-03 2026-10-18); immutable; never reused (`docs/metadata.md:108-125`). **At most 200 fields per object and tenant**, archived counted (D-16) (`:141-142`; `custom-fields.table.ts:63`). Archive, never delete (`:137-140`). Data type never changes; configuration may widen, not narrow (`:126-130`). - **Types (Verified)**: `text`, `large_text`, `html`, `email`, `phone`, `url`, `integer`, `decimal`, `currency`, `percent`, `boolean`, `file`, `image`, `list` (single or multiple, 1 to 500 options each with a stable `id`, `label`, `archived`), `date`, `datetime`, `reference` (`targetObjectId`) (`docs/metadata.md:36-54`). Required ladder `api`, `console`, `portal`, `none` (Decided (Andrew, 2026-10-09)) (`:131-136`). Field rules `requiredWhen`, `visibleWhen`, `unique`, `default`, `exposure` (`staff`, `owner`, `portal`, `public`) (`:144-161`). #### 6.3 Values: storage shape (Verified) Owner's table `_custom_field_values`, one row per record and field: PK `(tenant_id, record_id, custom_field_id)`, composite FK to the record cascading, `num_nonnulls(...) = 1` over ten value columns (`string_value`, `integer_value`, `numeric_value`, `boolean_value`, `file_url_value`, `list_value`, `date_value`, `date_time_value`, `reference_value`, `html_value`), plus `sys_last_modified_at` and `sys_last_modified_by_id`; "one partial index per filterable typed column"; no foreign key to the definition ("an archived field keeps its values") (`macroservice-libraries:packages/custom-fields/src/custom-field-values.ts:1-24`, `:88-156`; `docs/metadata.md:411-441`). On the wire a record carries one `customFields` map; one JSON form per data type; `resolveCustomFieldWrites` refuses unknown or archived keys at `customFields.` (`docs/metadata.md:415-434`). Decided (Andrew, 2026-10-08) the value column names (`semantic-query-layer.md:33-35`). CRM's tables: `crm.contact_custom_field_values` and `crm.organization_custom_field_values` (`services-crm:modules/entities/db/contact-custom-field-values.table.ts:1-20`). #### 6.4 The capability registry - **Opt-in per entity, only Andrew decides (Decided (Andrew, 2026-10-19), R2-ARCH-001, approved about 18:24 ET).** "Only Andrew decides that an entity gets custom fields. An agent that believes one should raises it as a product decision ... and builds nothing until he decides" (`standards/custom-field-capability.md:19-24`). The approved list: CRM contacts and organizations; Committees committees and nominations; Membership memberships, organizational units and applications; Events registrations and Awards applications "when registered"; custom objects by construction; everything else none (`:32-43`, `:53-56`). - **Mechanism (Verified).** `defineObject(..., { capabilities: { customFields, configurableForms, configurablePageLayouts } })`, each only `true`; stored as `supports_custom_fields`, `supports_configurable_forms`, `supports_configurable_page_layouts` on `platform.registered_objects`; the approved list is code (`APPROVED_CUSTOM_FIELD_OBJECTS`), registration refuses `customFields` on any other object, `POST /custom_fields` refuses an object without the flag, and `GET /metadata/{tenantId}/objects?capability=custom_fields` lists them (`docs/metadata.md:181-212`; `registered-objects.table.ts:42-44`; `standards/custom-field-capability.md:98-123`). An enriched entity table is therefore not a custom-field host unless Andrew approves it (Inferred application). - **What enabling requires (Verified)**: definitions in Platform, the owner's value table written in the record's transaction, API `custom_fields`, query registration naming the value table, history through the shared mechanism, console forms and layouts, acceptance scenarios (`standards/custom-field-capability.md:130-141`). #### 6.5 How an external capability could write a custom field value through a contract rather than directly - **Rule (Decided (Andrew, 2026-10-08)).** "Every macroservice reads any table; only its owner writes it." Writes into another service's tables go only through the owner; sharing a write means a database capability role (`services-platform:AGENTS.md:309-322`; `docs/proposals/platform-schema-access.md:1-3`). - **The write path of a value is the owner's.** Values are "written only through the contact (`services/persistence/contacts.ts`), which resolves every key against the tenant's contact definitions in the same transaction" (`services-crm:modules/entities/db/contact-custom-field-values.table.ts:7-9`). The contract surfaces are (a) the contact PATCH, whose bulk form is `crm.contacts` `update`, patchable `custom_fields.*` (`services-crm:modules/data-jobs/services/operations/contact-update.schema.ts:66-70`), (b) the contact import (`insert`/`upsert`, whole record), (c) the owner's HTTP API (`customFields` map). - **An enrichment-shaped contract already exists, Verified as a contract and unbuilt as a flow.** `crm.contact.enrich@1`, "Fields a provider knows about an address (legislative districts, representatives, geocodes), for CRM to write through its own persistence with app.source = 'integration' when a rule sees an address change"; request `{ address, fields_requested[] }`, response `{ matched, fields }` (`macroservice-libraries:packages/integration-contracts/schemas/capabilities/crm/crm.contact.enrich@1.json:3-9`, `:38-64`, `:65-109`; Appendix L). The Integration Platform that calls it is partly built (INTG-M1 Product Accepted, INTG-M2 In Progress, INTG-M5 sync Proposed; `docs/milestones/README.md:94-101`). Nothing in CRM calls this capability: `services-crm:docs/debt/image-files-and-enrichment.md:8` says "Still not here: enrichment and region assignment. Each is an integration of its own. The fields are stored; nothing fills them." Planned. - **Write standards a contract writer inherits (Decided (Andrew, 2026-10-19), D-05, D-37, D-38).** Every field-changing write carries the client-held version (400 missing, 409 stale), a no-op changes nothing, data jobs capture versions at validation and verify at application, patch imports respect source revision ordering (`standards/record-write-contracts.md:13-21`, `:46-50`, `:114-118`, `:163-201`). --- ### 7. Quick Search: identity matching and "include inactive" - **What it is (Verified).** "the console's type-ahead lookup ... the browser asks Platform once, about every fifteen minutes, for a short-lived search credential; after that every keystroke goes straight to Algolia, and Platform is never on the keystroke path" (`docs/quick-search.md:3-6`). One shared Algolia index for every tenant and both record types; isolation is a filter baked into the secured key (`quick-search.md:79-81`, `:178-181`). As of that guide production has no Algolia application (`:20-24`) and it lists "No console UI" (`:25-26`); the latter is stale, since `docs/proposals/status-visibility.md:220` records the console commit for the "Include inactive records" checkbox (Inferred). - **What it indexes (Verified).** A closed registry of record types defined in code, today `contacts` (dataset `crm.contacts`) and `organizations`; "There is no table, no administrative path and no way for a tenant to invent a record type" (`registry/record-types.ts:2-9`, `:142-170`). The tenant chooses a field list placed in three ranked tiers `qs_t1`-`qs_t3`; defaults for contacts: `name`, `number` tier 1, `primary_email`, `primary_organization.organization_id.name` tier 2, `preferred_address.city` and `.state` tier 3 (`record-types.ts:158-165`; `quick-search.md:83-87`). A phone is also indexed as digits (`quick-search.md:112-117`). `never_indexed` fields carry a reason (contacts: `notes` "free text", `date_of_birth` "personal data", `gender` "personal data") (`record-types.ts:166-170`). Fields come from the owner's `quickSearch` declaration read through the query layer (`quick-search.md:8-10`; `query-registration.ts:821`). - **How a record is found by e-mail or name (Verified, with the limit stated).** Type in the search box: Algolia matches word prefixes over the tiered attributes in the browser; a hit shows `qs_display` (title, subtitles, status, number, image). There is no server route that answers "which contact has this e-mail". Absent. For a server-side match the supported primitives are the query layer's `projectRecords` and `findRecordIds` (a system entry point over at most 1,000 ids, `semantic-query-layer.md:92-97`; used by the Quick Search indexer and named by the Integration Platform sync design, `17-synchronization.md:73`) or a `POST /queries/{tenantId}/datasets/crm.contacts/query` with a criterion on `primary_email` (Inferred: field id from `record-types.ts:152`). CRM's primary e-mail is unique among live contacts (`services-crm:docs/debt/duplicate-email-prevention.md:7-12`), which makes exact e-mail a safe match key for live contacts (Inferred). - **"Include inactive" (Verified, Decided (Andrew, 2026-10-11)).** Every live document carries `qs_discoverable` (from the dataset's discovery rule, evaluated against the raw option id) and `qs_archived`; the credential filters only tenant, `qs_state:live` and granted types, and "hiding non-discoverable records is the browser's half, deliberately": a console not including inactive records adds `AND NOT qs_discoverable:false`; the "Include inactive records" checkbox can widen status and never tenant or record type (`quick-search.md:89-98`, `:187-193`, `:797-808`; `docs/proposals/status-visibility.md:23`, `:127`, `:160`: pickers get no such checkbox, "Quick Search's alone"). An archived record is never discoverable (`quick-search.md:799-801`). - **Permission gate (Decided (Andrew, 2026-10-09)).** Searching requires `platform.quick-search.read` plus `crm.contacts.read` for both contacts and organizations (organizations deliberately gated by contact read) (`quick-search.md:169-174`; `record-types.ts:15-19`). - **Live sync (Verified).** CRM's change events feed the indexer (`platform_quick_search_indexer`, at most 100 events a call, `quick-search.md:67`), three repair passes reconcile; `rebuild_after_load` for initial imports is Planned, not implemented ("Quick Search indexing does not skip loading tenants", `03-existing-architecture-assessment.md:164`; `quick-search.md:198-257`). --- ### 8. Custom objects and associations: first-class table or association? #### 8.1 What they are (Verified) - A custom object is a record type a tenant defines at run time, with properties (custom fields on `objects.`), rules, layouts and forms, stored in **one shared typed store** `platform.custom_object_records` plus `platform.custom_object_record_values` (the shared value-table shape plus a `scaled_value bigint` shadow for decimals), `custom_object_unique_values`, and `platform.managed_assets` (`docs/custom-objects.md:3-8`, `:204-227`). Limits: **50 objects per tenant** (D-16), **200 properties per object** (`custom-objects.md:48-50`; `docs/metadata.md:141-142`). Object ids are `objects.` (Decided (Andrew, 2026-10-18) D-05); every active custom object is automatically a query dataset with no deploy and no registration (`semantic-query-layer.md:114-122`; Decided (Andrew, 2026-10-18)). - **Decisions D-01, D-03, D-04 (Decided (Andrew, 2026-10-18))**: first-class records in two Platform modules; plain property keys; "A general associations module; CRM affiliations unchanged" (`docs/proposals/custom-objects/README.md:85`; `09-decisions.md:55-67`; approval recorded at `docs/custom-objects.md:10-12` and `docs/associations.md:9-12`). Options on D-04 were (a) general `association_types`/`associations`, (b) generalise affiliations, (c) reference fields with junction objects; (a) recommended and approved (`09-decisions.md:62-67`). - **Associations model tenant-defined relationships between two registered records**: `association_types` (tenant configuration: `from_object_id`, `to_object_id`, labels both ways, cardinality, `dated`, `allow_primary`, delete rules, `is_active`) and `associations` (UUIDv7 `id`, the four endpoint columns, `is_primary`, `start_date`, `end_date`, `position`, `notes`); record ids "are verified when the row is written, never by a foreign key, because the records live in other schemas" (`docs/associations.md:3-5`, `:20-60`). **"Associations do not replace CRM affiliations"** (`associations.md:15-18`). - **Bulk load**: `platform.custom_object_records` and `platform.associations` are Platform data operations keyed `:` (Section 1.3); a reset empties records and associations and keeps object definitions and association types (`custom-objects.md:374-453`; `tenant-reset.md:80-98`). - **Merge**: a reference property pointing at a merged contact is repointed to the survivor by Platform's participant, history attributed to the merge (`record-merge-participant.md:95-108`). #### 8.2 Assessment for "enrichment suggestion" and "external identity" (Inferred) | Question | Custom object / association | First-class table | Basis | | ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Who designs the schema | the tenant administrator, per tenant, at run time | the product, once, versioned | `custom-objects.md:3-8`; a 50-object cap and 200-property cap are per tenant (`:48-50`) | | Shape fit: "provider P says contact C has field F = V with confidence X, retrieved at T, licence L, status pending/accepted/rejected" | would be a custom object with a reference property to `crm.contacts` and typed properties; no per-property provenance; a decimal `confidence` works (scaled shadow) | one row per proposed value with typed columns and constraints | value tables hold exactly one value per record and field (`custom-field-values.ts:119-122`) | | Shape fit: external identity link "provider key K identifies contact C" | an association type needs two Rhythm records as endpoints (`associations.md:27`); an external provider key is not a Rhythm record, so it would be a custom object holding `(provider, external_key)` plus a reference to the contact | a table keyed `(tenant, provider, external_key)` with a unique constraint and merge policy | endpoints are "any two registered object ids" (`associations.md:27`); the engine's own crosswalk is already keyed `(source_system, object_type, source_key)` (`data-id-mapping-columns.ts:72`) | | Queryable, exportable, reset | automatic dataset; export through query export; reset empties records and keeps definitions, so the **definition must be recreated per tenant** for any new tenant | needs the Section 5.3 registration steps once | `semantic-query-layer.md:114-122`; `custom-objects.md:444-453` | | Licence, retention, purge of withdrawn provider data | no concept; generic | can carry columns and a lifecycle participant | Section 12 | | Merge | a reference property to a contact is reassigned by the existing participant | requires its own `MergeReference` declaration | `record-merge-participant.md:95-112`; `merge-coverage.ts:6-9` | | Permissions | `platform.custom-records.*` at resource scope `custom_object:` | its own keys | `custom-objects.md:150-166`, `semantic-query-layer.md:117-120` | Reading of the evidence (Inferred, not a decision): a **product capability that ships to every tenant** (provider identity links, suggestions with provenance and a lifecycle) is a first-class table, because the custom object model is a tenant-authored schema with tenant-sized caps and per-tenant definitions. The **tenant-specific** places enrichment may land (the tenant's own "Congressional District" custom field) are ordinary custom fields on contacts, already supported. An association is the right model only for a dated relationship between two Rhythm records (for example "Contact is represented by Legislator" if legislators were records), not for a link to an external key. --- ### 9. The record merge participant protocol Verified (`docs/record-merge-participant.md`; `macroservice-libraries:packages/record-merge`). CRM merges duplicate contacts and organizations: "one record to keep, the others are merged away and hard-deleted, and every reference to a deleted record must be dealt with in the same transaction" (`record-merge-participant.md:3-5`). The merge objects are only `crm.contacts` and `crm.organizations` (`policies.ts:22`, `:32`). **What a module implements (Verified):** 1. Two SQL functions per object type in its own schema, called by CRM's merge worker **inside CRM's one transaction** over CRM's connection as `rhythm_crm_app`: `.merge_plan_(survivor_id text, merged_ids text[]) -> jsonb` (read-only) and `.merge_apply_(merge_id text, survivor_id text, merged_ids text[], decisions jsonb) -> jsonb`; no HTTP, no queue (`record-merge-participant.md:9-24`; `macroservice-libraries:packages/record-merge/src/participant.ts:1-22`, `:159`). Names come from `participantFunctionNames(schema, objectType)`. 2. `SECURITY DEFINER`, owned by the migration owner, pinned `search_path`, `EXECUTE` revoked from `PUBLIC` and granted only to `rhythm_crm_app`; assert `app.tenant_id` and an actor; row-level security stays forced (`record-merge-participant.md:114-129`). 3. Answer the strict v1 documents: `ParticipantPlan { contractVersion, actions[], decisions[], blockers[], warnings[] }` and `ParticipantApplyResult { contractVersion, actions[] }`, parsed with `parseParticipantPlan` / `parseParticipantApplyResult` (unknown keys refused) (`participant.ts:43-136`, Appendix J). The sentences in a plan are shown to staff **verbatim** (`record-merge-participant.md:37-43`). 4. Declare **every column that holds a contact or organization id** in the repository's registry with a policy: `owned`, `reassign`, `reassign-dedupe { businessKey, onCollision }`, `consolidate { handler }`, `preserve-history`, `clear`, `decide`, `block`, `not-a-reference { reason }` (`policies.ts:63-72`). `REASSIGNING_POLICY_KINDS` are `reassign`, `reassign-dedupe`, `consolidate`: after the apply no such column may still hold a merged id (`policies.ts:108`). Collision rules: `keep-survivor`, `keep-earliest`, `keep-latest`, `decide`, `{ combine }` (`policies.ts:44`). 5. **Coverage is enforced mechanically.** `merge-coverage.ts` reads the live catalog and fails the repository's integration tier on an unclassified column that is a foreign key to `crm.contacts`/`crm.organizations`, named like one, or a polymorphic `record_id`/`reference_value` pair, on stale entries, and on unclassified `jsonb` columns on a table that holds a reference (`merge-coverage.ts:6-24`); Platform's registry is `lib/record-merge/platform-references.ts` held by `tests/integration/db/merge-coverage.integration.spec.ts` (`record-merge-participant.md:32-35`). 6. Ids of merged-away records resolve through `crm.merged_record_aliases (tenant_id, object_type, merged_id, survivor_id, merge_id, merged_number, merged_origin_service, merged_origin_service_record_id, merged_sys_external_id, merged_at)`; "a bookmark, an integration's stored id ... still leads somewhere" (`merged-ids.ts:1-23`). **Consequence for an enrichment table keyed by contact id (Inferred, from the rules above):** the owning schema needs a registry entry and a policy for each such column (for example `reassign-dedupe` with `businessKey: [provider, external_key]` and `onCollision: keep-latest` for an identity link; `reassign` for a suggestion; `preserve-history` for an append-only audit), and either its own pair of participant functions or additions to Platform's `merge_plan_contacts`/`merge_apply_contacts`; without them the coverage check fails the repository that adds the table. Platform's existing examples: portal login (`platform.users.contact_id`, one login moves, two block), grants scoped to a record (`block`), associations and custom record references (`reassign`) (`record-merge-participant.md:64-112`; `lib/record-merge/platform-references.ts:42`, `:204-228`). --- ### 10. Explorer and snapshots (only insofar as an enrichment table participates) - **Explorer (Verified).** Holds saved views (a query anchored to a recipe, private or tenant-shared), navigation sessions (frozen id lists for Previous/Next) and a person's arrangement; "None of it holds record data" (`docs/explorer.md:3-25`). `POST /explorer/{tenantId}/selection` answers a whole result set as ids, **capped at 10,000** with `truncated: true`, "ids only, never record data. Whatever the console does next - a bulk edit, an export, a mail merge - loads each record through its own API" (`explorer.md:301-331`). An enrichment table participates only by being a registered dataset (a view anchors to a recipe whose datasets must be registered, `explorer.md:19-21`, `:76-94`). A frozen **server-side** selection is Planned/Absent (`03-existing-architecture-assessment.md:165`; `rhythm-data-job-engine-plan.md:256-267`, a Recommended design with `selection.queryId`, `evaluatedAt` columns present on `data_jobs.selection` but only `ids` accepted by `DataJobSelection`, `data-job-columns.ts:65-70`, `data-job-request.ts:47-50`). - **Snapshots (Verified).** Monthly, versioned, point-in-time snapshots on `@rhythmsoftware/macroservice-snapshots`; one shared run ledger `platform.snapshot_runs` for every service, with the owning service writing its own runs; a type is registered with `defineSnapshotType`, its rows name `run_id`, and it needs its own planner, dispatcher and job family (`docs/snapshots/README.md:1-23`, `:47-70`). An enrichment table needs a snapshot type only if "who was enriched with what as of month end" must be answerable later; the tenant data export already treats such a snapshot as customer data ("exported, because it is the customer's history", `docs/proposals/tenant-data-export/04-data-ownership-and-coverage.md:24-26`). --- ### 11. Bulk operations from the console: shape for "bulk enrich" to follow **Verified shape (the PLAT-M20 dialogs, CRM routes), PLAT-M20 built on CRM and unlanded in the console at the snapshot.** 1. The user selects records on a query result or the Explorer (page ids, "all matching" = the first 10,000 ids and the screen says so; the plan's assumption A4 is marked "Yes as built", not a dated ruling) (`docs/milestones/plans/PLAT-M20-plan.md:29`, `:11`). 2. **Update Fields...** opens a shared Mass Update dialog: choose a field from the operation's `patchableFields` (built-in and custom), enter the value with the field's own input, **Preview** runs the engine's validation ("35 will be updated, 0 conflicts, 0 rejected" with first refusals by code), **Apply** runs an interactive job with progress ("25 of 35") and then results (applied, conflicts, rejections) (`PLAT-M20-plan.md:16`). 3. The API: `POST /contacts/{tenantId}/data_jobs` with `{ operation: 'update', objectType: 'crm.contacts', contractVersion: 1, mode: 'validate' | 'apply', selection: { ids }, patch }` and an `Idempotency-Key` (`rhythm-data-job-engine-plan.md:1008-1035`; `services-crm:docs/data-jobs.md:45-57`); then `GET .../{jobId}`, `.../results`, `.../rejections`, `POST .../apply` (echoing `expected_impact_digest`), `.../pause`, `.../resume`, `.../cancel`, `.../retry`, `.../rollback`, `.../revalidate_conflicts` (`services-crm:docs/data-jobs.md:45-57`; `platform-data-operations.md:225-228`). 4. **Decided (Andrew, 2026-10-21 14:50 ET) PLAT-M20 D1 (a):** "every change is a data job; up to 500 records run on the interactive lane inside the dialog; above 500 the dialog hands over to the New Data Job wizard" (`PLAT-M20-small-bulk-operations.md:23`). The wizard is the five steps What, Source, Set Fields, Validate, Apply (`PLAT-M20-plan.md:37`; `milestone-plan.md:47-48`). 5. **Who may**: the object type's `.bulk` key plus `.update` or `.delete`; job visibility is the submitter's, plus holders of the bulk key (`PLAT-M20-plan.md:31`; `rhythm-data-job-engine-plan.md:1249-1250`). 6. **Same value on every record**: "A mass update sets the same value on every selected record; per-record values come from a file (the wizard)" (`PLAT-M20-plan.md:26`, approved plan section 6.2). A bulk apply of **different** suggested values per record is therefore the `records[]` form (an item per record with `values` and `expectedVersion`), the form the keyed second pass already uses (`macroservice-libraries:packages/data-jobs/src/data-job-request.ts:36-45`; `services-crm:modules/data-jobs/services/operations/contact-import.ts:16-20`). 7. **Settable fields**: "Fields a mass update may set are the operation's declared `patchableFields` (system columns and collections never)" (`PLAT-M20-plan.md:27`); for contacts these are the scalar fields, the reference keys and `custom_fields.*` (`services-crm:modules/data-jobs/services/operations/contact-update.schema.ts:66-70`). Hard delete from a list is a hard delete with the engine's safeguards, Decided (Andrew, 2026-10-19) CRM-Q10 (`PLAT-M20-plan.md:28`). --- ### 12. Tenant data export and data classification - **Verified.** `docs/data-exports.md` describes the built foundation of a tenant's complete export (WP-2: tables, definition and destination API, four permission keys, bucket, coverage registry, purge and reset participants); "Nothing generates a package yet" at that milestone (`data-exports.md:3-12`). All decisions of the design's chapter 20 "were all taken by Andrew on 2026-10-18" (`data-exports.md:4-6`). - **There is a classification, with seven classes.** The coverage registry's `classification` is one of `business`, `configuration`, `content` (exported inside a dataset) or `operational`, `proprietary`, `derived`, `deferred` (withheld, with a reason; `deferred` also names `until`) (`modules/data-exports/export-coverage/contract.ts:28-43`; `export-coverage-datasets.table.ts:8-12`). Every tenant-lifecycle resource carries an `ExportClassification`; a dataset may declare `withheldFields: [{ key, reason }]`, and the registry hashes them into the schema version (`contract.ts:62`, `:95-108`). The design lists six classes; class 5 "Platform operational data that belongs to Rhythm" is "no", class 6 "Proprietary implementation and infrastructure" is "never" (`tenant-data-export/04-data-ownership-and-coverage.md:11-20`); P-2 "Proprietary Rhythm Platform internals remain private ... a table is exported because a provider declares it, never because it carries `tenant_id`" (`02-principles-and-boundaries.md:14`). - **Derived data**: "exported when the customer cannot recompute it", a search index copy is withheld, a month-end snapshot is exported (`04-data-ownership-and-coverage.md:24-28`). - **Absent: third-party-licensed data.** A search of `modules/data-exports`, `docs/data-exports.md` and the 20 design chapters and README for "third-party", "licensed data", "licensed field", "data classification" and "enrich" finds nothing. The closest notions are `withheldFields` with a reason (per dataset field), `never_indexed` with a reason (Quick Search, `record-types.ts:166-170`), `exclude` with a reason (`defineObject`), field `exposure` `staff | owner | portal | public` (`docs/metadata.md:157`) and the export class `proprietary` (Rhythm's, not a vendor's). - **Other export paths carry no classification.** Query exports take "whatever running that query needs (`platform.queries.read` and the dataset's own key)" (`docs/queries/exports-and-schedules.md:29`); job result and rejection exports under `exports/` are 7-day objects "classified as tenant personal data" (`uploads.md:316-319`; `rhythm-data-job-engine-plan.md:1255-1256`). - **Verified, tenant lifecycle.** A purge or reset removes data by participants; a participant must be in the reset classification or "fails the unit guard rather than being quietly skipped" (`tenant-reset.md:56-64`); the coverage reconcile "refuses" a changed hash under an unchanged version (`data-exports.md:151-162`). --- ### Appendix. Quoted interfaces Comments are elided where marked `// ...`; every line shown is quoted from the cited source. #### A. The data operation contract (`macroservice-libraries:packages/data-jobs/src/define-data-operation.ts:266-343`, comments elided) ```ts export interface DataOperationDeclaration { readonly objectType: string; readonly operation: DataOperationKind; readonly contractVersions: Readonly>>; readonly permissions: { readonly submit: readonly string[]; readonly perRecord: string }; readonly patchableFields: readonly string[]; readonly matchingKeys: readonly DataMatchingKey[]; readonly dependsOn: readonly string[]; readonly groupBy?: (item: DataJobItem) => string | undefined; readonly itemDependencies?: (item: DataJobItem) => readonly DataItemDependency[]; readonly importBehaviour: { readonly suppress: readonly string[] }; readonly collectionMapping?: DataCollectionMapping; prevalidate(ctx: ExecutionContext, items: readonly DataJobItem[]): Promise; prepare?(ctx: ExecutionContext, items: readonly DataJobItem[]): Promise>; apply(ctx: ExecutionContext, item: DataJobItem): Promise; create?(ctx: ExecutionContext, item: DataJobItem): Promise; update?(ctx: ExecutionContext, item: DataJobItem & { readonly existingRecordId: string }): Promise; bulkApply?(ctx: ExecutionContext, items: readonly DataJobItem[]): Promise>; readonly rollback?: DataRollbackDeclaration; classifyError(error: unknown): DataErrorClassification; } ``` ```ts export interface DataJobItem { readonly itemSeq: number; readonly recordId?: string; readonly sourceKey?: string; readonly sourceSystem?: string; readonly sourceRevision?: string; readonly expectedVersion?: number; readonly groupKey?: string; readonly input: Input; readonly sourceRef?: DataJobSourceRef; readonly existingRecordId?: string; readonly prepared?: Prepared; } export interface DataApplyResult { readonly outcome: DataJobAppliedOutcome; readonly detail?: DataJobApplyDetail; readonly recordId: string; readonly recordVersion?: number; } export interface DataRollbackDeclaration { readonly compensatingOperation: DataOperationKind; readonly contractVersion: number; plan(ctx: ExecutionContext, request: DataRollbackRequest): Promise>; } ``` A worked declaration, CRM's contact import (`services-crm:modules/data-jobs/services/operations/contact-import.ts:83-97`): ```ts return defineDataOperation({ objectType: CONTACTS_IMPORT_OBJECT_TYPE, operation, contractVersions: { 1: { validate: (values, where) => contactImportValuesSchema(preferredFirst(crmValues(values)), where) } }, collectionMapping: CONTACT_COLLECTION_MAPPING, permissions: PERMISSIONS, patchableFields: libraryFieldPaths(CONTACT_IMPORT_FIELDS), matchingKeys: ['id', 'sourceKey'], dependsOn: [CONTACTS_IMPORT_OBJECT_TYPE], groupBy: (item) => item.sourceKey ?? item.recordId, importBehaviour: { suppress: [SuppressibleImportEffect.ComplianceNotice] }, ``` #### B. The submission request (`macroservice-libraries:packages/data-jobs/src/data-job-request.ts:19-69`, comments elided) ```ts export interface DataJobSource { readonly system: string; readonly deliveryId?: string; readonly objects?: readonly DataJobSourceObject[]; } export interface DataJobRecordInput { readonly id?: string; readonly sourceKey?: string; readonly sourceRevision?: string; readonly expectedVersion?: number; readonly groupKey?: string; readonly values?: unknown; readonly sourceRef?: DataJobSourceRef; } export interface DataJobRequest { readonly objectType: string; readonly operation: DataOperationKind; readonly contractVersion: number; readonly mode: DataJobMode; readonly purpose?: DataJobPurpose; readonly selection?: DataJobSelection; readonly patch?: Readonly>; readonly records?: readonly DataJobRecordInput[]; readonly source?: DataJobSource; readonly conflictPolicy?: DataJobConflictPolicy; readonly correctsJobId?: string; readonly expectedImpactDigest?: string; } ``` Wire examples, abridged from the plan's design examples (`services-platform:docs/proposals/rhythm-data-job-engine-plan.md:1015-1024`, `:1039-1055`; camelCase; the second shows only the plan's second record): ```json { "operation": "update", "objectType": "crm.contacts", "contractVersion": 1, "mode": "apply", "selection": { "ids": ["018f3a…01", "018f3a…02", "… 98 more"] }, "patch": { "jobTitle": "Board Member", "contactTypeId": "board" }, "notify": { "onCompletion": true } } ``` ```json { "operation": "upsert", "objectType": "crm.contacts", "contractVersion": 1, "mode": "apply", "source": { "system": "ams-sync", "deliveryId": "2026-10-11T02:00Z#118" }, "conflictPolicy": "reject", "records": [ { "sourceKey": "M-10443", "sourceRevision": "2026-10-10T22:15:40Z", "expectedVersion": 7, "values": { "jobTitle": "Treasurer" } } ] } ``` #### C. Job tables (`macroservice-libraries:packages/data-jobs/src/data-job-columns.ts`, comments elided) ```ts export function dataJobColumns() { // :77-142 return { id: idColumn(), tenantId: text('tenant_id').notNull(), objectType: text('object_type').notNull(), operation: text('operation').$type().notNull(), contractVersion: integer('contract_version').notNull(), mode: text('mode').$type().notNull(), purpose: text('purpose').$type().notNull().default(DataJobPurpose.Operational), selection: jsonb('selection').$type(), patch: jsonb('patch').$type>(), source: jsonb('source').$type(), conflictPolicy: text('conflict_policy'), correctsJobId: text('corrects_job_id'), rollsBackJobId: text('rolls_back_job_id'), dataRunId: text('data_run_id'), sysState: text('sys_state').$type().notNull().default(DataJobState.Accepted), sysLane: text('sys_lane').$type().notNull().default(DataJobLane.Interactive), sysRequestHash: text('sys_request_hash').notNull(), sysImpact: jsonb('sys_impact').$type>(), sysImpactDigest: text('sys_impact_digest'), sysValidatedVersions: jsonb('sys_validated_versions').$type>(), sysSuppressed: text('sys_suppressed').array(), sysItemCount: integer('sys_item_count').notNull().default(0), sysBatchCount: integer('sys_batch_count').notNull().default(0), sysFinishedBatchCount: integer('sys_finished_batch_count').notNull().default(0), sysAppliedCount: integer('sys_applied_count').notNull().default(0), sysRejectedCount: integer('sys_rejected_count').notNull().default(0), sysBlockedCount: integer('sys_blocked_count').notNull().default(0), sysConflictCount: integer('sys_conflict_count').notNull().default(0), sysSkippedCount: integer('sys_skipped_count').notNull().default(0), sysNotAttemptedCount: integer('sys_not_attempted_count').notNull().default(0), sysPauseCode: text('sys_pause_code'), sysFailureCode: text('sys_failure_code'), sysFailureSummary: text('sys_failure_summary'), sysOrchestrationJobId: text('sys_orchestration_job_id'), sysBulkState: jsonb('sys_bulk_state').$type(), sysQueuedAt: timestamptz('sys_queued_at'), sysStartedAt: timestamptz('sys_started_at'), sysFinishedAt: timestamptz('sys_finished_at'), sysValidatedAt: timestamptz('sys_validated_at'), sysExpiresAt: timestamptz('sys_expires_at'), ...systemColumns(), } as const; } ``` ```ts export function dataJobReceiptColumns() { // :224-244 return { tenantId: text('tenant_id').notNull(), jobId: text('job_id').notNull(), itemSeq: integer('item_seq').notNull(), batchSeq: integer('batch_seq').notNull(), recordId: text('record_id'), sysOutcome: text('sys_outcome').$type(), sysDetail: text('sys_detail').$type(), sysRecordVersion: integer('sys_record_version'), sysSuppressed: text('sys_suppressed').array(), sysAttempt: integer('sys_attempt').notNull().default(0), sysRecordedAt: timestamptz('sys_recorded_at').notNull().defaultNow(), sysCompletedAt: timestamptz('sys_completed_at'), } as const; } ``` Rejection columns (`data-job-columns.ts:250-273`, listed, not quoted): `tenantId`, `jobId`, `itemSeq`, `batchSeq`, `sysAttempt`, `sysOutcome` (`rejected | blocked | conflict`), `sysCode`, `sysFieldPaths text[]`, `sysMessage`, `sysConstraint`, `dependencyItemSeq`, `sourceKey`, `groupKey`, `sourceRef jsonb`, `sysRecordedAt`; no column holds a record value. Items, batches, batch queue and submissions: `data-job-columns.ts:145-166`, `:168-199`, `:206-217`, `:276-286`. Keys: `primaryKeyOn(tableName, t.tenantId, t.jobId, t.itemSeq)` for receipts and items, `(tenant, job, batch)` for batches, `(tenant, job, item, attempt)` for rejections, `(tenant, submitter, idempotencyKey)` for submissions (`:349`, `:374`, `:419`, `:439`, `:451`). The crosswalk (`data-id-mapping-columns.ts:33-53`, `:60-76`): ```ts export function dataIdMappingColumns() { return { tenantId: text('tenant_id').notNull(), sourceSystem: text('source_system').notNull(), objectType: text('object_type').notNull(), sourceKey: text('source_key').notNull(), recordId: text('record_id').notNull(), lastSourceRevision: text('last_source_revision'), sysLastAppliedVersion: integer('sys_last_applied_version'), sysFirstJobId: text('sys_first_job_id').notNull(), sysLastJobId: text('sys_last_job_id').notNull(), sysLastAppliedAt: timestamptz('sys_last_applied_at').notNull().defaultNow(), } as const; } // constraints primaryKeyOn(tableName, t.tenantId, t.sourceSystem, t.objectType, t.sourceKey), index(`${tableName}_record_idx`).on(t.tenantId, t.objectType, t.recordId), ``` States and outcomes (`data-job-states.ts:12-26`, `:79-89`): states `accepted, validating, validated, queued, running, paused, cancelling, cancelled, completing, succeeded, completed_with_rejections, failed, expired`; outcomes `applied, rejected, blocked, conflict, skipped_duplicate, skipped_stale, not_attempted`. Limits (`data-job-config.ts:18-23`): `{ maxItems: 500, batchSize: 25, maxSavepointObjects: 25, validationExpiryHours: 24 }`. #### D. Platform's run and upload tables (`services-platform:modules/data-jobs/db/`; column lists compacted from the sources, not verbatim) `data_runs` (`data-runs.table.ts:81-107`, comments elided): `id` (`dr_` + UUIDv7), `tenant_id`, `package_id`, `label`, `purpose` (`['initial_import']`), `mode` (`validate | apply`), `manifest_upload_id`, `source_system`, `source_tenant_id`, `format_version`, `sys_state` (`registered, running, paused, failed, succeeded, cancelled`), `sys_started_at`, `sys_finished_at`, `sys_failure`, system columns. `data_run_steps` (`data-run-steps.table.ts:72-111`): `id` (`drs_`), `tenant_id`, `run_id`, `dataset_id`, `object_type`, `contract_version`, `operation` (`insert, upsert, update, delete`), `format` (`jsonl`), `upload_id`, `bytes`, `sha256`, `record_count`, `depends_on text[]`, `sys_position`, `sys_state` (`pending, running, succeeded, blocked, failed, skipped`), `sys_job_id`, `sys_attempts` (max 2), `sys_succeeded_count`, `sys_rejected_count`, `sys_blocked_count`, `sys_skipped_count`, `sys_started_at`, `sys_finished_at`, `sys_failure`. `data_job_uploads` (`data-job-uploads.table.ts:112-143`): `id` (`upl_` + UUIDv7), `tenant_id`, `file_name`, `content_type` (five), `declared_size`, `declared_sha256`, `part_size`, `s3_key`, `s3_upload_id`, `sys_state` (`open, completed, aborted, expired`), `sys_version_id`, `sys_size`, `sys_etag`, `sys_sha256`, `sys_completed_at`. #### E. Custom field definition, rules, value and registry tables (E.1 and E.2 compacted; the value table is verbatim) `platform.custom_fields` (`services-platform:modules/metadata/db/custom-fields.table.ts:68-103`): `tenant_id`, `object_id`, `key`, `label`, `description`, `required` (`api|console|portal|none`, default `none`), `...fieldDefinitionColumns()` (data type, display type and typed configuration), `display_order`, `group_label`, `archived_at`, `...snakeSystemColumns()`; PK `custom_fields_pk (tenant_id, object_id, key)`; checks for key (`__c` for native objects, plain for `objects.%`), group label length. `platform.registered_objects` (`registered-objects.table.ts:24-58`): `id`, `application`, `label`, `plural_label`, `description`, `table_schema`, `table_name`, `record_id_field`, `supports_custom_fields`, `supports_configurable_forms`, `supports_configurable_page_layouts` (each `boolean NOT NULL DEFAULT false`), `release`, `retired_at`, system columns; unique `(table_schema, table_name)`. The value table every owner builds (`macroservice-libraries:packages/custom-fields/src/custom-field-values.ts:95-113`): ```ts tenantId: text('tenant_id').notNull(), recordId: text('record_id').notNull(), customFieldId: text('custom_field_id').notNull(), stringValue: text('string_value'), integerValue: bigint('integer_value', { mode: 'number' }), numericValue: numeric('numeric_value'), booleanValue: boolean('boolean_value'), fileUrlValue: text('file_url_value'), listValue: text('list_value').array(), dateValue: date('date_value', { mode: 'string' }), dateTimeValue: timestamptz('date_time_value'), referenceValue: text('reference_value'), htmlValue: text('html_value'), sysLastModifiedAt: timestamptz('sys_last_modified_at').notNull().defaultNow(), sysLastModifiedById: text('sys_last_modified_by_id').notNull(), ``` ```ts primaryKey({ name: `${tableName}_pk`, columns: [t.tenantId, t.recordId, t.customFieldId] }), foreignKey({ name: owner.foreignKeyName, columns: [t.tenantId, t.recordId], foreignColumns: [owner.tenantId, owner.id] }).onDelete('cascade'), check(`${tableName}_one_value_check`, sql`num_nonnulls(${t.stringValue}, ${t.integerValue}, ${t.numericValue}, ${t.booleanValue}, ${t.fileUrlValue}, ${t.listValue}, ${t.dateValue}, ${t.dateTimeValue}, ${t.referenceValue}, ${t.htmlValue}) = 1`), check(`${tableName}_list_check`, sql`${t.listValue} IS NULL OR cardinality(${t.listValue}) > 0`), ``` CRM's contact value table is one call (`services-crm:modules/entities/db/contact-custom-field-values.table.ts:18-22`): `customFieldValuesTable(crmSchema, 'contact_custom_field_values', { tenantId: contacts.tenant_id, id: contacts.id, foreignKeyName: 'contact_custom_field_values_contact_fk' })`. #### F. Dataset registration shape Owner's metadata declaration (`services-platform:docs/metadata.md:219-236`): ```ts export const contactsObject = defineObject(contacts, { id: 'crm.contacts', label: 'Contact', pluralLabel: 'Contacts', recordIdField: 'id', fields: { number: { label: 'Contact Number' }, status: { options: { active: 'Active', ... } }, spouseId: { target: 'crm.contacts' }, imageUrl: { label: 'Photo', dataType: 'image' }, notes: { dataType: 'large_text' }, firstName: { required: true }, fullName: { write: 'computed' }, }, collections: { emailAddresses: { label: 'Email Addresses', itemLabel: 'Email Address', maxItems: 3 }, addresses: { label: 'Addresses', itemLabel: 'Address', itemObject: 'crm.contact_addresses' }, }, }); ``` and the capability block (`standards/custom-field-capability.md:75-85`): `capabilities: { customFields: true, configurableForms: true, configurablePageLayouts: true }`; the exclusion member (`object-registration.ts:439`): `readonly exclude?: Readonly, string>>>;`. The query registration members (`query-registration.ts:723-822`, comments elided, abbreviated to the members that matter here): ```ts export const queryObjectSchema = z.object({ objectId: objectId, permission: z.string().max(200).regex(PERMISSION_KEY_PATTERN, 'must be a permission key ..'), scope: z.enum(QUERY_OBJECT_SCOPES).optional(), category: label.optional(), recordNumberField: key.optional(), recordLabelField: key.optional(), linkFields: z.array(key)...optional(), recordLinks: z.record(key, key)...optional(), defaultFields: z.array(fieldId).max(255).default([]), defaultSort: z.array(z.object({ field: fieldId, direction: z.enum(['asc', 'desc']).default('asc') }).strict()).max(5).default([]), defaultCriteria: criteriaGroupSchema.optional(), hiddenFields: z.array(fieldId).max(1000).default([]), discovery: discoveryRuleSchema.optional(), recipientRelationships: ...optional(), businessEvents: ...optional(), cancellation: cancellationSchema.optional(), customFieldValues: tableRef.optional(), source: tableRef.optional(), discriminator: queryDiscriminatorSchema.optional(), computedFields: z.array(computedFieldSchema).max(100).optional(), configuredRelationships: z.array(configuredRelationshipSchema).max(20).optional(), picks: z.array(queryPickSchema).max(100).default([]), collections: z.array(queryCollectionSchema).min(1).max(20).optional(), quickSearch: quickSearchSchema.optional(), }).strict() ``` The manifest (`query-registration.ts:935-944`): `{ formatVersion: 1, application, applicationLabel, release, objects: queryObjectSchema[] (max 500) }`, `.strict()`. Platform's key list entry (`services-platform:modules/queries/datasets.ts:71`): `'crm.contacts': { permission: 'crm.contacts.query', also_requires: ['crm.contacts.read'], scope: 'tenant' },` The Quick Search record type (`modules/quick-search/registry/record-types.ts:142-165`): `contacts: { owner_application: 'crm', dataset_id: 'crm.contacts', required_permission: 'crm.contacts.read', dataset_query_permission: 'crm.contacts.query', providers: ['algolia'], projection: { display: {...}, default_fields: [{ field: 'name', tier: 1 }, ...], never_indexed: {...} } }`. #### G. Platform data operation values (`services-platform:docs/data-jobs/platform-data-operations.md:11-15`, `:41-54`) Object types and keys: `platform.custom_fields` (`insert`, `upsert`; keyed `sourceKey` within `source.system`; record id `:`), `platform.custom_object_records` (`upsert`; `:`), `platform.associations` (`upsert`; each endpoint `:` or `id:`). A definitions line: ```json { "sourceKey": "CF-BOAT-HULL", "values": { "objectId": "objects.boats", "key": "hull_number", "label": "Hull Number", "dataType": "text", "defaultDisplayType": "single_line_text", "configuration": {}, "rules": { "unique": true, "exposure": "portal" } } } ``` Read-back by source key (`platform-data-operations.md:166-173`): `POST /custom_fields/{tenantId}/data_jobs/records_by_source_key { "source_system": "rhythm1", "source_keys": ["CF-1", "CF-2"] }` answers `{ "items": [ { "source_key", "record_id", "record" } ], "missing": [...] }`. #### H. Record merge participant contract (`macroservice-libraries:packages/record-merge/src/participant.ts`, `policies.ts`; the interfaces are compacted, the policy union is verbatim) ```ts export const MERGE_PARTICIPANT_CONTRACT_VERSION = 1; export interface ParticipantAction { table; column; policy: MergePolicyKind; rows: number; summary: string; destructive: boolean; rowIds?: readonly string[]; } export interface ParticipantPlan { contractVersion; actions; decisions; blockers; warnings; } export interface ParticipantApplyResult { contractVersion; actions: ParticipantApplyAction[]; } ``` ```ts export type MergePolicy = | { readonly kind: 'owned' } | { readonly kind: 'reassign' } | { readonly kind: 'reassign-dedupe'; readonly businessKey: readonly string[]; readonly onCollision: CollisionRule } | { readonly kind: 'consolidate'; readonly handler: string } | { readonly kind: 'preserve-history' } | { readonly kind: 'clear' } | { readonly kind: 'decide' } | { readonly kind: 'block' } | { readonly kind: 'not-a-reference'; readonly reason: string }; ``` SQL entry points (`record-merge-participant.md:14-18`): `platform.merge_plan_contacts(survivor_id text, merged_ids text[]) -> jsonb`; `platform.merge_apply_contacts(merge_id text, survivor_id text, merged_ids text[], decisions jsonb) -> jsonb`; the same for organizations. #### I. Export declaration (`services-platform:modules/data-exports/export-coverage/contract.ts:28-43`, `:51-75`; the declaration interface is compacted) ```ts export const EXPORT_CLASSES = ['business', 'configuration', 'content', 'operational', 'proprietary', 'derived', 'deferred'] as const; export type ExportClassification = | { readonly class: 'business' | 'configuration' | 'content'; readonly dataset: DatasetKey } | { readonly class: 'operational' | 'proprietary'; readonly reason: string } | { readonly class: 'derived'; readonly reason: string } | { readonly class: 'deferred'; readonly until: string; readonly reason: string }; export interface ExportDatasetDeclaration { readonly key: DatasetKey; readonly schemaVersion: string; readonly objectId?: string; readonly label: {...}; readonly description: string; readonly resources: readonly string[]; readonly identifier: 'id' | string; readonly fields?: readonly ExportFieldDeclaration[]; readonly withheldFields?: readonly { readonly key: string; readonly reason: string }[]; readonly relationships: readonly { field; dataset; kind: 'many_to_one' | 'one_to_one' | 'link' }[]; readonly embedded?; readonly assets?; readonly history?: boolean; readonly consistency: 'as_of_cut_off' | 'snapshot_per_batch' | 'current'; readonly order: {...}; ... } ``` #### J. The integration contract for enrichment (`macroservice-libraries:packages/integration-contracts/schemas/capabilities/crm/crm.contact.enrich@1.json`) `$id: https://contracts.rhythmsoftware.com/capabilities/crm.contact.enrich/1`, title "Enrich a contact". `request` (required `address`, `fields_requested`): `address` is `/common/1#/$defs/address`; `fields_requested` is an array of 1 to 100 unique strings matching `^[a-z][a-z0-9_]*$`. `response` (required `matched`, `fields`): `matched: boolean`; `fields` is an object of at most 100 properties whose values are `string (max 1000) | number | boolean | null | string[] (max 100 items)`; "a field it could not answer is absent". --- ### Inferred: How enrichment runs, bulk apply and provenance map onto the Data Job Engine and the query layer; what is missing Everything in this section is my reading of the evidence above, not a source statement. #### I-1. What maps cleanly (reuse as is) 1. **Applying accepted enrichment values to contacts is a data job.** Per-record values with an `expectedVersion` for each contact is the `records[]` form of `crm.contacts` `update` (the form the keyed spouse pass already uses, `services-crm:modules/data-jobs/services/operations/contact-import.ts:16-20`), settable fields being the scalar fields, reference keys and `custom_fields.*` (`services-crm:modules/data-jobs/services/operations/contact-update.schema.ts:66-70`). That gives, without new engine code: validate then apply with an impact digest, version capture and a named conflict when staff edited the contact after the suggestion was made (D-37), per-item outcomes and rejections, pause/resume/cancel, a correction job (`correctsJobId`), 30-day submission idempotency, the interactive lane (500) and the CRM bulk lane (files), and a revision per contact with `sys_source = 'job'`, `sys_job_id` and the submitter (Section 1.8). 2. **The console shape is the PLAT-M20 dialog**: select records, preview, apply, results with a link to the job. A bulk **apply of the same value** is `selection + patch`; a bulk apply of **per-record suggested values** is the file/records form behind the five-step wizard (above 500, or by D1 (a) interactive up to 500). 3. **A provider result file can be ingested through the existing upload protocol** (ndjson, csv, gzip), read by a service that has a bulk lane; per-line rejections by line number come free. 4. **Provider key to contact link as a crosswalk.** `data_id_mappings (tenant, source_system, object_type, source_key)` plus `records_by_source_key` is already "which record does their key name", with replay identity (`sourceRevision`) and a conflict policy. Using `source_system = 'enrichment:'` is consistent with the Integration Platform's own convention (`source: { system: 'integration:', deliveryId: : }`, `17-synchronization.md:81-84`), which is Planned. 5. **Status/history of an enrichment run could sit on the engine's job tables** if the run's unit of work is "apply these values": counts, retry of blocked rows, rejection export, retention (90 days), tenant purge/reset participants (`platform.data-operations` participant for Platform's tables, `data-operations-participant.ts`). #### I-2. What does not map (missing, with the evidence) | # | Gap | Evidence | | --- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | | G1 | **The engine applies supplied values; it does not fetch.** A provider call per record is outbound I/O that must not sit inside a batch transaction (the engine "calls nothing AWS inside a transaction", `README.md:540-542`; `prepare` runs with no transaction open but "Preparation is therefore repeated whenever a batch is, and must be safe to repeat", `define-data-operation.ts:313-317`, which would repeat billable provider calls on redelivery). The fetch side needs a different durable mechanism (a `lib/jobs` job family, as quick-search rebuild and tenant reset use) or the Integration Platform gateway; the engine can carry only the apply side. | Inferred from the cited text | | G2 | **No `enrichment` object type or operation exists** in Platform or CRM; a new object type means a new `defineDataOperation`, the routes under that record path, a console entry (no served capability registry: "the console hard-codes which operations exist"), and a `DATA_RUN_SERVICES` entry only if a Data Run were ever used. | `03-existing-architecture-assessment.md:167`; `data-operations.ts:2-4` | | G3 | **Platform has no bulk lane** (500-item cap, runs in the request). A Platform-owned enrichment table loaded from a large provider file needs either a lane built for Platform (queue, dispatcher, worker, Step Functions, budget share: the shares are CRM and Accounting only, `bulk-batch-budget.ts:33-36`) or the file ingested by a service that has one. | `platform-data-operations.md:195-199`, `:254-262` | | G4 | **Data Runs cannot carry recurring work** (initial import only, implementation tenants in `loading`). A recurring or on-demand enrichment run is not a Data Run. | `data-runs.table.ts:17-19`, `:62` | | G5 | **No schedules, no export lane, no frozen server-side selection.** "All matching" is a browser-held list capped at 10,000 ids; a scheduled enrichment sweep has no engine support (the proposal says recurring imports would be Integration Platform sync runs that submit data jobs). | `03-existing-architecture-assessment.md:165`, `:169-170`; `data-engine-architecture/06-core-data-model.md:831` | | G6 | **Rollback of enriched custom field values and collections is not available**; contact scalar fields are. If enrichment writes custom fields (district, congressional member) or an address, undo is manual from history. | `services-crm:docs/debt/rollback-of-collections.md:3-8` | | G7 | **Bulk update cannot set collections** (e-mail, phone, address); only whole-record upsert can, which replaces the contact "whole" and "the fields it omits are cleared". Enriched addresses and phones cannot be applied by a patch. | `PLAT-M20-plan.md:27`; `contact-import.ts:213-219`; `services-crm:docs/data-jobs.md:272-275` | | G8 | **No per-field provenance anywhere.** Value rows carry the last modifier only; a revision names the job. Which provider and retrieval produced a value would be derivable only by joining revision `sys_job_id` to the job's `source.system` and to the run that produced the job (Inferred); there is no source, confidence, retrieved-at or licence on any value table, and `app.source = 'integration'` is an unused slot, planned for the Integration Platform. | `custom-field-values.ts:111-112`; `17-synchronization.md:91` | | G9 | **No identity-match primitive.** The engine matches by `id`/`sourceKey` only; the contract's natural-key and `ambiguous_match` support is unimplemented; Quick Search is a browser type-ahead. A matching service would use the query layer on `primary_email` (unique among live contacts) or CRM's duplicate match keys (`email`, `phone`, `name_sound`, `origin_id`) as evidence, and would own its own confidence and review states. | Sections 2 and 7 | | G10 | **No licensed-data classification** in exports, queries, Quick Search or field permissions (only reasoned `withheldFields`, `never_indexed`, `exclude`, and the dataset-level query key). Field-level security was withdrawn from the semantic layer (decision 11). A licence restriction on provider fields would have to be enforced by choosing, per surface, an exclusion: separate dataset with its own key, `exclude`/`hiddenFields`, `never_indexed`, export `withheldFields` or a new export class. | Sections 5.4 and 12 | | G11 | **Engine-side housekeeping that a long-lived enrichment history would want** is not built: receipt/rejection retention sweep, `expireValidatedJob` scheduled caller, an events table for interactive jobs. | `03-existing-architecture-assessment.md:173-174`; Section 1.2 | | G12 | **Upload digest verification above 256 MiB**, and zip unsupported. For a large provider file, integrity rests on size and part ETags unless the digest is measured by a longer-lived step. | `uploads.md:300-309`, `:343-345` | | G13 | **Custom field capability is a product decision**: enriched values written to custom fields require the entity to be approved (contacts and organizations are); a new entity table with custom fields needs Andrew's decision first (R2-ARCH-001 rule 3). | `standards/custom-field-capability.md:19-24` | #### I-3. Where enrichment data would live, and what each choice triggers | Choice | Triggers | Evidence | | --------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------- | | Enriched **values on the contact** (scalar column or `custom_fields`) | written only via CRM's own persistence (patch/import, or the `crm.contact.enrich@1` contract); history/revisions already exist; rollback only for scalar fields; queryable already (`contacts` dataset); per-field provenance absent | Sections 1.7, 6.5 | | **Suggestions and identity links in a new table** keyed by contact id | table design (typed columns, RLS, normalization trigger); metadata and query registration plus a `QUERY_DATASETS` entry and keys (`.query`, and a read key if read one at a time); description-stamp triggers; merge registry entries and participant functions; tenant lifecycle participant (purge/reset/export declaration and classification); optional snapshot type; optional Quick Search declaration (a contact-dataset `quickSearch` can only name fields of its own dataset, one hop through references) | Sections 5.3, 9, 12; `record-merge-participant.md:32-35` | | A **tenant-defined custom object** | no product deploy, automatic dataset, 50/200 caps, per-tenant definition recreated after reset, no licence/retention concepts | Section 8 | | The **crosswalk alone** for provider identity | free replay identity and `records_by_source_key`; no confidence, status or match method columns; owned by the data-job writer; "only the owner writes them" | `data-id-mapping-columns.ts:13-15`, `:33-53` | #### I-4. Cross-cutting rules an enrichment design will meet - Every read is allowed; **only the owner writes** (a provider adapter or an enrichment service cannot write `crm.contact_custom_field_values` directly; it supplies values through CRM's contract surface). - Hard permissions: the applier needs the object's `.bulk` key plus `.update` (re-checked per batch as the submitter); the viewer of suggestions needs the new dataset's `.query` key and, per CRM-ACC-0064, the contact `.read` key when results are joined to contacts. - Naming and storage conventions that bind a new module: `sys_` for everything the system fills in (Decided (Andrew, 2026-10-11/12)); a plural table name; every composite key leading with `tenant_id`; forced RLS (`data-job-columns.ts:9-17`; `README.md:491-495`). - Deployment order is a real constraint: library bump and Platform first, owner second (Section 5.1); a manifest member unknown to Platform makes the owner's deploy fail at registration. - Two approvals gate scope, not code: custom field capability per entity (Andrew only), and any change to the platform's "initial import only" Data Run purpose. #### I-5. Open items this evidence cannot settle 1. Whether Andrew intends enrichment to be a Platform capability (then it inherits "no bulk lane") or a new service owning its own tables and bulk lane; neither is decided in the documents read. 2. Whether `crm.contact.enrich@1` is the intended contract for the apply side or only for address-triggered district lookups; its description names a rule that "sees an address change" and the contract has no run, confidence or licence members. 3. Whether PLAT-M21's server-side results API (R3) and frozen selections arrive before an enrichment design needs them (M21 is `Proposed`). ## 22. R5. Rhythm 1.0 FullContact and Google Civic enrichment, and the customer-360 precedent ## R5. Rhythm 1.0 FullContact and Google Civic enrichment, and the customer-360 precedent Evidence report for the Rhythm 2.0 Data Enrichment architecture proposal. Read-only investigation, 2026-10-22. ### How to read this report - Every claim carries a label: **Verified** (read in code or config, cited `path:line`), **Inferred** (a conclusion drawn from what was read; the reasoning is stated), or **Absent** (searched for and not found; the search is stated). - Paths are relative to `/Users/andrew/rhythm` (Rhythm 1.0, DynamoDB and Lambda) unless prefixed `R2:` (= `/Users/andrew/rhythm2`). - "Contacts engine" = `rolodex-contacts/src/logic/full-contact-enrichment-engine.js`. "Organizations engine" = `rolodex-organizations/src/logic/full-contact-enrichment-engine.js`. "C360" = `customer-360-commitments-dashboard/api` (the file set in `customer-360/api` has identical `src/fullcontact/`). - Secondary evidence: `.csi-worktrees/docs-contract-draft/specs/apps/rolodex/` is an earlier spec-retrofit of Rolodex 1.0 written from the same code. I use it only to corroborate, never as the sole source (cited as "retrofit"). - A literal FullContact key sits in a comment in the contacts engine (line 68). It is **not** reproduced here. It is relevant only as a finding (section 6). --- ### Summary (10 bullets) 1. **1.0 enrichment was purely lookup-on-demand, never a background process.** FullContact was called only from three places: the console contact form (on email field blur, new contacts only), the portal sign-up and user-link components (every new self-registered member with an email), and a server step that creates an organization from a new member's email domains. There is no stream, queue or schedule that enriched existing records. (Verified: only entry points are `rolodex-contacts/src/lambda/api.js:68-76`, `rolodex-organizations/src/lambda/api.js:84-104`; `rolodex-eventbridge` has no enrichment code.) 2. **Two FullContact v3 endpoints, nothing else.** Person: `POST /v3/person.enrich` with only `{email}`. Company: `POST /v3/company.enrich` with only `{domain}`. No name, company or phone hints were sent. (Verified: contacts engine `:70-75`, organizations engine `:70-77`.) 3. **The raw vendor response was returned to the client and the client decided what to write.** The service stored and returned the whole FullContact payload (plus `email_address`, `image_url = avatar`, lowercased `gender`). The Angular forms mapped it onto fields: photo, name, title, bio to notes, gender, Twitter, Facebook, LinkedIn for contacts; name, logo, bio, socials, website, first location for organizations. (Verified.) 4. **No tenant opt-in, no budget, no throttle.** One global key in AWS SSM (`/fullcontact/api-key`) used by every tenant. The only switches were per-web-form checkboxes in the form designer ("Disable Social Lookup?") and two portal settings that govern organization matching and creation. The Civic integration, by contrast, was gated by a per-tenant permission-profile flag `integrations.google_civic`. (Verified.) 5. **The cache was global and permanent.** Both caches are DynamoDB tables keyed by email (or domain) alone, with no tenant in the key. The tables declare a TTL on `sys_expires_at`, but neither engine sets it, so entries never expired, and misses were never cached (a repeat of a failed lookup paid again). Third-party personal data fetched for one tenant was served to all. (Verified in code; the "paid again" and "served to all" consequences are Inferred; the retrofit independently records the same finding FND-ROL-005.) 6. **Re-enrichment cadence: never.** `date_social_profile_last_updated` is written when the form is populated and read by nothing that refreshes. (Verified by grep: no reader outside field pass-through and exports.) 7. **Photos were stored as pointers to the vendor's CDN, not copied.** `image_url` held `https://img.fullcontact.com/static/...`; the platform file-move logic only acts on `s3://` staging URLs and deliberately ignores everything else. No licensing note exists anywhere. (Verified; licensing is Absent.) 8. **Failure handling was thin.** Every vendor failure (404, 401, 429, 5xx, timeout) collapsed into one string, `ProfileNotFound: ...`; no retry, no backoff, no negative cache; the UI showed "No information found" or a generic error. Tests that hit the live API were skipped with the note "re-enable once usage limits have been addressed". (Verified.) 9. **Google Civic was the better-engineered 1.0 example**: a separate partner microservice, change-stream driven, per-tenant gated, address-diff gated, US-only, plus a bulk re-run state machine. It stored only six district fields on the contact and fetched officials live, on view, never stored. (Verified.) 10. **The 2025/26 customer-360 tool is the modern precedent and it fixes most 1.0 weaknesses**: provider behind a domain-owned interface, a minimum-confidence gate, typed outcomes, bounded retry with exponential backoff, a persisted per-record status (`resolved`, `not_found`, `failed`, `ineligible`) that makes misses terminal and failures retryable, source-wins precedence, SSRF-guarded download and re-hosting of the image, and non-blocking failure. It also shows the limits: it is single-tenant, has no result cache, ran only from local scripts, and one path (create-time) still persists the vendor URL. (Verified.) --- ### 1. What 1.0 enrichment actually did #### 1.1 Entry points and triggers | # | Trigger | Who/where | Calls | Evidence | | --- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | A | Staff types an email into the new-contact form and leaves the field (blur) | Console web form component `rolodex-forms-contact-social-lookup` | `GET /contacts/{tenant}/socialProfile/{email}` then, if no organization matches the domain, `GET /organizations/{tenant}/socialProfile/{domain}` | `rolodex-ngx-controls/projects/rolodex-ngx-controls/src/lib/contacts/webForms/contact-social-lookup/contact-social-lookup-web-forms.component.html:22` (`(blur)="processEmailChange()"`), `.ts:85-116`, `.ts:215`, `.ts:380` (Verified) | | B | Staff types a domain into the new-organization form and leaves the field | `rolodex-forms-organization-social-lookup` | `GET /organizations/{tenant}/socialProfile/{domain}` | `.../organizations/webForms/organization-social-lookup/organization-social-lookup-web-forms.component.html:20`, `.ts:34-81` (Verified) | | C | Portal member signs in for the first time and has no contact: the component builds a contact from the identity provider, looks up the email, then saves | Portal components `create-and-link-new-contact` and `user-contact-association` | `GET /contacts/{tenant}/socialProfile/{email}` | `platform-ngx-portal-controls/projects/platform-ngx-portal-controls/src/lib/components/signup/create-and-link-new-contact/create-and-link-new-contact.component.ts:80`; `.../user-contact-association/user-contact-association.component.ts:125` (Verified) | | D | Same portal flow, server side: the "create and link" workflow tries to match the email domain to an organization; if none and creation is not disabled, it creates the organization from FullContact company data | Step function `create-and-link` calls `rolodex_organization_create_from_domain` | `company.enrich` per non-public domain, first success wins | `rolodex-contact-steps/build/stateMachines/create-and-link.step.json:76-90`; `rolodex-contact-steps/src/lambda/steps.js:35-61`; `rolodex-organizations/src/lambda/api.js:89-104` (Verified) | | E | Corporate-membership flow: adding an employee by email triggers a lookup, but the result is discarded (`// todo - prase name from social info`) | `ngx-portal` corporate membership flow | `GET /contacts/.../socialProfile/{email}` | `ngx-portal/src/app/membership/pages/corporate-membership-flow/corporate-membership-flow.component.ts:175-188` (Verified) | | F | Legacy console (`console/`) contact form | Old Angular console | same endpoint; contains a hard-coded fake profile for the developer's own address | `console/src/app/rolodex/controls/contact-form/contact-form.component.ts:466-505` (Verified) | **Not triggers (Verified by absence):** - No create/update/delete stream handler in `rolodex-contacts` or `rolodex-organizations` calls either engine. Searched: `grep -rIn -i "fullcontact|enrich|socialProfile"` over both repos' `src/`; the only hits are the API handlers and the engines. `rolodex-eventbridge` contains no enrichment code (same grep, no hits). `libraries-rolodex-records` and `libraries-rolodex-processes` contain no enrichment helpers (hits only in `package-lock.json` for unrelated package names). - Contact edits of existing records never enrich: `if (this.contact?.sys_version) { ... declining to perform social lookup }` (`contact-social-lookup-web-forms.component.ts:126-129`, Verified). - CSV/clover imports and merges never enrich. Inferred from the same absence: import tooling (`pso-clover-membersuite`) only carries the field `date_social_profile_last_updated` through (`pso-clover-membersuite/graph/rolodex/contacts.grf:1657`). #### 1.2 What was sent to FullContact | Lookup | Endpoint | Body | Auth | Evidence | | ------- | ------------------------------------------------------ | ----------------- | ----------------------------- | ---------------------------------------- | | Person | `https://api.fullcontact.com/v3/person.enrich` (POST) | `{ email }` only | `Authorization: Bearer ` | contacts engine `:69-76` (Verified) | | Company | `https://api.fullcontact.com/v3/company.enrich` (POST) | `{ domain }` only | same | organizations engine `:70-77` (Verified) | - API version is **v3**. No v2 call exists in the repo: `grep -rIn "api.fullcontact.com"` returns only the two engines (v3) and the C360 config (Verified). (An older shape is visible in the legacy console's generated model, which uses `full_name`, `image_url`, `job_title` rather than the v3 `fullName`, `avatar`, `title`: `console/src/app/api/rolodex/model/socialProfile.ts:18,30,34`. Inferred: the response contract changed at some point and the generated clients were not all regenerated.) - Name, organization, phone or address were **never** sent. Match quality therefore rested entirely on FullContact's email-to-person and domain-to-company resolution (Inferred from the request body). - Person: the raw email as typed; the contacts engine does not lowercase or trim the key it caches under (`:27,:48`). The UI trims and lowercases only for its own "last looked up" check (`...ts:105-110`). Inferred consequence: `Jim@X.com` and `jim@x.com` are two cache entries and two paid calls. - Company: the domain is the host part of the member's email (e.g. `mail.example.org`, not the registrable domain) (`rolodex-contact-steps/src/logic/domain-logic.js:6-9`, Verified). The organizations engine caches under `domain.toLowerCase()` but sends the original-case value (`:31,:48` vs `:76`). #### 1.3 Fields written back: contacts The service returns the whole FullContact person payload; the contact schema documents these fields: `fullName, emailAddress, gender (male|female), avatar, organization, title, twitter, facebook, linkedin, bio, details` (`rolodex-contacts/src/schema/social-profile.schema.js:8-19`, Verified). The engine itself adds `email_address`, `image_url` (= `avatar`), and lowercases `gender` (contacts engine `:37-45`). Mapping applied by each caller: | Contact field | Console form (A) `contact-social-lookup-web-forms.component.ts:145-157` | Portal sign-up (C) `create-and-link-new-contact.component.ts:100-111` | | -------------------------------- | ------------------------------------------------------------------------- | --------------------------------------------------------------------- | | name | `profile.fullName` (fallback: existing) | `profile.fullName`, only if the identity provider gave no name | | job_title | `profile.title` (also tries `profile.job_title` into a key named `title`) | `profile.title` | | notes | `profile.bio` | `profile.bio` | | gender | lowercased `profile.gender` | `profile.gender` | | image_url | `profile.avatar` | `profile.image_url` (the engine's copy of avatar) | | twitter / facebook | `profile.twitter` / `profile.facebook` | same | | linked_in | `profile.linkedin` | `profile.linkedin` | | date_social_profile_last_updated | now | now | Networks populated: **Twitter, Facebook, LinkedIn** only. Instagram exists on the contact form (`rolodex-ngx-controls/.../services/contact-forms.service.ts:~137`) but no lookup fills it (Verified by reading the patch list). `details` (arbitrary vendor sub-object), `organization`, `ageRange`, `location` and similar are returned to the browser but **not** written to any contact field (Verified: not in either mapping). Organization name: the contact lookup does not set `organization_name` or `organization_id` from `profile.organization`. Organization linkage is domain-based only (section 1.5). #### 1.4 Fields written back: organizations Server-side creation (D), `convertToOrganization` (organizations engine `:101-149`): requires `name`; then `domain_name`, `logo_url` (= `logo`), `facebook`, `linked_in` (= `linkedin`), `twitter`, `website`, `description` (= `bio`), and an `address` from `details.locations[0]` (`line1, line2, city, state = region, postal_code, country = countryCode`). Verified. Client-side form (B), `generateOrganizationFromSocialProfile` (`organization-social-lookup-web-forms.component.ts:89-135`): `name, description (bio), logo_url (logo), twitter, facebook, linked_in, domain_name`, first location as address (`state = regionCode` here, whereas the server uses `region`), and `main_phone_number = details.phones[0].value`. Verified. **Not written** although the response schema carries them: `employees`, `founded`, `category` (`rolodex-organizations/src/schema/social-profile.schema.js:8-12`), `locale`, additional locations and phones. So **no company size, industry or founded year ever reached an organization record** (Verified: absent from both mappers). #### 1.5 Organization matching around the lookup (context for "enrichment" scope) - Console form: email domain is first matched against existing organizations through the Algolia index of `rolodex-organizations`, honoring `domain_name` and wildcard `alternate_domain_names` (`.../contact-social-lookup/organization-lookup.service.ts:14-67`). Zero hits then triggers the FullContact company lookup; the resulting organization is a **proposed new organization** (`id: '__new:from_org_info'`) attached to the form and created on save unless `disable_organization_auto_creation` (`...web-forms.component.ts:372-410`). One hit auto-selects (and pre-ticks "link work address/phone to organization"); several hits ask the user which. Verified. UI strings: "No matching organization in our system, checking the internet..." (`...component.html:62-65`). - Portal: server step `getOrganizationToLink` honors two tenant settings, `disable_organization_matching` and `disable_organization_creation_during_sign_up` (`rolodex-contact-steps/src/lambda/steps.js:35-61`; settings checkboxes at `rolodex-ngx-console/src/app/setup/pages/settings/settings.component.html:13-16`). Verified. #### 1.6 Storage semantics: overwrite vs fill-blanks | Surface | Semantics | Evidence | | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Console contact form | **New contacts only.** For a new contact the profile **wins over what the user already typed**: pattern `profile.x \|\| form.x` means vendor value first, form value as fallback. The user can still edit before saving. | `...web-forms.component.ts:126-129,145-157` (Verified) | | Console org form | Vendor values patched over the form (except the domain, deleted from the patch). | `organization-social-lookup-web-forms.component.ts:70-74` (Verified) | | Portal sign-up | Vendor value assigned **unconditionally** for title, notes, gender, image, socials, so a missing `profile.image_url` assigns `undefined` and removes the identity provider's picture (`contact.image_url = profile.image_url` after `image_url: usr.picture`). | `create-and-link-new-contact.component.ts:68,105`; `user-contact-association.component.ts:113,149` (Verified code; the lost-picture outcome is Inferred, since `deepRemoveObjectPropertiesThatAreWhitespace` runs afterwards) | | Server-side org creation | Create-only (a new organization from the vendor profile). | `rolodex-organizations/src/lambda/api.js:89-104` (Verified) | There were **no separate "suggested" fields, no provenance and no per-field source stamp**. The only provenance was the single timestamp `date_social_profile_last_updated` (contact only; organizations have none). Verified: contact schema `rolodex-contacts/src/schema/object.schema.js:112-117`; the 2.0 CRM table also carries only this column (`R2:services-crm/db/migrations/0001_contacts.sql:186`). --- ### 2. Configuration, ownership, opt-in, budget | Question | Answer | Evidence | | --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Where does the API key live? | One **global** SSM SecureString `/fullcontact/api-key` in the AWS account (per stage), read by the Lambda role of each of the two services. Not per tenant. | `rolodex-contacts/build/template-resources.yaml:30`; `rolodex-organizations/build/template-resources.yaml:53`; read at contacts engine `:58`, organizations engine `:62` (Verified) | | Is the key cached in process? | Contacts: yes, in a module-level variable for the life of the container, with no refresh (`let apiKey;` `:52`, `:57-65`). Organizations: no; one SSM read per FullContact call (`:62`). | Verified | | Per-tenant opt-in or opt-out for FullContact? | **Absent.** The tenant permission profile has a `google_civic` integration flag but no FullContact flag: the list of permissible integrations is `ai, blue_sky, ups, higher_logic, comm_partners, quickbooks_online, zoom, eshow, clowder, apply, new_target, informz, canvas, open_water, mailchimp, google_civic, outlook, zoom_pro, zapier, lockbox, honor_education, omni` (`platform-ngx-console/src/app/admin/pages/tenants/configure-tenant-permissions/configure-tenant-permissions.component.ts:69`). Searched `platform_tenant_permission_profiles.json` for `fullcontact`: no hits. | Verified | | What switches exist? | (1) Per web form instance, designer checkboxes: contact form "Disable Social Lookup?", "Disable Organization Matching?", "Disable Organization Auto Creation?" plus duplicate-detection toggles; organization form "Disable Social Lookup?" (`rolodex-ngx-console-controls/.../contact-social-lookup-designer.component.html:38-59`; `.../organization-social-lookup-web-forms-designer.component.html:47-48`; properties model `contact-social-lookup-properties.model.ts:1-7`). (2) Tenant settings `disable_organization_matching`, `disable_organization_creation_during_sign_up` (portal only). **The portal sign-up and user-link components have no switch for the person lookup**: they call `getSocialProfile` whenever the member has an email (`create-and-link-new-contact.component.ts:78-84`). | Verified | | Who may call the person lookup? | The contact service lists `getSocialProfile` among `portalConfiguration.authorizedOperations`, i.e. a signed-in portal user may call it for **any** email address in the path (no check that it is their own). | `rolodex-contacts/config.js:227-228` (Verified list; Inferred: arbitrary-address lookups by any portal user, with cost and privacy exposure, because the handler `api.js:68-76` passes the path parameter straight through) | | Budget, quota, throttle? | **Absent.** No per-tenant counter, no daily cap, no concurrency limit, no concurrency setting for these functions in either service's `build/template-logic.yaml` (grep for `Concurrency` returns nothing; Inferred: the shared `Custom::Rhythm` function resource may apply defaults not visible here). The only trace of a cost problem is a test comment: `// todo - re-enable once usage limits have been addresses` (`rolodex-organizations/tests/integration/full-contact-entrichment.spec.js:24-25`). | Verified (comment), Absent (controls) | | Public-email-domain filter (cost and correctness guard for company lookups) | Two **different** hand-copied lists. Organizations engine (9): `outlook.com, gmail.com, yahoomail.com, inbox.com, mail.com, aol.com, mindspring.com, yahoo.com, hotmail.com` (`rolodex-organizations/src/logic/full-contact-enrichment-engine.js:5`). Contact-steps (8, **no hotmail.com**): `rolodex-contact-steps/src/logic/domain-logic.js:1`. Neither includes `icloud.com`, `me.com`, `live.com`, `msn.com`, `comcast.net`, `proton.me`, etc. | Verified lists; Inferred consequence: a sign-up from an unlisted consumer domain would call `company.enrich` for the mailbox provider and could create an organization named after it | --- ### 3. Failure handling, retries, rate limiting, caching, cadence #### 3.1 Failure handling - **One error for everything.** Contacts engine: any exception from the vendor call is rethrown as `ProfileNotFound: ` (`:90-94`). Organizations engine: same, plus a 202 is turned into an error `request is being processed - check again later` that is then also reported as `ProfileNotFound` (`:86-95`). Verified. Authentication failure, quota exhaustion (429), timeouts and 5xx are therefore indistinguishable from "no such person" to callers (Inferred from the single catch). - **202 handling is inconsistent.** The organizations engine rejects 202. The contacts engine does not check status; the browser tests `profile.status !== 202` (`contact-social-lookup-web-forms.component.ts:141`). Inferred: if the vendor answers 202 with a body, the contacts engine would cache that placeholder (the `putItem` at `:48` runs on any resolved response) and serve it forever from cache. - **No retries or backoff in 1.0.** Neither engine retries; there is no `backoffParams` passed on the vendor call (the DynamoDB helper has its own backoff, irrelevant to the HTTP call). Verified by reading both files end to end. - **Surfaces.** Console contact form shows four states: checking ("Looking up email..."), success ("Email information found."), not found ("No information found for this email address."), error ("We encountered an error looking up the email address."). Note that the component swallows errors with `catchError(() => of(null))` (`...ts:136`), so the `error` branch is effectively unreachable; every failure reads as "not found" (Verified code; Inferred outcome). Portal: failures are swallowed (`catchError(() => of(null))`) and sign-up continues (`create-and-link-new-contact.component.ts:84-87`). Server org creation: all exceptions are logged and the loop moves to the next domain; no result means `{id: 'notfound'}` (`api.js:89-98`, organizations engine `:151-167`). - **A latent crash path in the organization mapper.** `if (socialProfile.details) { if (socialProfile.details.locations.length > 0)` (`:135-136`) throws when `details` exists without `locations`. The surrounding `try/catch` in `getOrganizationFromDomains` (`:156-165`) logs it and moves on, so a valid company profile would be silently dropped. Verified code; the real-world frequency is unknown. #### 3.2 Rate limiting - 1.0 FullContact: **Absent** (no client-side limiter, no queue, no budget). - 1.0 Google Civic: explicit sequential calls "because of rate limiting at Google" and an error-swallowing wrapper because the API "is both notoriously flaky and has some low throttling levels" (`partners-google-civic/src/logic/representative-logic.js:30-35`). The bulk refresh state machine runs batches with `MaxConcurrency` 1 (`build/stateMachines/refresh-all.step.json`, `RefreshAllRecords`), and the change-queue consumer uses `BatchSize: 1` (`build/template-logic.yaml:67`). Git history: commit `ba7494c` "Efficiency improvement with Google rate limiting" (2024-04-29). Verified. #### 3.3 Caching | | Contacts | Organizations | Evidence | | ----------------- | ---------------------------------------------------------------- | ------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------- | | Store | DynamoDB `rolodex_contacts_social_profiles`, key `email_address` | `rolodex_organizations_social_profiles`, key `domain` | `rolodex-contacts/build/template-resources.yaml:244-262`; `rolodex-organizations/build/template-resources.yaml:274-295` | | Tenant in key? | No | No | same (Verified). Retrofit FND-ROL-005 and ADR-ROL-008 record the same: the profile is "served to every tenant" | | TTL | Attribute `sys_expires_at` declared with TTL **enabled** | same | same; neither engine nor `libraries-aws` `putItem` sets it (`libraries-aws/src/dynamodb-service.js:48-72`) (Verified). Inferred: entries never expire | | Backup | Daily backup tag and point-in-time recovery on both | same | same (Verified); so deleted cache entries persist in backups | | What is stored | The whole vendor payload plus engine additions | whole payload plus `domain` | contacts engine `:36-48`, organizations engine `:43-49` | | Negative caching | None; a failed lookup throws before `putItem` | None | Verified | | Bypass | None | `options.skipCacheCheck` (used by a skipped live test) | organizations engine `:29` | | Cache read errors | Silently swallowed (empty `catch`), falls through to a paid call | traced and swallowed | contacts `:33-34`; organizations `:39-41` | #### 3.4 Re-enrichment cadence - **Never refreshed** (Verified): cache is permanent, `date_social_profile_last_updated` is never read to decide anything, existing contacts are excluded from lookup, and no scheduled job exists. Inferred: stale titles, employers and logos persisted on records until staff edited them by hand, and a wrong cached profile was returned for every tenant forever (the retrofit says the same: "no tenant can refresh it", ADR-ROL-008). --- ### 4. Photo handling - **Pointer, not copy** (Verified). The photo field is `image_url`; the engine sets `image_url = avatar` (contacts engine `:39-41`), the form patches `image_url: profile.avatar` (`...web-forms.component.ts:150`), and the contact is saved with that HTTPS URL. Test and demo fixtures show the shape stored: `image_url: "https://img.fullcontact.com/static/_"` (`rolodex-contacts/tests/e2e/create-and-link-user.spec.js:31`; `awards-ngx-portal-controls/src/app/app.component.ts:36`). - The platform's "copy staged upload to permanent storage" mechanism exists for both `image_url` (contacts) and `logo_url` (organizations) via `fieldsContainingFileUrls` (`rolodex-contacts/config.js:143-145`; `rolodex-organizations/config.js:91-93`), but `libraries-dynamodb/src/s3/file-upload-logic.js:179-185` returns early for anything that does not start with `s3://` ("for now, we let it go"). So a vendor URL **bypasses** the pipeline by design (Verified code; the R2 debt note states the same: "Rhythm 1.0 stored it as a pointer", `R2:services-crm/docs/debt/image-files-and-enrichment.md:8`, and `R2:services-crm/docs/migrations/asset-persistence/README.md:51`: "External URLs are kept as pointers, for example FullContact avatars"). - Staff could replace it with an upload: the form's image control is `rs-uploadable-image` bound to `image_url` (`...component.html:12-13`; logo: `organization-social-lookup-web-forms.component.html:11`). Verified. - Display: contact summary and portal profile render `contact.image_url` directly (`rolodex-ngx-controls/.../contact-summary/contact-summary.component.html:2-6`; `portal/src/app/platform/pages/home/home.component.html:9-11`). Verified. Inferred: the page hotlinks the vendor CDN on every view; if the vendor URL changes or is revoked the image breaks. No broken-image fallback was found. - **Licensing or terms concerns noted anywhere: Absent.** Searched the engines, tests, swagger, templates, retrofit findings and `customer-360` docs for "licen", "terms", "attribution", "copyright" near FullContact: the only related text is the retrofit's open question on whether cross-tenant sharing of provider data is contractually allowed (`FINDINGS.md` FND-ROL-005, "Residual uncertainty", `OD-AREA-ROL-CON-04`). The C360 interface comment "never persist; download + re-host" (section 8) is the only design stance on vendor-hosted images; its stated reason is source-of-truth and durability, not licence. --- ### 5. UI: how enrichment appeared #### 5.1 Staff (console) - **Automatic, inline, on new records only.** There was **no "Enrich" button, no re-run, and no badge** on the contact 360 or the organization pages. Searched `rolodex-ngx-console/src` and `rolodex-ngx-controls/projects` (`.html` and `.ts`) for "enrich" (no hits) and for social/image fields in templates: the only social UI is the lookup component embedded in the New Contact and New Organization web forms. Verified. - Contact form: after email blur the form shows stacked status lines for (a) social information, (b) duplicate email check ("This email address is currently registered to "), (c) organization ("We were able to automatically locate a matching organization." / "checking the internet..." / "Unable to automatically locate a matching organization."), plus an advanced duplicate check on name blur (console only). Verified (`...component.html:25-72,102-111`). - The photo appears as a round avatar in the upload control; the Twitter/Facebook/LinkedIn handles populate their text boxes; the user may edit all of it before saving. Verified (`...ts:145-157`). - Organization form: "Information lookup successful." / "No information found on that domain." (`organization-social-lookup-web-forms.component.html:22-30`). Verified. - Form designers can disable individual behaviors per web form (section 2). - Console Contact view has a **Google Civic** tab, shown only when the tenant integration is enabled (`*ngIf="enableGoogleCivicIntegration"`), with a count of officials in the tab header; it lists officials with photos, parties, addresses, phones, emails, Twitter and Facebook, in three groups (national, state upper, state lower) (`rolodex-ngx-console/src/app/contacts/pages/view-contact/view-contact.component.html:37-45`; `.../google-civic-one-click/google-civic-one-click.component.html`). Verified. #### 5.2 Portal - Sign-up shows a progress message "Gathering your profile information..." while the lookup runs, then continues even if it fails (`create-and-link-new-contact.component.ts:79-84`). The member **never sees or approves** what was pulled; it is written into their new contact record (Verified code; the "never sees" part is Inferred from the absence of any confirmation step in the component). - The portal profile and home pages then show `image_url` (Verified, section 4). Corporate-membership employee add runs the lookup and throws the result away (E, section 1.1). --- ### 6. Known problems (code comments, tests, docs) | # | Problem | Evidence | Label | | ---- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- | | 6.1 | Usage/cost: live FullContact tests are skipped "until usage limits have been addressed"; the whole organization e2e suite is `it.skip` and the contacts e2e suite is commented out | `rolodex-organizations/tests/integration/full-contact-entrichment.spec.js:24-25`; `.../tests/e2e/full-contact-entrichment-e2e.spec.js:6,17,25,36,45`; `rolodex-contacts/tests/e2e/full-contact-entrichment-e2e.spec.js:3-5` | Verified | | 6.2 | Some vendor fields were unreliable in tests: `facebook` and `bio` checks are `it.skip` for a known good address; twitter, LinkedIn and avatar are asserted | `rolodex-contacts/tests/integration/full-contact-entrichment.spec.js:43,59` | Verified skip; reason Inferred (vendor did not return them) | | 6.3 | Cross-tenant, never-expiring, backed-up caches of third-party personal data | section 3.3; retrofit `FINDINGS.md` FND-ROL-005 | Verified | | 6.4 | A literal API key committed (commented out) in the contacts engine | `rolodex-contacts/src/logic/full-contact-enrichment-engine.js:68`; retrofit FND-ROL-035 (confirmed independently; value not reproduced) | Verified | | 6.5 | Any signed-in portal user can request a person lookup for any email | `rolodex-contacts/config.js:227-228` | Verified list; consequence Inferred | | 6.6 | Public-domain filter lists differ and are incomplete (section 2) | `domain-logic.js:1` vs organizations engine `:5` | Verified | | 6.7 | Org mapper dereferences `details.locations.length` without a guard | organizations engine `:135-136` | Verified | | 6.8 | Form mapping bugs: `title:` key patched from `profile.job_title` while `job_title:` is patched from `profile.title`; `linked_in:` falls back to `form.value.linkedin` (wrong key); org address uses `region` server side and `regionCode` client side | `contact-social-lookup-web-forms.component.ts:146,149,153`; organizations engine `:141`; `organization-social-lookup-web-forms.component.ts:123` | Verified code; impact Inferred | | 6.9 | Portal name parsing passes an undefined variable: `nameParser.parseFullName(name)` where only `contact.name` exists (in a browser `name` is `window.name`) | `create-and-link-new-contact.component.ts:117`; `user-contact-association.component.ts:161` | Verified code; impact Inferred | | 6.10 | Hard-coded fake profile for the developer's own address in the legacy console | `console/src/app/rolodex/controls/contact-form/contact-form.component.ts:~478-487` | Verified | | 6.11 | Wrong matches or privacy complaints | Searched code, tests, docs and retrofit: **Absent** (no comment records a wrong-match incident or a complaint). Absence of evidence only. | Absent | | 6.12 | Git history for the engines is flattened (a template sweep and one unrelated 2026 fix), so no commit explains the 1.0 design decisions | `git log --follow` on both engine files | Verified | --- ### 7. Google Civic districts (second enrichment example) Repo: `partners-google-civic` (its own microservice, created 2024-04 under ticket RB-3889; 27 commits). All Verified. | Aspect | Detail | Evidence | | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | Gate | Per-tenant: `integrations.google_civic` on the tenant permission profile, cached 15 minutes in process | `src/services/tenant-permission-profiles-service.js:31-38`; `config.js:16` (`cacheLifespan: 1000 * 60 * 15`) | | Trigger 1 (continuous) | Change stream of `rolodex-contacts` (SNS to SQS, filter: `Microservice = rolodex-contacts`, `EventName` INSERT or MODIFY); queue `partners-google-civic-record-changes`, redrive to a DLQ after 3 receives, `BatchSize: 1` | `build/template-resources.yaml:119-126,156-181`; `build/template-logic.yaml:62-70` | | Trigger 2 (backfill) | A change to the tenant's permission profile that turns `google_civic` on (unless `notes` = `NOREBUILD`) starts the state machine `partners-google-civic-export-refresh-all` which exports the tenant's contacts and re-sends them as synthetic INSERT messages; also re-runs after a full DynamoDB rebuild | `src/lambda/events.js:44-57,59-82`; `src/logic/refresh-logic.js:57-99`; `build/template-logic.yaml:82-125` | | Skip rules | Integration disabled; no `home_address`; non-US on both old and new address; **no change to `home_address`** (`util.diff`), which also stops the write-back from looping | `src/lambda/events.js:19-39` | | Input | The contact's **home address only**: line1, line2, line3, city, state, postal code concatenated to one string; query includes `includeOffices: false`, `levels: ['country','administrativeArea1']` | `src/services/google-civic-api.js:42-75`; `src/logic/process-update.js:22` (requires line1, city and state) | | API | `GET https://www.googleapis.com/civicinfo/v2/representatives?...` with `X-Goog-Api-Key` header; the key in SSM `/google/civic/api`, cached 15 minutes in process; a second call per district to list officials | `src/services/google-civic-api.js:25,73,77-80` | | Parsing | Regex over OCD division ids: US congressional `cd`, state upper `sldu`, state lower `sldl` (also via `alsoKnownAs`) | `src/logic/parse-google-civic-response.js:1-55` | | Output (stored) | Six fields **on the contact**: `national_district_ocdId`, `national_district`, `state_upper_district_ocdId`, `state_upper_district`, `state_lower_district_ocdId`, `state_lower_district` | `config.js:8-15`; schema `rolodex-contacts/src/schema/object.schema.js:84-90`; 2.0 table `R2:services-crm/db/migrations/0001_contacts.sql:168-173` | | Write path | Fetch a clean copy of the contact through the contacts SDK, set or delete each of the six fields, call `updateContact` (a full-object write; Inferred: a lost-update race with a concurrent user edit is possible). If the address is removed or non-US, the six fields are cleared | `src/logic/process-update.js:27-37` | | Output (not stored) | Officials (name, party, phones, URLs, photo URL, emails, Twitter, Facebook, addresses) fetched **live on demand** per ocdId for the console contact page; sequential calls; any error is swallowed and an empty result returned | `src/logic/representative-logic.js:13-60`; `src/logic/parse-google-civic-response.js:57-84`; console `.../show-google-civic-official.component.html` | | Provenance | None on the contact (no "updated at", no source stamp) | `object.schema.js` (Verified absent) | | Tests | Unit for parsing and events; integration for `processUpdate` against the live Google API using real addresses; end-to-end contact lifecycle with a 10 s wait; manual specs for "no response" and "rerun failed" | `tests/` (Verified) | | 2.0 status | Fields exist on the 2.0 contact table; nothing fills them; Civic was **not ported**. Region assignment, a different enrichment, was ported on 2026-10-08 | `R2:services-crm/docs/debt/image-files-and-enrichment.md:1-8` | Observations for the proposal: - **Inferred (external knowledge, not from the repo; verify before relying)**: Google announced the Civic Information Representatives API would be turned down in 2025. If still accurate, the district lookup would need a different source (the districts could instead come from a geocoder plus boundary dataset). The code comments already call the API flaky. - The pattern worth keeping: a change-stream consumer scoped to one field's diff, a per-tenant integration flag, a bulk refresh that re-feeds the same consumer, and clearing derived data when the input is removed. --- ### 8. customer-360 precedent (2025/26 internal tool) Repo: `customer-360-commitments-dashboard/api` (TypeScript, Postgres via Drizzle, Neon; Lambda and CLI). Verified throughout. It enriches **images only** (person photo, company logo), not social profiles. #### 8.1 Provider abstraction - Domain-owned interfaces; vendor DTOs never leave the provider class: "FullContact-specific DTOs stay inside the provider implementation" (`src/contacts/contact-image-enrichment.ts:1-4`) and "Implementations must not leak vendor DTOs into the synchronization service" (`src/customers/customer-logo-enrichment.ts:1-4`). The full interfaces are quoted in Appendix A.8; the retry loop and acceptance gates are cited by line in 8.1-8.5 rather than quoted. - Two small providers, each taking an identity and returning a typed outcome: `findContactImage(identity)` and `findCustomerLogo(identity)` (`src/fullcontact/contact-image-enrichment-provider.ts:17`, `company-logo-enrichment-provider.ts:19`). Injected `fetch` and `sleep` for tests. - Outcomes (not exceptions) for expected non-results: person `found | not_found | rejected_match | no_photo`; company `found | not_found | no_logo | rejected_match | invalid_domain`. Exceptions only for auth, config, timeouts, network and non-retryable 4xx. Each result carries `provider: "fullcontact"`, a confidence and a `providerRef` (vendor `updated` timestamp or `personId`/`companyId`). - "Only extracts photo/avatar + confidence ... Unrelated personal data from the response is discarded and never logged in full" (`contact-image-enrichment-provider.ts:3-5`): data minimization is a stated rule. #### 8.2 What is sent (matching inputs) - Person (`POST /v3/person.enrich`): `email` (trimmed, lowercased) **plus optional** `name{given,family,full}`, `organization` (company name) and `website` (company domain as `https://...`) "for match confidence" (`contact-image-enrichment-provider.ts:50-71`). This is richer than 1.0, which sent email only. - Company (`POST /v3/company.enrich`): `{ domain }` only, after normalization (strip protocol, path, port, `www.`; lowercase; reject localhost, internal TLDs, IPs, and 24 generic mailbox domains; an email contributes only its host) (`src/customers/company-domain.ts:13-101`). Domain precedence: HubSpot company domain, then website, then local primary domain, then local website; "Does not invent domains from contacts" (`:103-126`). #### 8.3 Match acceptance (identity rules) - **Confidence gate**: configurable `FULL_CONTACT_MIN_CONFIDENCE` (default 0.5); accepted values include numbers and the words high/medium/low mapped to 0.9/0.7/0.3; below the threshold the outcome is `rejected_match` and nothing is stored (`src/fullcontact/config.ts:11-12`; provider `:100-110,209-226`). A missing confidence is accepted (Inferred consequence: if the vendor omits a numeric confidence the gate is inert; I could not confirm what v3 returns). - **Name sanity check for companies**: if the local company name and the vendor `name` do not contain one another after normalization (lowercase, NFKD, strip punctuation) the match is `rejected_match` (`company-logo-enrichment-provider.ts:96-109,264-276`). - **Person identity resolution** (`src/people/person-resolution.ts`, documented in `docs/people.domain.md`): - one canonical person per human; match key is `normalized_email` (partial-unique index); - **role/shared-inbox addresses never match or merge**: `support, info, sales, billing, admin, help, contact, hello, noreply, no-reply, donotreply, office, team, accounts, ar, ap` (configurable) get `normalized_email = NULL` (`src/people/identity-config.ts:26-43`; `person-resolution.ts:136-139,189-191`); - already-linked rows with unmatchable emails are never re-resolved, otherwise a new person would be minted each run (`person-resolution.ts:117-126`); - existing non-blank person fields win; source data only fills blanks (`:216-244`); unique-violation on a concurrent insert re-reads the winner (`:152-173`); - internal staff classified only by a configured domain list (`INTERNAL_EMAIL_DOMAINS`, default `rhythmsoftware.com`) (`identity-config.ts:23,75-82`). - **Personal vs work email**: there is no personal/work distinction for person matching (email is the key whichever kind it is). For **company** lookups the distinction is enforced by the generic-provider denylist; for people the company is only a hint. Verified. - Name-plus-company matching is **not** used as an identity key anywhere; name and company are only hints sent to the vendor. #### 8.4 Caching, TTL and re-enrichment - **No result cache and no TTL.** Plan §13 states "No result caching beyond the status field is recommended initially" (`docs/implementation-reports/person-identity-refactor-plan.md:562`). - A persisted **lookup status** on each record is the cache of the outcome: `NULL` never attempted; `resolved`; `not_found` (terminal "prevents repeated paid lookups" unless `--force`); `failed` (retryable); `ineligible` (no usable identity); legacy `succeeded`, `no_email`, `no_match` still readable (`src/images/image-enrichment-state.ts:10-62`; columns `people.image_enrichment_status/_attempted_at`, `customers.logo_enrichment_status/_attempted_at/_error`; `docs/customers.domain.md` "Customer logo lifecycle"). - Eligibility function `shouldAttemptImageLookup`: skip if a usable image already exists; no identity means retry only when never attempted or failed; terminal statuses block (`image-enrichment-state.ts:49-62`). - Cadence: person enrichment is declared **create-only** ("only when a person is first created without a source image — never during normal updates", `src/people/person-image-enrichment.ts:5-11`), but the HubSpot contact sync also runs `ContactImageSynchronizationService` for any person without an image and without a terminal status; the 2026-07-16 fix report explains why (a later-arriving email could never trigger enrichment otherwise) (`reports/customer-contact-image-resolution.md:14-20,40-55`). Companies run from the company sync for inserts and for any customer still lacking a logo (`src/hubspot/sync-companies.ts:296-330`). A terminal miss is never retried automatically, so there is **no periodic refresh** (Inferred: new photos that appear later at the vendor are never picked up). #### 8.5 Rate limiting, retries, timeouts - Timeout 30 s (`FULL_CONTACT_TIMEOUT_MS`), `maxRetries` 2 (`FULL_CONTACT_MAX_RETRIES`), exponential backoff `250ms * 2^(attempt-1)`; retry on 429, 5xx, network error and abort; no `Retry-After` handling (`src/fullcontact/config.ts:9-10`; provider `:137-189`). Verified. - No client-side rate limiter or budget. Batch tool `images:backfill` uses a fixed worker pool with default **concurrency 3**, `--limit`, `--dry-run`, `--retry-failed`, `--force`, `--include-not-found` (`scripts/images-backfill.ts:76,225-243`; report `reports/customer-contact-image-resolution.md:123-141`). Spend is visible as a counter `fullContactRequests`. - Observed yield: on 2026-07-16 the backfill made **890 FullContact requests** for 959 attempts and produced **29 company logos and 0 new contact photos** ("FullContact mostly returned `not_found` / `no_photo` for those emails") (`reports/customer-contact-image-resolution.md:101-121`). This is the only hit-rate datapoint in any of the investigated material. #### 8.6 Error taxonomy - `FullContactConfigError` (HTTP 500, code `FULLCONTACT_CONFIG_ERROR`); `FullContactApiError` carrying `fullContactStatus`, `retryable`, and a code among `FULLCONTACT_API_ERROR`, `FULLCONTACT_AUTH_ERROR` (401 and 403, never retried), `FULLCONTACT_TIMEOUT` (408), `FULLCONTACT_NETWORK` (502) (`src/fullcontact/errors.ts:6-31`; provider `:83-98,168-184`). 404 is a normal outcome `not_found`. - Sync-level statuses (contacts): `existing_hubspot_image, enriched_and_uploaded, person_unresolved, no_email, not_found, rejected_match, skipped_unconfigured, failed`; (companies): `existing_hubspot_logo, enriched_and_uploaded, no_domain, not_found, rejected_match, skipped_unconfigured, failed`. Mapped to the persisted lifecycle by `mapContactImageSyncStatusToEnrichment` and `mapCustomerLogoSyncStatusToEnrichment`; `skipped_unconfigured` is deliberately **not** recorded as an attempt, so adding the key later retries (`image-enrichment-state.ts:64-105`; report `:84-85`). - Gap, Inferred: HTTP 202 ("queued") is not specially handled by either provider; `response.ok` is true for 202, so the payload is parsed and, with no photo, classified `no_photo`, which is then persisted as the terminal `not_found`. I did not verify whether the vendor returns 202 for these calls; the 1.0 UI shows its authors expected it. #### 8.7 Image storage and safety - **Never persist the vendor URL**: "Provider photo URL — never persist this; download + re-host" (`contact-image-enrichment.ts:19`); the logo doc: "Local DB stores the HubSpot-hosted URL, never the FullContact URL" (`docs/customer-logo-sync.md`, "Behavior notes"). - Pipeline: download, validate, upload to HubSpot Files, write the HubSpot URL to the native `photo` property (companies: a custom property `customer_360_company_logo_url`), then mirror on the person or customer; recovery is idempotent (next run reads HubSpot's photo) (`src/contacts/contact-image-synchronization-service.ts:11-44,89-201`). - Downloader: raster allowlist (jpeg, jpg, png, webp, gif; **SVG rejected** for lack of a sanitizer), 5 MB cap, empty-body check, content sniffing for HTML or JSON masquerading as an image, SSRF guard on the initial URL and every redirect hop (blocks localhost, link-local, private ranges, metadata endpoints), 5 redirect hops maximum, 30 s timeout (`src/images/external-image-downloader.ts:14-197`; `src/images/url-safety.ts:26-60`). - **Inconsistency to note**: the create-time person path (`PersonImageEnrichmentService.enrichNewPersonImage`) writes the vendor `imageUrl` straight to `people.image_url` with `imageSource = FullContact` (`src/people/person-image-enrichment.ts:96-112`), contradicting the interface rule above. HubSpot contact sync avoids it with `skipCreateTimeImageEnrichment: true` (`src/hubspot/sync-contacts.ts:177`), and `docs/people.domain.md` says "Other source systems retain create-time `PersonImageEnrichmentService` behavior". So Zendesk and Asana people can hold vendor-hosted URLs. Verified. #### 8.8 Precedence and failure isolation - Source image always wins (HubSpot photo, Zendesk photo, Asana avatar); existing HubSpot image means FullContact is **never called**; a null never replaces a valid image; FullContact is a fallback only (`docs/customer-logo-sync.md`; `contact-image-synchronization-service.ts:236-281`). - Enrichment "never throws" and never blocks sync (`person-image-enrichment.ts:12`; `sync-contacts.ts:768-795`; `sync-companies.ts:296-330`). Missing key is a silent no-op, status stays `NULL` (`person-image-enrichment.ts:13-14`). - Credentials: one env var `FULL_CONTACT_API_KEY`; "Never log the key value"; the key is **not provisioned to AWS**, so FullContact currently runs only from local scripts (`docs/aws-deployment.md:44-47`; `config.ts:1-5`). Inferred: in deployed Lambdas enrichment is skipped as `skipped_unconfigured`. - Logging: entity ids, outcome, confidence, domain host, no raw email, no signed URL query strings (`url-safety.ts:1-24`; report `:87-93`). #### 8.9 What the tool says it learned (docs) - Before the 2026-07 fix, FullContact "re-attempts every run" with no guard (`person-identity-refactor-plan.md:188`), image sync was never invoked from contact sync, and a missing key could block mirroring an existing HubSpot image (`reports/customer-contact-image-resolution.md:16-29`). - The fix: explicit status, terminal vs retryable, `--force`, HubSpot-first precedence, and a UI avatar that falls back to initials without re-requesting a broken URL (`:142-148`). - Operational follow-ups noted: provider rate limits and invalid HubSpot URLs to monitor; the observability plan lists the sync as "manual CLI / ad-hoc / no run history" (`docs/architecture/operations-observability-plan.md:224`). --- ### 9. Lessons for Rhythm 2.0 (Inferred) #### Keep 1. **Provider interface owned by the domain**, outcomes as values (`found`, `not_found`, `rejected_match`, `no_data`, `ineligible`) and exceptions only for auth/config/transport. (C360 8.1.) 2. **Lookup state on the record** (`NULL`, `resolved`, `not_found` terminal, `failed` retryable, `ineligible`), with `attempted_at`, so misses are not re-bought. 1.0 had no negative cache and paid again. (C360 8.4; 1.0 3.3.) Add a **re-check horizon** for `not_found` (neither tool has one) so a vendor that later learns a person is picked up. 3. **Source-wins, fill-blanks precedence**, and never replace a valid value with null. 1.0 let vendor values overwrite typed values and, in the portal, wipe the identity provider's picture. (C360 8.8; 1.0 1.6.) 4. **Minimum-confidence gate plus an identity sanity check** (company name vs vendor name), and **send every identity hint you have** (name, company, domain), not just email. 1.0 sent email alone. (C360 8.2, 8.3.) 5. **Role-address and generic-mailbox denylists as one shared, configurable list** reused by person matching, company lookups and portal organization creation. 1.0 had two diverging lists missing iCloud, Live and MSN. (C360 8.2-8.3; 1.0 section 2.) 6. **Copy and re-host vendor images** through the platform asset pipeline, with the C360 downloader's guards (type allowlist, size cap, SSRF, no SVG). 1.0's pointer approach left hotlinks with no licence statement. (C360 8.7; 1.0 section 4; the 2.0 debt note already says the avatar URL "is imported through the same pipeline".) 7. **Non-blocking enrichment**: a primary write never fails because enrichment failed; failures land in status plus log. (Both tools.) 8. **Trigger design from Google Civic**: change-stream consumer scoped to the diff of the input field (so write-back cannot loop), per-tenant integration flag, a bulk "re-feed everything" path, clear derived data when the input is removed. (1.0 section 7.) 9. **Typed errors and bounded retry with exponential backoff and a timeout**; add `Retry-After` handling, which C360 lacks. 10. **Observability counters** per run: attempted, requests to vendor, resolved, not found, ineligible, failed. (C360 8.5.) This is the only way to see the true yield (29 logos and 0 photos from 890 calls). #### Avoid 1. **Tenant-less caches** of personal data keyed by email. If a shared cache is wanted for company profiles (public data), key and label it explicitly as shared and give it a TTL that is actually set; keep person-level data per tenant, expirable and deletable. (1.0 3.3; retrofit FND-ROL-005.) 2. **A TTL attribute that nothing writes.** Verify expiry end to end in a test. 3. **Browser-orchestrated enrichment** that returns the raw vendor payload to the client and lets each form decide what to write. It produced inconsistent mappings (section 6.8), duplicated paid calls across surfaces, and let any portal user call the vendor for any address. Do enrichment server side, return only the proposed, typed fields, and authorize it to the caller's own record or to staff. 4. **No per-tenant opt-in or budget.** Make enrichment a tenant-enabled integration (as Civic was), with a per-tenant spend cap and counters. 5. **Silent automatic population of member-facing records.** The portal wrote vendor data into a member's contact with no confirmation. Prefer "suggested values with source and time" that staff or the member accepts, or at minimum stamp provenance per field (1.0 had a single timestamp, contact only). 6. **Collapsing every vendor failure into "not found".** Distinguish auth, quota, transport, no-match, and in-progress (202). 7. **Putting the vendor key in source (even commented out)**, and one global key with no way to attribute spend per tenant. Use a secret with rotation, and tag calls by tenant for cost. 8. **A hand-rolled public-domain list per call site.** (see Keep 5) 9. **Persisting the vendor URL on any path** (C360's create-time person path still does). Pick one rule and enforce it in the service, not by caller discipline. 10. **Assuming the vendor API is stable.** Google Civic's flakiness is documented in a code comment; the FullContact 202 contract was handled three different ways across the three code bases. #### Questions the investigation could not answer (for the proposal's open decisions) - Actual 1.0 hit rate, spend and call volume per tenant: **Absent** (no metrics in the repos; C360's 890-call sample is the only datapoint). - Whether the FullContact contract permits sharing one tenant's lookup results with another tenant, or storing/hotlinking vendor images: **Absent** from code and docs; the retrofit flags it as an open product question. - Whether v3 returns a numeric confidence for these calls (the C360 gate assumes it may): not determinable from code. - Whether Google's Representatives API is still available: external, not in the repo. --- ### Appendix A. Quoted code #### A.1 Organizations engine, cache-then-call (`rolodex-organizations/src/logic/full-contact-enrichment-engine.js:19-53`) ```js async function retrieveSocialProfile(options, securityContext) { const external = module.exports.dependencies(); const logger = util.log.getLogger('logic', securityContext); const { domain } = options; if (!domain) throw new Error('No value was provided for parameter: domain'); // first, check dynamodb!! let profile; if (!options.skipCacheCheck) try { profile = await external.dynamodbService.getItem(dynamoTableName, { domain: domain.toLowerCase() }, null, null, securityContext); profile = profile.Item; if (profile) { logger.info(`Successfully located profile for ${domain} in dynamodb`); profile.cached = true; return profile; } } catch (err) { logger.trace(err); } profile = await executeFullContactApiCall(domain, securityContext); if (profile) { profile.domain = domain.toLowerCase(); // save it for later await external.dynamodbService.putItem(dynamoTableName, { domain: profile.domain }, profile, null, securityContext); } return profile; } ``` Note: the put carries no `sys_expires_at`; the lookup carries no tenant. #### A.2 Organizations engine, the vendor call and its error collapse (`:55-99`) ```js async function executeFullContactApiCall(domain, securityContext) { const external = module.exports.dependencies(); const logger = util.log.getLogger('logic', securityContext); if (public_email_domains.includes(domain.toLowerCase())) throw new Error('IllegalOperation: Unable to locate profile for public email hosting domain'); const fullContactApiKeySetting = await external.getParameter('/fullcontact/api-key', true); if (!fullContactApiKeySetting) throw new Error('No API key found'); let apiKey = fullContactApiKeySetting.Parameter.Value; let options = { url: 'https://api.fullcontact.com/v3/company.enrich', method: 'POST', headers: { Authorization: 'Bearer ' + apiKey, }, data: { domain }, }; logger.info('Looking up FullContact API for domain: ' + domain); let profile; try { let response = await external.axios(options); // if there's a 202, we didn't get any data back! if (response.status === 202) throw new Error('request is being processed - check again later'); profile = response.data; } catch (err) { logger.error('Request for ' + domain + ' failed with response:' + err.message); throw new Error('ProfileNotFound: ' + err.message); } return profile; } ``` #### A.3 Organizations engine, profile to organization (`:101-149`, trimmed to the fields written) ```js function convertToOrganization(socialProfile, domain) { if (!socialProfile || !socialProfile.name) { return; } const organization = { name: socialProfile.name, domain_name: domain.domain_name, }; if (socialProfile.logo) { organization.logo_url = socialProfile.logo; } if (socialProfile.facebook) { organization.facebook = socialProfile.facebook; } if (socialProfile.linkedin) { organization.linked_in = socialProfile.linkedin; } if (socialProfile.twitter) { organization.twitter = socialProfile.twitter; } if (socialProfile.website) { organization.website = socialProfile.website; } if (socialProfile.bio) { organization.description = socialProfile.bio; } if (socialProfile.details) { if (socialProfile.details.locations.length > 0) { organization.address = { line1: socialProfile.details.locations[0].addressLine1, line2: socialProfile.details.locations[0].addressLine2, city: socialProfile.details.locations[0].city, state: socialProfile.details.locations[0].region, postal_code: socialProfile.details.locations[0].postalCode, country: socialProfile.details.locations[0].countryCode, }; } } return organization; } ``` (The single-line `if`s are condensed from the original multi-line blocks; behavior unchanged.) #### A.4 Contacts engine, cache, post-processing, request (`rolodex-contacts/src/logic/full-contact-enrichment-engine.js:18-97`, commented-out key line omitted) ```js async function retrieveSocialProfile(emailAddress, securityContext) { const external = module.exports.dependencies(); const logger = external.getLogger(securityContext); let profile; try { profile = await external.dynamodbService.getItem(dynamoTableName, { email_address: emailAddress }, null, null, securityContext); profile = profile.Item; if (profile) { logger.info(`Successfully located profile for ${emailAddress} in dynamodb`); return profile; } } catch (err) {} profile = await executeFullContactApiCall(emailAddress, securityContext); profile.email_address = emailAddress; if (!profile.image_url) { profile.image_url = profile.avatar; } if (profile.gender) { profile.gender = profile.gender.toLowerCase(); } // save it for later await external.dynamodbService.putItem(dynamoTableName, { email_address: emailAddress }, profile, null, securityContext); return profile; } ``` ```js let options = { url: 'https://api.fullcontact.com/v3/person.enrich', method: 'POST', headers: { Authorization: 'Bearer ' + apiKey }, data: {email: emailAddress}, }; ... try { let response = await external.axios(options); fullContactProfile = response.data; } catch (err) { logger.info('Request for ' + emailAddress + ' failed with response:' + JSON.stringify(err.message)); throw new Error('ProfileNotFound: ' + err.message ); } return fullContactProfile; ``` #### A.5 Console form, profile to contact (`rolodex-ngx-controls/.../contact-social-lookup-web-forms.component.ts:126-157`) ```ts if (this.contact?.sys_version) { console.log('Existing contact... declining to perform social lookup.'); return; } // we're going to do a social profile lookup this.socialNetworkCheckupStatus = 'checking'; this.contactsService.getSocialProfile(this.ss.CurrentTenantId, email) .pipe( catchError(() => of(null)) ).subscribe(profile => { if (profile && profile.status !== 202) { this.socialNetworkCheckupStatus = 'success'; this.parentForm.patchValue({ title: profile.job_title || this.parentForm.value.job_title, notes: profile.bio || this.parentForm.value.notes, gender: profile.gender ? profile.gender.toLowerCase() : this.parentForm.value.gender, job_title: profile.title || this.parentForm.value.job_title, image_url: profile.avatar || this.parentForm.value.image_url, twitter: profile.twitter || this.parentForm.value.twitter, facebook: profile.facebook || this.parentForm.value.facebook, linked_in: profile.linkedin || this.parentForm.value.linkedin, date_social_profile_last_updated: this.dateTimeService.Now(), name: profile.fullName || this.parentForm.value.name }); } ``` #### A.6 Platform file-move logic that leaves vendor URLs alone (`libraries-dynamodb/src/s3/file-upload-logic.js:179-185`) ```js if (!fileUrl.startsWith('s3://')) { if (!fileUrl.startsWith('service://')) //throw new Error(`ObjectInvalid: Invalid file specified for ${property}: ${fileUrl}`); return; // for now, we let it go return; } ``` #### A.7 Google Civic, the stream gate and the write-back (`partners-google-civic/src/lambda/events.js:13-42`; `src/logic/process-update.js:16-41`) ```js module.exports.processRecordChanges = util.events.handleQueueMessages( async (event, context, record, messageAttributes, logger, securityContext) => { const external = module.exports.dependencies(); const isActive = await external.isGoogleCivicEnabled(context, securityContext); if (!isActive) { logger.addFieldToContext(context, 'mode', 'disabled'); return; } if (!record.newObject?.home_address && !record.oldObject?.home_address) { logger.addFieldToContext(context, 'mode', 'no-address'); return; } if ((record.newObject?.home_address?.country || 'US') !== 'US' && (record.oldObject?.home_address?.country || 'US') !== 'US') { logger.addFieldToContext(context, 'mode', 'international'); return; } const diff = util.diff(record.newObject.home_address, record.oldObject?.home_address); if (diff?.length) { await external.processUpdate(context, securityContext, record.newObject); } else { logger.addFieldToContext(context, 'mode', 'no-diff'); } }, ); ``` ```js async function processUpdate(context, securityContext, contact) { let updatedDistricts = {}; if ( contact.home_address?.line1 && contact.home_address?.city && contact.home_address?.state && (contact.home_address?.country || 'US') === 'US' ) { const googleResponse = await external.searchAddress(contact.home_address); updatedDistricts = external.parseDistricts(googleResponse); } if (config.districtFields.find((f) => contact[f] !== updatedDistricts[f])) { //Get a clean version let cleanContact = await external.getContact(context, securityContext, contact.id); config.districtFields.forEach((f) => { cleanContact[f] = updatedDistricts[f]; if (!cleanContact[f]) { delete cleanContact[f]; } }); cleanContact = await external.updateContact(context, securityContext, cleanContact); } } ``` (Both are condensed from the original (logging lines and the `external` constants removed); logic unchanged.) #### A.8 customer-360 provider interfaces (`C360 src/contacts/contact-image-enrichment.ts:5-31`, `src/customers/customer-logo-enrichment.ts:5-27`) ```ts export interface ContactImageIdentity { email: string; firstName?: string | null; lastName?: string | null; fullName?: string | null; companyName?: string | null; companyDomain?: string | null; } export type ContactImageEnrichmentOutcome = 'found' | 'not_found' | 'rejected_match' | 'no_photo'; export interface ContactImageEnrichmentResult { outcome: ContactImageEnrichmentOutcome; /** Provider photo URL — never persist this; download + re-host on HubSpot. */ imageUrl: string | null; confidence: number | null; provider: 'fullcontact'; /** Safe trace id / updated stamp from provider when available. */ providerRef: string | null; } export interface ContactImageEnrichmentProvider { findContactImage(identity: ContactImageIdentity): Promise; } ``` ```ts export interface CustomerLogoIdentity { domain: string; companyName?: string | null; } export type CustomerLogoEnrichmentOutcome = 'found' | 'not_found' | 'no_logo' | 'rejected_match' | 'invalid_domain'; export interface CustomerLogoEnrichmentResult { outcome: CustomerLogoEnrichmentOutcome; /** Provider-hosted logo URL — never persist; re-host on HubSpot. */ logoUrl: string | null; confidence: number | null; provider: 'fullcontact'; providerRef: string | null; matchedName?: string | null; } export interface CustomerLogoEnrichmentProvider { findCustomerLogo(identity: CustomerLogoIdentity): Promise; } ``` ### Appendix B. Search log and Absent items | Searched | Result | | ---------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `grep -rIil "fullcontact\|full-contact"` across `/Users/andrew/rhythm` (excluding `node_modules`, `.git`) | Only: the two engines and their tests/templates; generated API clients; `customer-360*`; one demo constant in `awards-ngx-portal-controls`. | | Stream or queue handlers calling enrichment in `rolodex-contacts`, `rolodex-organizations`, `rolodex-eventbridge`, `libraries-rolodex-*` | None. | | `.csi`/spec-retrofit docs for FullContact | Present under `.csi-worktrees/docs-contract-draft/specs/apps/rolodex/` (used as secondary: FND-ROL-005, FND-ROL-035, ADR-ROL-008). | | Per-tenant FullContact flag, budget, counter | None. | | "Enrich" button, "refresh profile" action, enrichment badge in console or portal | None. | | Licence or terms notes about vendor photos | None. | | Wrong-match or privacy complaint records | None. | | 2.0 enrichment code (`R2:services-crm`, `services-platform`, `services-rolodex`) | None. Fields exist (`image_url`, `logo_url`, `twitter`, `facebook`, `linked_in`, `instagram`, `gender`, `job_title`, `notes`, six district fields, `date_social_profile_last_updated`) and nothing fills them: `R2:services-crm/docs/debt/image-files-and-enrichment.md`, `R2:services-crm/db/migrations/0001_contacts.sql:168-186`, `0003_organizations.sql:69-76`. | ## 23. W1 Evidence: How association management systems (and the nearest nonprofit CRMs) handle data enrichment and data quality ## W1 Evidence: How association management systems (and the nearest nonprofit CRMs) handle data enrichment and data quality Research date: 2026-10-22 (all "accessed" dates below are 2026-10-22 unless stated). Prepared for: "Rhythm 2.0 Data Enrichment" architecture proposal. Scope: AMS platforms (iMIS, GrowthZone/ChamberMaster, MemberSuite, netFORUM, Fonteva, Nimble AMS, Personify360, MemberClicks, YourMembership, Wild Apricot, Aptify, Impexium, Novi, Glue Up), constituent platforms (Tessitura, Blackbaud Raiser's Edge NXT, Bloomerang, Neon One) and the Salesforce ecosystem that Fonteva and Nimble AMS sit on. --- ### 0. How to read this report #### 0.1 Evidence labels Every claim carries one of these labels. - **Verified**: stated in a vendor help page, developer doc, release note or other primary documentation that I fetched and read (through the fetch tool) during this session. It is about what the product does, not about how well it works. - **Reported**: stated by a third party (forum thread, customer, consultant, directory, news article, a search-result summary of a page I did not open myself). Treat as plausible, not confirmed. - **Marketing**: a vendor claim on a product or press page with no supporting operational documentation (no setup steps, no limits, no behaviour description). - **Unavailable**: I searched or tried to fetch and found nothing, or the page was blocked. This is not proof the feature does not exist. #### 0.2 Method limits (read this before relying on any negative finding) 1. The fetch tool answers questions about a page through a small summarising model. Quotes below are short and taken from its output; where the summary was thin I say so. Nothing here was read as raw HTML. 2. The shared web-search budget for this session ran out before I could chase every lead. The "pages not fetched" list in section 6 names the leads left open. Negative findings for Impexium, Glue Up, YourMembership, MemberSuite, Fonteva, Raklet, Higher Logic and Neon One are therefore weak ("Unavailable"), not strong. 3. Several vendor knowledge bases blocked fetching (GrowthZone Zendesk help centre returned 403; Aptify Confluence requires login; Blackbaud and Neon community and academy pages returned 403 or empty JavaScript shells). Where I lean on a search-result summary of such a page, the claim is labelled **Reported**, not Verified. 4. Documentation versions drift. The iMIS duplicate documentation I fetched exists in at least two versions that disagree (section 2.1). Always check which version a claim belongs to. #### 0.3 Corrections to the brief's premises (ownership as of 2026-10-22) - **Community Brands is now Momentive Software** (rebrand after TA Associates agreed to buy its Association & Events and Nonprofit divisions, July 2024). Reported (BusinessWire, NonProfit Times via search results). Momentive's own press page for the January 2026 acquisition of Personify is Verified. - **Momentive Software acquired Personify on 2026-01-06.** Verified (Momentive press release, fetched). Personify is described as AMS, events, job board and engagement; the release does not name Personify360, MemberClicks or Wild Apricot. Momentive's site navigation lists MemberClicks and WildApricot among its products (Verified from the release page's nav, so inferred as post-acquisition). A Wild Apricot wishlist moderator in 2016 signed "Wild Apricot by Personify" (Verified, forum page fetched). - **MemberSuite is owned by GrowthZone, not Community Brands.** Verified: GrowthZone's press release of 2023-09-21 says MemberSuite is rebranded "MemberSuite by GrowthZone". GrowthZone's own site (2026) lists three AMS lines: MemberSuite AMS, GrowthZone AMS, ChamberMaster. GrowthZone was bought by Lead Edge Capital from Greenridge in 2023 (Reported, search result; no later ownership change found). - **Nimble AMS** (Salesforce-native) came to Community Brands via the 2017 NimbleUser acquisition (Reported) and is now a Momentive product. - **NetForum** is a Momentive (ex-Community Brands) product (Momentive site navigation, Verified). The documentation I could fetch is old (netFORUM Enterprise 2015.1 and 2017.1, hosted at documentation.mip.com). - **Impexium** was acquired by Billhighway in November 2020 and its suite is branded re:Members (Reported, Preqin and CB Insights via search results). I could not confirm any 2025 rebrand. - **Fonteva** ownership: Unavailable. Its own site (2026) says founded 2010, Salesforce-native, Salesforce ISV and Summit partner. --- ### 1. Executive summary 1. **No AMS in this study documents a native, continuous firmographic enrichment engine** (employer, title, organisation attributes) that a customer can turn on. Verified for iMIS, GrowthZone, netFORUM, Nimble AMS, Personify, Wild Apricot, MemberClicks, Aptify (in the fetched documentation). Unavailable for the rest. The one marketing exception is Momentive's MomentiveIQ ("Member data cleansing and enrichment is built into MomentiveIQ Agentic Workers"), which has no operational documentation I could find (Marketing). 2. **What the market does have is postal-data hygiene** (address standardisation, NCOA movers, deceased flags, Do Not Call), delivered through **bureau or partner services** rather than built by the AMS. Verified examples: Blackbaud Raiser's Edge NXT (Address Finder and Deceased Finder, every 90 days), Bloomerang (nightly NCOA through TrueGivers), Tessitura (Loqate at about $1 per 1,000 records per service; Experian real-time), netFORUM and Aptify (plug-in connectors to Melissa and others, manual or on-entry). 3. **Email, phone and demographic append lives in nonprofit-CRM services bureaus** (Blackbaud EmailFinder, PhoneFinder, CellPhoneFinder, AgeFinder, GenderFinder; Bloomerang's TrueGivers demographic fields and DonorSearch/Dataro). These are batch or scheduled, mostly opaque on data sources and match rates, and carry little published consent or confidence information. 4. **Employment-change detection exists in exactly one documented place**: ZoomInfo's "Person Has Moved" setting in Scheduled Enrich for Salesforce (Verified, primary guide). That is the stack Fonteva and Nimble AMS sit on, but neither vendor documents using it. ZoomInfo's own guide says the tool "does not associate contacts with their new companies"; a human must adjust associations. No AMS offers employer-change alerts natively (Reported: associations use job-board alerts and spreadsheets). 5. **Duplicate detection is the mature, well-documented half of data quality**, and the patterns converge: scored or formula-based candidate pairs, side-by-side review, "Not a duplicate" memory, merge log. Differences that matter: scheduling (iMIS), confidence scores and minimum-likelihood filter (Blackbaud), real-time scan on change (Bloomerang), configurable Salesforce rules (Nimble AMS), and **no automatic finder at all** (Wild Apricot, netFORUM merge tool, Aptify merge). 6. **Merge reversibility is poor almost everywhere.** No undo: iMIS (no documented undo; CSI's iMerge states it cannot be undone), Bloomerang, MemberClicks, Wild Apricot (archived record can be restored, merge cannot be undone), netFORUM (hide-instead-of-delete is the only partial reversal). Only the third-party Causeis Smart Duplicates add-on for iMIS claims restore ("Merged information can be restored if required", Marketing). 7. **Provenance and approval are thin.** Best documented patterns: Bloomerang tags NCOA writes with a note and a "Last Modified By: TrueGivers" value; Blackbaud shows a per-pair confidence rating and a minimum-likelihood filter; ZoomInfo applies an accuracy-score floor and a "No Match Reason"; Salesforce Data Cloud identity resolution keeps source values unaltered and reconciles by rule. MomentiveIQ markets "review-and-approve" with configurable confidence thresholds and an audit trail (Marketing). 8. **Personal-email handling for professional members is not addressed in any AMS documentation I found.** Wild Apricot treats the email as the primary identifier and forbids two contacts sharing it (Verified via search summary of help and wishlist pages). Most B2B enrichment vendors key on work email; some accept personal email as an input, some do not (Reported). 9. **Metering is unlike AMS subscription pricing**: per-record-per-service (Loqate via Tessitura), per-contact credits (ZoomInfo), "request pricing" (Blackbaud), bundled in subscription (Bloomerang NCOA, MomentiveIQ core), separate licence tier (iMIS Duplicate Merge Premium). --- ### 2. Platform-by-platform findings Format for each platform: answers to the ten questions. Where a question has no evidence I say "Unavailable". #### 2.1 iMIS (Advanced Solutions International, ASI) Sources: documentation.advsol.com pages (Verified), CSI and Causeis vendor pages (Marketing), ASI news pages (Reported). 1. **Native, marketplace, bureau or absent?** Native enrichment: **absent** (Unavailable in ASI documentation; search of ASI/iMIS material found no data-enrichment feature). Native duplicate management: yes (licensed). Enrichment-adjacent capability arrives from **partners**: - CSI iDMS (iImport, iUpdate, iSchedule, iTransfer, iWorkflow, iHook) is an import/update/transfer toolkit, not enrichment (Reported, search summary of csiinc.com and documentation.advsol.com iDMS pages). - CSI **iValidate**: address standardisation, geolocation, address-move change detection, email verification against unnamed "trusted databases", phone validity, deceased identification, NCOA/CCOA with required forms, US and Canada. The page says "COMING SOON", availability "mid-2025"; I could not confirm it shipped (Marketing; fetched 2026-10-22; pricing, providers and integration mechanics not stated). - ASI partner **Datascout** (AI for iMIS, announced 2025-02-20, a joint venture of Objeto Inc and Bursting Silver): conversational insights, next-best-action, AI-drafted engagement emails. No external enrichment described (Verified page content; Marketing as to value). - ASI partner FUSE Next (AI content discovery, Dec 2025) is not data enrichment (Reported). - Zapier for iMIS (iAppConnector, Armstrong Enterprise Communications, authorised partner May 2022): no-code two-way integration to 4,000+ apps; no enrichment apps named (Verified page). - ASI's roadmap session (Austin, 2026-04-16) says AI-driven features "from predictive analytics to task automation" with no release dates (Reported). 2. **What is enriched:** nothing natively. (iValidate would cover postal, email, phone, deceased if shipped.) 3. **When:** Unavailable natively. Duplicate search runs on demand ("Find Duplicates Now") or on a schedule (one time, daily, weekly, monthly) (Verified). 4. **Duplicate detection and merge** (Verified, documentation.advsol.com): - Two licence tiers. Standard merge is manual, one pair at a time. **Duplicate Merge - Premium** (older doc calls it "Contact Merge - Plus") is a separately licensed add-on that finds all potential duplicates and groups them into pairs. "Potential-duplicate suggestions in the drop-down require a Duplicate Merge - Premium license." - **Matching is formula-based, not fuzzy scoring.** Current doc: individual formulas None, Standard (last name, first name, postal code), Email, Mobile Phone, Strict; organisation formulas None, Standard, Email, Strict, Name and Domain, Company Administrator, Company Name. "Names that differ only in accents or special characters are not flagged." - **Version drift:** the older v3.0 doc says the default "ASI" formula flags two contacts only when the first four characters of last name, the first character of first name and the primary-address zip all match, and "currently cannot be modified." The current doc lists a selectable set of formulas. Do not treat the older rule as current. - Resolve flow: pair list with date found and the formula that matched; profile pages show an alert to staff; "Not Duplicates permanently marks them as not duplicates of each other"; merge keeps one record and fills blanks from the discarded record; related orders, invoices, event registrations, group memberships, activity and usernames move to the survivor; gift history is recalculated. - Limits and blockers: when results exceed 100 potential pairs, not all display; only system administrators can clear potential duplicates (clearing is not "not duplicates", so they can reappear); records with AutoPay cannot merge until unenrolled; overlapping event registrations, memberships, subscriptions or certification enrolments can block; organisations "cannot currently be merged" in the older v3.0 doc (the current doc has an organisation flow, so this blocker may be fixed). - Audit: "Merge logs" show status and changed properties; the deleted contact ID is recorded; merged accounts reportable through the `CsNameLog` business object (Sub Type = MERGE). - Undo: not described (Unavailable). CSI's iMerge docs state there is "no unmerge procedure" (Reported, search summary). - Integration side effect: a developer-forum post notes merges in iMIS confuse an integrated LMS (TopClass) and asks how to identify which record was the master (Reported; no answer on the page). - Third-party: **Causeis Smart Duplicates** (custom matching rules incl. same email, postal code, organisation name, contact name within a company; side-by-side compare; changelog; restore; dashboard) (Marketing, fetched). **CSI iMerge** (custom formulas; merges activities, financial transactions, orders, event registrations, certification; cannot be undone) (Reported). 5. **Employment-change detection:** Unavailable. 6. **Personal email handling:** Email is an available duplicate formula. No statement on personal versus work email (Unavailable). 7. **Provenance, confidence, approval:** the formula name per pair is the only "reason"; no score. Merge is approval-gated by a human with module authorisation level 8 (Verified). 8. **Privacy and consent controls:** Unavailable in the enrichment context. 9. **Pricing:** Premium duplicate merge is a separately licensed feature, price not published (Verified that it is licensed, via the "Potential-duplicate suggestions…require a… license" statement). Causeis and CSI pricing not published. 10. **UX patterns:** worth copying: scheduled scan; "Find Duplicates Now"; the matching formula shown on every pair; staff-only alert on profile; export of the pair list; merge log with reportable object. Worth avoiding: no undo, fixed formulas in older versions, silent display cap at 100 pairs, "clear" that does not equal "not duplicates", merge blockers that are only discovered at merge time. URLs: https://documentation.advsol.com/imis/docs/managing-multiple-duplicates; https://documentation.advsol.com/imis/docs/merging-individuals-and-organizations; https://documentation.advsol.com/imis/v3.0/docs/managing-duplicate-contacts; https://documentation.advsol.com/imis/docs/reporting-on-potential-duplicate-accounts-and-merged-accounts; https://www.csiinc.com/?p=2355 (iValidate); https://www.causeis.com/smartduplicates; https://news.advsol.com/datascout-strategic-partner; https://blog.imis.com/zapier-for-imis; https://developer.imis.com/discuss/68b6e97fae5fe7cfa88e0920. #### 2.2 GrowthZone and ChamberMaster (and MemberSuite, now under GrowthZone) 1. **Native, marketplace, bureau or absent?** Enrichment: **Unavailable**. No enrichment, address-validation, NCOA or email-validation feature or partner found in GrowthZone's help or integration pages. The company's 2026 lineup lists add-ons GZ Pay, GZ CMS, GZ Community, GZ Learn, "AI Powered Newsletter" and Integrations; the only AI product named is the newsletter (Verified, GrowthZone press releases 2023-09-21 and 2026-01-13). The integration site names no data partners (Verified, fetched; it lists guides, an API page, email/SPF and SSL topics). 2. **What is enriched:** n/a. 3. **When:** n/a. Imports are the main data-entry path. 4. **Duplicate detection and merge:** - Import: "Enforce Account Number Uniqueness" rejects rows that match an existing account number; to update existing contacts the import file must contain account numbers; there is no general duplicate check on email in the main Import Contacts tool (Reported, search summary of helpdesk.growthzone.com articles 45935938310043 and others; page not opened by me). - List/committee contact import matches by email address and creates no record when email is missing (Reported). - Merge Contacts: when merging, "All information, billing history, communications, activity, etc. will be merged to the destination contact"; conflicting NationalID numbers keep the destination's value (Reported, search summary of supportwiki.growthzone.com and the Managing Merging Duplicates article; the article itself returned 403). - Automated duplicate finder: Unavailable. 5. **Employment-change detection:** Unavailable. 6. **Personal email handling:** Unavailable. 7. **Provenance and approval:** Unavailable. 8. **Privacy and consent:** GrowthZone publishes GDPR/CCPA policy pages, not enrichment controls (Verified existence only). 9. **Pricing:** Unavailable for data-quality features. 10. **UX patterns:** account-number-keyed import (explicit external ID as the only reliable match) is the pattern to note; absence of a finder is the gap. URLs: https://www.growthzone.com/news-press/growthzone-accelerates-momentum; https://www.growthzone.com/news-press/growthzone-membersuite-join-forces; https://integration.growthzone.com/?p=57108; https://helpdesk.growthzone.com/hc/en-us/articles/45934904318747-Managing-Merging-Duplicates (403); https://helpdesk.growthzone.com/hc/en-us/articles/45935938310043-Import-Contacts (search summary only). #### 2.3 MemberSuite (GrowthZone-owned) - Enrichment, duplicate detection, address validation: **Unavailable**. I could not reach MemberSuite-specific documentation before the search budget ended. Ownership and product placement are Verified (GrowthZone press release). - Action for the proposal: treat MemberSuite as unevaluated rather than as lacking. #### 2.4 netFORUM (Momentive Software, ex-Community Brands) Sources: documentation.mip.com (netFORUM Enterprise 2017.1 and 2015.1). Old versions; current netFORUM Pro documentation not found. 1. **Native, marketplace, bureau or absent?** Address correction is a **plug-in system option**, not enrichment. `AddressCorrection` (Admin > System Options, category CRM, introduced in 2006.01 or earlier) selects an address-correction product: MelissaDataGlobal, MelissaDataPersonator (USA and Canada), Mailers+4, CDYNE, AccuMail, AccumailWebServices (SmartSoft), Group 1, AddressDoctor, FirstLogic IQ8. Default is none. "DLL and INI files must be installed on the server." (Verified.) 2. **What is enriched:** postal address standardisation/correction only. 3. **When:** depends on the product; the doc points to each product's page. Not stated. 4. **Duplicate detection and merge** (Verified): - **"The Combine/Merge tool does not identify duplicates."** Staff find candidates with Query or Find. A separate _Individual Duplicate Check Report_ applies "configurable matching algorithms". - Merge: choose a master (Keep radio button), choose data categories to move (all selected by default), hide or delete the others. "The data selected is not checked for duplicates", so addresses can double up on the master. - **No un-merge.** Hidden records can be unhidden (restores the customer and Customer ID but not moved or deleted memberships or committee assignments); deleted records are gone. - Merge can fail on dependent records (foreign keys); organisation merge drops the dropped organisation's custom demographics; administrators can change the tables used via `md_dedup_table`. 5. **Employment-change detection:** Unavailable. 6. **Personal email handling:** Unavailable. 7. **Provenance and approval:** Log tab on the kept customer lists hidden merged customers, not deleted ones (Verified). 8. **Privacy and consent:** Unavailable. 9. **Pricing:** Unavailable. 10. **UX patterns:** the hide-versus-delete choice is the only soft-delete in this set and is worth copying (and extending to full restoration); the "browser must stay open or the merge is lost" behaviour is a design smell to avoid. 11. **Newer layer:** MomentiveIQ (section 2.20) is Momentive's AI and "Connections" layer and covers netFORUM only by inference; its page does not name which products it supports. URLs: https://documentation.mip.com/netforum-enterprise/2017.1/Content/CRM/Individuals/Combine_and_Merge_Duplicate_Records.htm; https://documentation.mip.com/netforum-enterprise/2017.1/Content/SystemOptions/AddressCorrection.htm. #### 2.5 Fonteva (Salesforce-native) 1. **Native, marketplace, bureau or absent?** Fonteva-specific enrichment documentation: **Unavailable**. Fonteva's own pages describe a Salesforce-native AMS (membership, events, communities, payments, commerce; 800+ association customers per vendor claim, Marketing). Its "Fonteva 15" article (undated; image path suggests September 2025) lists fundraising tools, drag-and-drop dashboards, flexible community spaces, refunds and credit memos; it does not mention AI, data quality, duplicate management, Data Cloud or Agentforce (Verified, fetched). 2. **What is enriched:** n/a. Fonteva inherits what the Salesforce org installs: Salesforce standard duplicate management and any AppExchange enrichment package (see 2.6 and 3.1). 3. **When:** n/a. 4. **Duplicates:** Fonteva documentation not found. Salesforce platform behaviour applies (section 3.1). Reported gap, not Fonteva-confirmed. 5. **Employment-change detection:** none documented by Fonteva. ZoomInfo's Salesforce app can do it if licensed (section 3.2). 6. **Personal email handling:** Unavailable. 7-8. Unavailable. 7. **Pricing:** Fonteva does not publish enrichment pricing; third-party enrichment priced by the third party. 8. **UX pattern:** the architectural choice (use the platform's duplicate and enrichment ecosystem rather than build) is the lesson. It also inherits Salesforce's limitation that matching rules cover Account, Contact and Lead only (Reported). URLs: https://fonteva.com/ (about, partners pages); https://fonteva.com/?p=12312. #### 2.6 Nimble AMS (Momentive; Salesforce-native) 1. **Native, marketplace, bureau or absent?** - Enrichment: no Nimble-specific enrichment feature or named data partner found. **Nimble Intelligence** (announced March 2024; built on Salesforce) says Nimble AMS and Salesforce data "easily integrates with external data" but names no sources, methods or partners; no pricing, models or governance stated (Marketing, fetched 2026-10-22). Nimble AMS **Predictions** (Salesforce Einstein; lapse risk, recruitment and donation propensity, course recommendations) is analytics on existing data, not enrichment (Reported, trade press). - The Marine Corps Association & Foundation lapse-risk tool was a custom build, not a standard feature (Verified from the press page's wording). 2. **What is enriched:** n/a. 3. **When:** n/a. 4. **Duplicate detection** (Verified, help.nimbleams.com): - Uses **Salesforce Duplicate Management** end to end: a matching rule ("the logic that determines if an account is considered a 'duplicate' account") plus a duplicate rule (what happens on create or edit). Separate rules for business accounts and person accounts. Shipped values: "Nimble AMS Account Duplicate Rule" and "Nimble AMS Person Account Duplicate Rule", Action on Create "Allow: Alert and Report", Action on Edit "Allow: Report". Matching fields and fuzzy-versus-exact are "not specified" on the Nimble page; it defers to Salesforce's standard rules. - **Self-service coverage**: in Community Hub, administrators control whether duplicate checking runs during Create Account (self-registration), Add Individual and Add Affiliation. When checking is off, "the account is simply created." - Documentation warns not to make Create or Edit actions more restrictive, because Community Hub treats "Alert and Report" on edit like a block with a generic error. - Troubleshooting references two duplicate services: DupeService/ContainsDupeService2 (Community Hub only, limited matching) and DatacloudDupeService (Salesforce duplicate rules). That naming ("Datacloud") is the legacy Data.com-era Salesforce duplicate service name, not Salesforce Data Cloud (Reported inference; the page does not explain the name). - Reporting: admins can report on duplicates users chose to save. - Merge tooling: the pages fetched link to a "Merge Duplicate Accounts" page and an FAQ but do not describe them (Unavailable here). 5. **Employment-change detection:** none documented. 6. **Personal email handling:** Unavailable. Person Account duplicate rules depend on the field set configured and exposed on Community Hub forms. 7. **Provenance and approval:** Duplicate reports record bypassed alerts ("Report"); no enrichment provenance. 8. **Privacy and consent:** none documented for enrichment. 9. **Pricing:** Nimble Intelligence pricing: not stated. 10. **UX patterns:** alert-on-create, silent-report-on-edit is a reasonable low-friction default; the Community Hub interference (staff rules breaking public forms) is the trap to avoid: scope rules by actor. URLs: https://help.nimbleams.com/help/live/duplicate-account-management; https://help.nimbleams.com/help/live/configure-matching-and-duplicate-account-rules; https://help.nimbleams.com/help/live/troubleshoot-duplicate-account-management; https://www.communitybrands.com/company/news/community-brands-introduces-nimble-intelligence/. #### 2.7 Personify360 (now Momentive) 1. **Native, marketplace, bureau or absent?** Unavailable for enrichment in the fetched help. The Personify360 help (v7.7.0) page on duplicates does not mention address standardisation (Verified). 2-3. n/a. 2. **Duplicate detection and merge** (Verified, partial): - Duplicate processing applies only to Individual and Company records, not Committees or Subgroups (same names can legitimately exist across companies). - Duplicates enter by data entry, lockbox imports and marketing-list imports; "each path may warrant different rules". - Merge is "essentially a search-and-replace on Customer ID values" across tables, run as a batch process; there is also an Online Duplicate Customer Merge screen for known pairs. Orders and payments move to the retained customer; duplicate orders and line items must be cancelled manually. - Related topics named but not retrievable: "Identifying Potential Duplicate Customers" and "Soundex vs. Phonetic" (so phonetic matching is implied, not confirmed). 5-8. Unavailable. 3. Unavailable. 4. **UX note:** different matching rules per entry path (import, data entry, lockbox) is a good idea; the manual cancellation of duplicate orders after merge shows what a merge engine must handle for financial records. URL: https://resource1.personifycorp.com/PersonifyOnlineHelp/7.7.0/mergedProjects/Customers/Duplicate_Customers_Overview.htm. #### 2.8 MemberClicks, YourMembership and Wild Apricot - **MemberClicks:** Actions menu > "Find A Duplicate" on the profile you want to keep; merge up to two profiles at once; choose attribute values from each; member type, name and status cannot be merged; messages, invoices and submission histories are retained; organisation member profiles cannot be merged; merges cannot be undone; duplicate profile moves to deleted status. (Reported: search summary of help.memberclicks.com article 16319885298701; page returned 403 to me.) No automatic duplicate flagging found. - **Wild Apricot:** - No automated finder. The 2012 wishlist request "check for duplicates" has 73 votes, comments to 2022; the only vendor reply (2016, "Wild Apricot by Personify") asks "How would you define a duplicate contact? What are criteria to recognize duplicates?" and nothing later. Verified (forum page fetched). - Help article "Merging contacts": per the search summary, the article states there is no mechanism to search for duplicates; merge is manual (click Merge, pick contact, choose active record, type MERGE), the other record is archived, merge cannot be undone though the archived contact can be restored; event registrations, donations, posts and invoices move; privacy settings, email preferences and admin rights do not; the only full administrator or a member with active recurring payments cannot be merged (Reported; help page itself returned only a heading to the fetch tool). - **Email is the primary identifier**: an import with a matching email updates the existing record; two contacts cannot share the same main email (Reported). Users ask for multiple emails per contact (work and personal) in the wishlist thread (Verified): relevant to the personal-email gap. - **YourMembership:** Unavailable (no YourMembership-specific help found). - Enrichment for all three: Unavailable. URLs: https://forums.wildapricot.com/forums/308932-wishlist/suggestions/8827018-automate-finding-duplicate-contact-member-records; https://gethelp.wildapricot.com/en/articles/129-merging-contacts; https://help.memberclicks.com/hc/en-us/articles/16319885298701-Merging-duplicate-member-profiles. #### 2.9 Aptify 1. **Native, marketplace, bureau or absent?** Address verification is a **third-party connector**: Aptify ships "Address Verification Systems" records for Melissa Data Webservice (ID 3, Rank 1), Accumail (ID 2, Rank 2) and Melissa Data (ID 1, Rank 3). Aptify recommends the Melissa Data Webservice record using the Address Object API; setup needs the Melissa Customer ID; Melissa client software must be installed on each workstation using it; users click **Verify Address Against Postal Database** in the Address Management toolbar (Reported: search summary of aptifykb.atlassian.net pages; the pages require login so I could not read them). 2. **What is enriched:** postal address verification; NCOA not mentioned in the pages found. 3. **When:** on demand by the user (button), per the above. 4. **Duplicates** (Reported, same limits): Aptify ships sample Persons and Companies "Duplicate Check" objects (same first and last name; same exact company name) in the Object Repository but **the checking is not enabled by default**. An admin adds a "Duplicate Check" plug-in to the entity, then restarts; checks run on save. The documentation says the samples are only samples because duplicates vary by entity and organisation. No merge tool or batch finder found. 5-8. Unavailable. 5. Melissa licence is the customer's own (Customer ID). 6. **UX note:** button-driven verification with a ranked list of fallback providers is a clean provider-adapter pattern (ranked providers per capability). URLs: https://aptifykb.atlassian.net/wiki/spaces/PK55/pages/15417383271/Integrating+With+Third-Party+Address+Verification+Systems (login required); https://aptifykb.atlassian.net/wiki/spaces/A6PK/pages/21935068686/About+the+Baseline+Persons+and+Companies+Sample+Duplicate+Objects (login required). #### 2.10 Impexium, Novi AMS, Glue Up, Raklet, Higher Logic - **Impexium (re:Members):** no vendor documentation found. One association's internal procedure (ALTA) shows a "Merge Customers" tab under Customers for Individuals or Organizations, administrator-only, with a "Merges" note recording the reason (Reported). Enrichment: Unavailable. - **Novi AMS:** merge is manual from Batch Actions > Merge Records, two records of the same type at a time (two people or two companies), main record wins but the admin can choose which data wins field by field; both records must be visible on the same list page. Member import requires an Import ID unique across people and companies (Reported, search summary of help.noviams.com; the merge page returned 404 on direct fetch). Enrichment: Unavailable. - **Glue Up:** Unavailable (no Glue Up pages surfaced). - **Raklet:** not researched (budget). - **Higher Logic** (Thrive marketing automation, Vanilla community): connectors to Salesforce, HubSpot, netFORUM, iMIS, Personify; roadmap language about enriching existing systems with engagement data; no ZoomInfo integration, no enrichment or dedupe feature found (Reported). - **Naylor** AMS: integrates third-party add-ons through "Naylor Marketplace"; career center feeds non-member identification into the AMS at registration (Verified page content); no "member data platform" described on the fetched page. #### 2.11 Tessitura (constituent platform, arts and culture; useful for the "services bureau inside a CRM" pattern) 1. **Native, marketplace, bureau or absent?** **Partner bureau, built into a standard utility.** Tessitura has a standard NCOA Processing utility; vendors plug in. Experian Data Quality provides "fully integrated, real-time" address standardisation at data entry in the client application and online (TNEW) (Reported, Tessitura vendor page via search). **Loqate** (GBG) provides NCOA, Deceased and Do Not Mail; real-time verification for new records in-app and online; NCOA for existing records; announced 2026-04-21; Tessitura "will maintain" its standard NCOA utility "to stay compatible" with Loqate (Verified, Tessitura news and vendor pages fetched). 2. **What is enriched:** postal address, movers, deceased, Do Not Mail. 3. **When:** real-time at entry; NCOA per processing run (the vendor page prices "per processing run", implying batch; real-time/API options not stated there). 4. **Duplicates:** Reported from community forums only: auto-flagged potential duplicates cannot be removed individually from the merge queue; sites use a "Not a Duplicate" association or the "Void Merge" attribute (which blocks the record from merging with anything, not only the mismatched pair); some sites run SQL scripts weekly and merge manually; suggestion to use soundex or metaphone for names. 5-6. Unavailable. 5. Unavailable. 6. Do Not Mail is a service line (Verified). 7. **Pricing: Verified.** "$1 per 1,000 records processed, per service"; "Charges are calculated based on the number of records submitted for each processing run"; Tessitura members get preferred pricing under a community purchasing agreement. An older Zeta Global partnership listing also existed (Reported; status unclear). 8. **UX patterns:** the consortium purchasing agreement (collective bargaining on per-record price) is notable; the "void merge" blunt attribute is the anti-pattern (an exclusion should be pairwise). URLs: https://www.tessitura.com/items/articles/news/2026/loqate; https://www.tessitura.com/Items/Vendors/Loqate; https://tessitura.com/items/vendors/experian (search result). #### 2.12 Blackbaud Raiser's Edge NXT (and the Blackbaud "Data Health" enrichment services) This is the closest analogue to what an association would want: a services bureau packaged into the product. 1. **Native, marketplace, bureau or absent?** Both: **in-product scheduled services** (Address Finder, Deceased Finder, duplicate monitoring) plus **bureau services sold separately** (EmailFinder, PhoneFinder, CellPhoneFinder, AgeFinder, GenderFinder, AddressAccelerator, Identity Thread). The marketing page says data enrichment services for phones, emails and addresses are included with an NXT subscription depending on the Analytics package ("Availability … based on your Raiser's Edge NXT Analytic package") (Reported, search-result quote of Blackbaud page; the page I fetched gave only "Request Pricing"). 2. **What is enriched** (Marketing for the service list, Verified for the in-product pieces): - AddressFinder: NCOA screening, USPS compliance. AddressAccelerator: validates and corrects addresses as entered and in batch. DeceasedRecordFinder. EmailFinder (adds missing emails). PhoneFinder (adds numbers, flags National Do Not Call). CellPhoneFinder. AgeFinder and GenderFinder (demographic append for segmentation). **Identity Thread**: identity resolution "powered by LiveRamp" against a master referential database, returning individual and household keys, claimed to find duplicates beyond name-and-address matching. - No employer or job-title append found. Employer and job title are entered by hand in the relationships window (Reported, community threads). 3. **When** (Verified, NXT Data Health help page): - **Address Finder: every 90 days**, compares addresses to USPS NCOALink and CASS-certified processing; shows most recent, next and previous updates; feeds an "Address health" score. - **Deceased Finder: every 90 days**, compares against the Social Security Administration death benefits data and other sources; flagged records are managed for review. - **Possible duplicates: continuous monitoring**; user reviews and decides. - The scorecard is "Great / Good / Average / Below average" per area; the page notes automatic and manual scorecard updates were "currently paused … during a solution transition" when I fetched it (a live example of a vendor pausing enrichment while changing provider). - The help page does not mention email or phone checks, so those appear to be separate batch services. For Blackbaud CRM (a different product) PhoneFinder is a **file exchange**: "use Data Tune-Up to create and submit files … to Blackbaud, then import the returned information"; numbers are checked connected/disconnected and listed/delisted; DNC matches are auto-marked "Do not call this phone number" when the batch is committed (Verified, Blackbaud CRM help; not NXT). - AddressAccelerator is the only service stated to work in real time at entry (Marketing page). 4. **Duplicate detection and merge** (Verified, NXT help): - **Scored fuzzy matching**: "Each possible match has a confidence rating"; matches are based on name, mailing address, phone and email and "indicate which of these fields it relies on"; filters include Include deceased, Include inactive and **Minimum match likelihood** (zero shows everything). - **Automatic matching on forms and donations is deliberately strict**: "name and email fields must match exactly" before an existing constituent is updated. The looser fuzzy tool is for clean-up of duplicates that arrive through imports, integrations and online forms. (Verified.) - Merge: compare source and target, pick data, "Merge records"; optionally delete the source; web view merges only listed fields, the rest needs database view; the "Merged duplicates" list shows status and errors. "Not a duplicate" removes the pair from the list. A community answer says the database view and web view use different algorithms and return different results (Reported). 5. **Employment-change detection:** Unavailable. 6. **Personal email handling:** Unavailable (donor context; the question does not arise). 7. **Provenance, confidence, approval:** match confidence per duplicate pair (Verified); Address Finder shows previous and next updates (Verified); whether the old address is retained after an NCOA update is not documented in what I could read (Unavailable). Setup: "to enable Address Finder, in Tools, Data health, select Get started under Address Finder to set up how it should handle your addresses and to acknowledge Blackbaud can process your data" (Reported, search-result quote of a Blackbaud release note; community page not readable by me). Role-based control: admins can allow manage or view-only by role (Reported). 8. **Privacy and consent:** PhoneFinder flags National Do Not Call (Verified for Blackbaud CRM). The product page cites USPS standards for NCOA and no other compliance standard; no match-rate, source or consent information for EmailFinder (Verified absence on the page I read). Community members raise opt-in and opt-out concerns about appended emails and "old data" (Reported, thread dated 2015-2017). 9. **Pricing:** "Request Pricing" only (Verified). Community statement that address updates can run up to four times a year at no extra charge (Reported, 2026 community post via search). Case studies: Houston Community College Foundation reported 27-28% return rates after EmailFinder (vendor-hosted PDF, ~2010, Marketing); Lowcountry Food Bank credits the tools with "over 25,000 email addresses and nearly 30,000 phone numbers" (vendor testimonial, Marketing). A Blackbaud community post cites average 29% of supporters getting an email appended and 42% phone updates (Reported; Blackbaud's own averages, unverified). 10. **UX patterns:** worth copying: a "data health" scorecard that rolls enrichment status into one visible dashboard; per-pair confidence plus a minimum-likelihood filter; separation of strict automatic matching from loose review matching; the fields-the-match-is-based-on column. Worth avoiding: two algorithms (web versus database view) giving different results; a scorecard that goes "paused" with no ETA; enrichment fields not documented in one place. URLs: https://www.blackbaud.com/nonprofit-direct-marketing/data-enrichment; https://www.blackbaud.com/?p=3072; https://webfiles-sc1.blackbaud.com/files/support/helpfiles/rex/content/data.html; https://webfiles-sc1.blackbaud.com/files/support/helpfiles/rex/content/data-identify-duplicates.html; https://webfiles-sc1.blackbaud.com/files/support/helpfiles/rex/content/bb-duplicate-records.html; https://webfiles-sc1.blackbaud.com/files/support/helpfiles/crm/uk/40/Content/ADMPhoneFinderProcess.html; https://community.blackbaud.com/blogs/13/5744 (release note, not readable by fetch); https://www.blackbaud.com/files/pdf/directmarketing/CaseStudy_Houston_Community_College_Foundation.pdf; https://www.blackbaud.com/?p=17236 (Lowcountry). #### 2.13 Bloomerang 1. **Native, marketplace, bureau or absent?** **Built-in via partner, plus premier partner integrations.** Nightly NCOA and deceased updates run through **TrueGivers**; wealth insights built in; email-list cleaning built in; card updater built in. Premier integrations: **DonorSearch** and **Dataro** (prospect research, predictive donor signals). Email, phone and address _append_: no Bloomerang product found (Verified on integration page that none is listed; Reported otherwise). 2. **What is enriched** (Verified, help.bloomerang.com NCOA basics): current address standardised to USPS guidelines; if not mailable marked **Bad**, if a new address is found it becomes **Primary**; deceased constituents marked Deceased (NCOA dashboard shows date and obituary link). Optional demographic fields (Birth Year, Home Owner, Income Decile, Market Value Range, Net Worth Decile, Religion, Heritage) are opt-in checkboxes. 3. **When:** **nightly**; day and time schedule per action (download, update, sync) or manual Synchronize; pausing the integration stops nightly updates. 4. **Duplicates** (Verified, help.bloomerang.com article 12632743 and the feature announcement): - Compares **name, address, email and phone**; **scans in real time** whenever constituents are created, deleted, merged or edited; flags pairs when name and contact info match or when they share an email alone; household members are not flagged; previously dismissed pairs are not flagged; an individual and an organisation can be paired. - No numeric threshold published. Stated philosophy: "the only thing worse than having duplicate records is merging records that aren't really duplicates"; so it matches only when highly confident. - Review flow: notification bell "You Have Possible Duplicates" or Data Tools > Merge Duplicates; compare view with red strikethrough for information that will be lost and green bold for information added; per pair choose Merge, Not a Duplicate (permanent) or **Create Household**; batch merge multiple pairs ("Select all on Page", "Merge Selected"). - **No undo**: "Merges are permanent, and Bloomerang cannot restore the separate records or deleted data." - Journey Automation and group side effects of merge are documented in detail (e.g., merging a First Ever Donation participant with a prior donor removes them from that automation). - No auto-merge (the announcement describes only user-initiated merges). 5. **Employment-change detection:** Unavailable. 6. **Personal email handling:** a shared email is itself a duplicate signal; household creation is the resolution when two people legitimately share an address. 7. **Provenance and approval:** (Verified) NCOA address writes carry an **Address Note = "NCOA"**; the report can filter **Last Modified By contains TrueGivers** and Last Modified Date (= sync date); **Archive Addresses** setting keeps the prior address when on, "no history is kept" when off (default not stated); optional notification emails report counts after each sync. There is **no per-change approval step** for NCOA address changes. 8. **Privacy and consent:** NCOA processing "must be activated in the integration settings"; demographic fields opt-in; customisation reviewed before the first sync (Verified). USPS requires NCOA processing within 90 days of a mailing for preferred rates (Reported, Bloomerang blog). 9. **Pricing:** "included at no extra cost" per the integrations page FAQ (Wealth insights, NCOA processing, card updater, email list cleaning); the help article says to contact an account manager to add NCOA processing, and a feature post says standard subscribers can opt in at no extra cost. Sources disagree, so treat as "bundled, but confirm" (Verified conflict). DonorSearch and Dataro priced separately, not published. 10. **UX patterns:** worth copying: auto-apply with an audit note and an "archive previous value" switch; real-time duplicate notification; the "Create Household" third option; tri-state hide of the notification (1 hour, until next login, always); side-by-side with loss and gain colouring. Worth avoiding: no undo and no approval gate on writes. URLs: https://bloomerang.com/product/integrations-data-management; https://help.bloomerang.com/en/articles/12632612-truegivers-national-change-of-address-ncoa-basics; https://help.bloomerang.com/en/articles/12632564-report-on-national-change-of-address-ncoa-updates; https://help.bloomerang.com/en/articles/12632743-review-and-merge-duplicate-constituents; https://bloomerang.com/news/new-feature-real-time-duplicate-constituent-management. #### 2.14 Neon One (Neon CRM) - Enrichment and address validation: Unavailable. Duplicate handling: the Neon Academy course covers an "account match" feature, a partial-match queue, a duplicate scan, and Email Link Authentication and Login Prompt to prevent duplicate accounts (Reported, search summary; the academy page returned 403). Imports reference a Duplicate Management tool (Reported). Fuzzy detail not available. #### 2.15 Salesforce Nonprofit Cloud and the Salesforce platform (context for Fonteva and Nimble AMS) - **Duplicate management (Verified, Trailhead):** matching rules find potential duplicates; duplicate rules decide the response (alert or block, on create or edit); up to three matching rules per duplicate rule and five active matching rules per object; standard rules use fuzzy algorithms (Jaro-Winkler and Metaphone 3 for names, Edit Distance for mailing streets, weighted address sections); custom rules can add fuzzy phone and street logic; report types can include Duplicate Record Items to see bypassed alerts; matching rules are supported on Account, Contact and Lead (Reported for the object limit). - **Data Cloud (Data 360) identity resolution (Verified, Trailhead):** rulesets define match rules (examples: Fuzzy Name and Normalized Address; Fuzzy Name plus normalised phone and email); **reconciliation rules** choose which value wins in the unified profile by frequency, recency or source priority; "data from the source records isn't altered", with unified link objects so every value traces to its source. This is the strongest provenance model I found in the whole study and is worth copying directly. - **Native enrichment:** Salesforce **retired Data.com**: Connect closed May 2019; Clean and Prospector not renewed after 2019-07-31 and retired 2020-07-31; contact data removed 2021-02-01 (Reported, third-party blogs and a Salesforce developer object reference via search). The lesson: bundled vendor-owned data was withdrawn from a platform; the ecosystem moved to AppExchange vendors. - **Nonprofit Cloud / Agentforce for Nonprofits:** third-party blogs describe AI agents and Data Cloud unification; Winter '26 release notes list a Participant Management Agent and gift planning; **no enrichment or deduplication feature found** (Reported). URLs: https://trailhead.salesforce.com/content/learn/modules/sales_admin_duplicate_management/sales_admin_duplicate_management_unit_2; https://trailhead.salesforce.com/content/learn/projects/explore-data-cloud-core-functionality/unify-your-data; https://developer.salesforce.com/docs/atlas.en-us.object_reference.meta/api/sforce_api_objects_leadcleaninfo.htm (search result only); https://www.salesforce.com/ca/nonprofit/artificial-intelligence/ (search result only). #### 2.16 Momentive Software MomentiveIQ (cross-product, newest and most relevant claim) Announced 2026-01-05; Personify acquisition 2026-01-06. (Verified that the pages say this; Marketing for what it does.) - **Marketing claims, quoted from vendor pages:** - "Member data cleansing and enrichment is built into MomentiveIQ Agentic Workers." (FAQ, fetched on the AI-agent page and solution page.) - "The platform surfaces data quality gaps and resolves duplicate identities as part of normal operation." - **Connections**: "resolving cross-system identity so your data agrees on who your members and donors are", through "pre-built adapters and an open API", working "alongside competitor AMS, CRM, and third-party tools". - Approval and audit: "AI That Does the Work. Your Team Approves It."; "Start with full review-and-approve"; "Configure confidence thresholds as your team gets comfortable"; autonomy can be granted later; "Actions are logged with an audit trail, so your team and your board know what the AI did and why"; agents "write back to the system of record". - Launch agents (per a trade-press summary): Membership Assistant and Community Assistant (Reported; the vendor page I read does not list them). - Pricing: "Momentive clients receive core MomentiveIQ capabilities as part of their subscription. Premium add-ons are available." No prices. - Professional services "to accelerate cleanup before launch". - **What is missing:** no description of data sources for enrichment, no provider names, no field list, no consent or privacy statement, no description of how confidence is computed, and no statement of which products (netFORUM, Nimble AMS, YourMembership, MemberClicks, Wild Apricot, Personify360) it covers. I found no independent reviews or customer results. - **Significance:** it is the only AMS-vendor statement that combines enrichment, duplicate identity resolution, confidence thresholds, human approval and audit in one pitch, which matches the design direction of this proposal. It is not yet evidence that the capability works. URLs: https://momentivesoftware.com/solutions/momentiveiq/; https://momentivesoftware.com/solutions/momentiveiq/ai-agent/; https://momentivesoftware.com/momentiveiq/; https://momentivesoftware.com/press-releases/momentiveiq-launch/; https://momentivesoftware.com/press-releases/personify-acquisition/; https://momentivesoftware.com/solutions/momentiveiq/ai-analytics/. --- ### 3. Cross-cutting evidence #### 3.1 What the Salesforce-based AMS vendors (Fonteva, Nimble AMS) actually inherit - Duplicate rules, matching rules and Duplicate Record Items reports (Verified above). - A package-based enrichment ecosystem, notably ZoomInfo (section 3.2), whose guide documents a credit-metered, scheduled and instant enrichment model. Neither Fonteva nor Nimble AMS documentation (that I could reach) references it. - Data Cloud identity resolution as an optional layer (Salesforce documentation; no Fonteva or Nimble AMS link found). - The Nimble AMS help page shows the sharp edge of inheriting platform rules: staff-facing duplicate rules can break public self-service forms (Verified). #### 3.2 ZoomInfo "Enrich for Salesforce" (primary guide; shows the shape of a B2B enrichment integration an AMS could be wired to) Verified from ZoomInfo's integration guide (PDF, fetched and read locally): - **Two modes**: _Instant Enrich_ appends lead, contact and account records "as they are created"; _Scheduled Enrich_ runs recurring jobs on a cadence. Jobs begin at 12:00 AM ET on the scheduled day. - **Credit controls per job**: No limit (default), only enrich records currently owned (no credit), or limit credits per run (processing stops at the cap; remaining records roll to the next run). Each contact exported or enriched consumes one credit. Using a Salesforce sandbox still incurs credit charges. - **Accuracy gate**: "You can control contact enrichment by Accuracy Score"; records below the minimum are marked **No Match** with the reason "Below selected Accuracy Score" and are not exported to the custom object. - **Employment change**: the **Person Has Moved** setting (default enabled) enriches a moved contact to the new organisation; if disabled, company fields continue as if the person had not moved and the status becomes "Company Match Only". A mapped field holds Yes / No / Uncertain; users can report on "Yes". "Enrich does not associate contacts with their new companies. It is up to your organization to evaluate these records and adjust associations." Most of the association work happens on the first run; later runs "pick up a small number of deltas". - **Overwrite policy**: each mapped field is "Complete if missing" (default) or "Overwrite"; ZoomInfo recommends Overwrite only for custom fields created for ZoomInfo and leaving standard Salesforce fields on Complete if missing. - **Identity keys**: ZoomInfo Contact ID and Company ID are stored on the Salesforce record; default match inputs cannot be removed, but extra match inputs can be added. - **Job filters and batch**: enrich all, a list view, or criteria; batch sizes 10K, 1K, 500 or 200 per API call; a dashboard reports credits and API use. - Cost context: third-party sources put ZoomInfo contracts at roughly $15,000 to $40,000 per year (Reported, vendor-comparison blog, biased). - This guide documents B2B employer data on business contacts; nothing in it addresses members who use personal email or the privacy position of enriching individuals (Unavailable). URL: https://tech-docs-library.zoominfo.com/enrich-for-salesforce-integration-guide.pdf. #### 3.3 Other enrichment-provider facts surfaced incidentally (not AMS-specific; other workstreams likely cover them) - Clearbit is now **Breeze Intelligence** (HubSpot acquired Clearbit December 2023; rebranded 2024). Enrichment credits and pricing reports conflict across third-party blogs (Reported); a paid HubSpot subscription is required (Reported). Not usable as a Salesforce-side provider without HubSpot. - FullEnrich states it does not currently provide personal email addresses; other vendors (Snov.io, MixRank, Datagma named in a Clay community post) claim to accept personal email as an input (Reported, vendor claims). - Melissa, Experian, Loqate, Smarty, PostGrid and Service Objects all sell CASS/NCOA/address services; none was found with a documented iMIS connector (Unavailable). #### 3.4 Employment-change detection across the study | Source | Mechanism | Label | | ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------ | | ZoomInfo Enrich for Salesforce | "Person Has Moved" flag (Yes/No/Uncertain), optional automatic re-enrichment to the new company; no automatic re-association of records | Verified (primary guide) | | Associations Now (NAFA example) | Manual: job-board alerts on key positions, spreadsheet tracking, then database check | Reported | | Momentive / Naylor career centre pitches | Job boards as retention/prospecting signals; no automated job-change alert | Marketing | | All AMS and nonprofit CRMs in this report | No native job-change detection found | Unavailable | #### 3.5 Personal-email handling across the study - **No AMS or CRM documentation** I read addresses enrichment lookups for members who join with a personal address. - Wild Apricot: email is the primary identifier; no two contacts may share a main email; users request multiple emails per contact (work and personal) (Reported/Verified as noted in 2.8). - Bloomerang: shared email is a duplicate signal; households resolve legitimate sharing (Verified). - iMIS: Email and Mobile Phone are separate duplicate formulas (Verified). - Salesforce Data Cloud: fuzzy name plus normalised email or phone as match rules (Verified example). - Enrichment vendors differ on whether personal email is supported (Reported). The gap is real, and unaddressed by AMS vendors. --- ### 4. Section (a): Comparison matrix Legend: V = Verified, R = Reported, M = Marketing, U = Unavailable. "Native enrichment" means a first-party feature that fetches external data, not duplicate management. | Platform | Native enrichment | Provider used | Trigger modes | Duplicate detection | Employment change | Personal-email handling | Provenance / confidence / approval | Consent controls | Pricing model | | ----------------------------------------- | -------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | ------------------------------- | ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ | ------------------------------------------------------------------------------ | | iMIS (ASI) | No (U). Partner add-ons only | CSI iValidate (unnamed "trusted databases", may not have shipped) (M); none native | Duplicate search on demand or scheduled daily/weekly/monthly (V) | Formula-based pairs (Standard, Email, Mobile Phone, Strict; org formulas); licensed Premium tier; manual tier (V) | No (U) | Not addressed (U) | Matching formula shown per pair; merge logs; CsNameLog; no scores; human merge only (V) | None found (U) | Premium duplicate merge is a separate licence; price unpublished (V) | | GrowthZone / ChamberMaster | No (U) | None | Imports; merge manual (R) | Account-number uniqueness on import; manual merge; no finder found (R) | No (U) | U | U | U | U | | MemberSuite (GrowthZone) | Not assessed (U) | U | U | U | U | U | U | U | U | | netFORUM (Momentive) | No; address-correction plug-in (V) | Melissa (Global, Personator, Mailers+4), CDYNE, AccuMail, Group 1, AddressDoctor, FirstLogic (V) | Per product; on entry or batch not stated (V) | None automatic in merge tool; separate duplicate-check report with configurable algorithms (V) | No (U) | U | Log tab for hidden merges only (V) | U | Customer pays Melissa etc. directly (inferred) | | Fonteva | Not documented (U) | Whatever the Salesforce org installs | Inherited from Salesforce | Salesforce matching and duplicate rules (V for platform) | None documented | U | Salesforce Duplicate Record Items report (V platform) | U | Third-party priced | | Nimble AMS (Momentive) | No (M: "integrates with external data") | None named (M) | Alert on create, report on edit (V) | Salesforce rules, person and business accounts, Community Hub aware (V) | No | U | Duplicate reports; "Allow: Alert and Report" (V) | U | Nimble Intelligence price not stated (M) | | Personify360 (Momentive) | No (U) | None | Batch merge process and online merge screen (V) | Phonetic/Soundex implied; rules per entry path (V partial) | No | U | U | U | U | | MemberClicks | No (U) | None | Manual "Find A Duplicate" (R) | Manual lookup only (R) | No | U | None; irreversible merge (R) | U | U | | YourMembership | Not assessed (U) | U | U | U | U | U | U | U | U | | Wild Apricot (Personify, now Momentive) | No (U) | None | Manual merge; no finder (V) | None; wishlist open since 2012 (V) | No | Email is the unique ID; multiple emails requested (R/V) | Archive and restore of merged contact (R) | U | Included in subscription (inferred) | | Aptify | No; address verification connector (R) | Melissa Data Webservice (rank 1), Accumail, Melissa Data (R) | On demand button (R) | Sample duplicate-check objects, off by default; checks on save (R) | No | U | U | U | Customer licenses Melissa | | Impexium (re:Members) | Not assessed (U) | U | Manual merge by admin (R, one customer's SOP) | U | U | U | "Merges" note with reason (R) | U | U | | Novi AMS | Not assessed (U) | U | Manual merge, field-level winner choice (R) | None found | U | U | U | U | U | | Glue Up, Raklet | Not assessed (U) | U | U | U | U | U | U | U | U | | Tessitura | Standard NCOA utility plus partner bureaus (V) | Loqate (NCOA, Deceased, Do Not Mail), Experian real-time (V/R) | Real-time at entry; NCOA per run (V) | Potential-duplicate queue; manual (R, forums) | No | U | Not documented | Do Not Mail service (V) | $1 per 1,000 records per service via consortium deal (V) | | Blackbaud Raiser's Edge NXT | **Yes** in-product: Address Finder, Deceased Finder (V); bureau: Email/Phone/Cell/Age/Gender finders (M) | USPS NCOALink/CASS, SSA death data "and other sources" (V); Identity Thread uses LiveRamp (M) | Every 90 days (address, deceased); continuous duplicate monitoring; batch file exchange for phone (V) | Scored fuzzy matches with confidence and min-likelihood filter; strict exact for auto-match (V) | No (U) | U | Confidence per pair; previous/next address updates visible (V); "acknowledge Blackbaud can process your data" at enablement (R) | Do Not Call flag (V, Blackbaud CRM); data-processing acknowledgement (R) | "Request pricing"; community says address updates up to 4/yr no extra cost (R) | | Bloomerang | **Yes** via TrueGivers for NCOA, deceased, demographics; wealth insights built in (V) | TrueGivers (NCOA); DonorSearch, Dataro as premier (V) | Nightly, schedulable; manual sync (V) | Real-time scan on change; name, address, email, phone; no published threshold; no auto-merge (V) | No | Shared email = duplicate signal; household option (V) | "NCOA" Address Note; Last Modified By TrueGivers; Archive Addresses switch; no approval (V) | Activation required; demographic fields opt-in (V) | NCOA bundled in standard (sources conflict); partners priced separately (V) | | Neon One | Not found (U) | U | U | Account match and duplicate scan (R) | U | U | U | U | U | | Salesforce platform (Fonteva/Nimble base) | No bundled data since Data.com retirement (R) | AppExchange vendors, e.g., ZoomInfo (V for ZoomInfo) | Instant Enrich on create; Scheduled Enrich recurring (V for ZoomInfo) | Fuzzy matching rules, alert/block, Data Cloud identity resolution with reconciliation (V) | ZoomInfo "Person Has Moved" (V) | U | Accuracy score, No Match reason; source values unaltered in Data Cloud (V) | U | ZoomInfo credits per contact (V); contracts ~$15-40K/yr (R) | | Momentive MomentiveIQ | "Cleansing and enrichment built into Agentic Workers" (M) | None named (M) | "As part of normal operation" (M) | "Resolves duplicate identities" (M) | U | U | Review-and-approve, confidence thresholds, audit trail (M) | Governed and auditable (M) | Core included, premium add-on (V as stated) | --- ### 5. Sections (b) Conclusions for Rhythm These are inferences from the evidence above, not facts about any vendor. Each cites the evidence it rests on. 1. **The gap is real and unclaimed in documentation.** No AMS documents a working, tenant-controlled enrichment feature for employer, title and organisation data. The only AMS statement is MomentiveIQ's marketing (2.16). Rhythm can credibly differentiate, but the evidence of demand inside AMS products is for _postal hygiene and duplicates_, not firmographics (Blackbaud, Bloomerang, Tessitura, iMIS add-ons). Plan the first release around what customers already pay for, and treat firmographic enrichment as the differentiator layered on top. 2. **Build an adapter layer, not a data asset.** Every successful pattern is a provider plugged in: Aptify's ranked list of verification providers, netFORUM's pluggable AddressCorrection option, Tessitura's Loqate and Experian, Bloomerang's TrueGivers, Blackbaud's bureaus. Salesforce's retirement of Data.com shows the risk of a platform owning the data. A provider-neutral interface with ranked providers per capability (the Aptify rank pattern) and per-tenant provider choice is the evidenced shape. 3. **Separate two classes of enrichment with different rules.** (a) _Postal and deceased hygiene_: mature, cheap (about $1 per 1,000 records per service via Loqate/Tessitura), 90-day cadence tied to USPS rules (Blackbaud and Bloomerang notes), safe to auto-apply with an archived prior value. (b) _Professional and organisational data_: no AMS precedent beyond ZoomInfo's flag, higher error cost, should propose rather than overwrite. 4. **Make write policy per field and visible.** ZoomInfo's "Complete if missing" versus "Overwrite" per mapped field, with the vendor advising overwrite only for fields dedicated to the provider, is the best-documented policy model. Add a third state, "propose for review" (MomentiveIQ markets the same idea; Blackbaud and Bloomerang have no approval step on enrichment writes). 5. **Provenance should be stronger than anything on the market.** Best precedents to combine: Salesforce Data Cloud's "source values unaltered, reconciliation rule chooses the winner, every value traceable"; Bloomerang's per-write note plus "Last Modified By: provider"; ZoomInfo's accuracy score and "No Match Reason"; Blackbaud's per-pair confidence plus fields-matched column. Store provider, retrieval time, confidence, prior value and the rule that applied for every enriched field. 6. **Do duplicate detection before and around enrichment, with strict versus loose matching split.** Blackbaud's design (strict exact name-and-email for automatic matching on forms; loose scored matching for human review) and Bloomerang's "confident only" philosophy are consistent. Enrichment writes should key on a strict identity match only, never on the loose review candidates. 7. **Make merges reversible.** No documented AMS merge can be undone (iMIS, Bloomerang, MemberClicks, Wild Apricot, netFORUM). Reversibility is a cheap, visible differentiator. Also learn from the merge side-effects documented by iMIS (AutoPay, event registration conflicts, engagement score recalculation), Personify (duplicate orders) and Bloomerang (automation and group membership changes): a merge touching money and automations needs a preview of consequences. 8. **Keep a permanent, pairwise "not a duplicate" memory, and avoid blunt exclusions.** iMIS and Blackbaud do this; Tessitura's "Void Merge" attribute blocks a record from all merges and is the anti-pattern (2.11). Add an "in the same household" outcome (Bloomerang's Create Household) and an "organisation relationship" outcome for AMS contexts. 9. **Employment change needs an AMS-specific design, not only a flag.** ZoomInfo gives a Yes/No/Uncertain "moved" field and explicitly leaves re-association to the customer. For an AMS, a job change affects organisation membership, directory listing, billing contact and portal permissions; the evidence shows nobody does this. A safe design raises a _suggested employment change_ task with provenance, and only an authorised staff member (or an explicit tenant rule) end-dates the old affiliation and creates the new one. 10. **Personal-email members need a first-class path.** Documentation across the AMS market is silent, and enrichment vendors differ on personal-email support. Evidence-based minimum: treat the personal email as an identity key for dedupe (Bloomerang, Data Cloud, iMIS), but do not assume a work-email lookup is possible; support multiple email types per person (the Wild Apricot wishlist ask), and allow name plus employer matching as an alternative provider input. 11. **Consent and activation are mostly tenant-level, if present at all.** Best precedents: Bloomerang requires activating NCOA and opt-in per demographic field; Blackbaud requires an acknowledgement that Blackbaud can process the data (Reported); Blackbaud flags Do Not Call. No product documents per-person suppression of enrichment, region-aware rules, or disclosure of which provider holds which data. This is open design space and also a risk area: the proposal should state its position on person-level opt-out and provider data-processing disclosure. 12. **Meter and cap cost per tenant.** The precedents are credits per contact with a per-run cap (ZoomInfo), per-record per-service pricing (Loqate $1 per 1,000), "request pricing" (Blackbaud) and bundling (Bloomerang, MomentiveIQ core with a premium add-on). A per-tenant budget cap with a pre-run estimate and a "dry run, no charge" mode is supported by ZoomInfo's design. Plan for sandbox runs to be metered too (ZoomInfo charges in sandbox). 13. **Visible data-health UX is a proven pattern.** Blackbaud's Data Health scorecard (Address health, Deceased health, Possible duplicates count, last and next update) and Bloomerang's notification bell with "hide for 1 hour, until next login, always" are lightweight and understood by nonprofit staff. Include a status that honestly shows when a provider is paused (Blackbaud's scorecard displayed a "paused" notice with no schedule). 14. **Treat agent-run enrichment as an extension of the same controls.** MomentiveIQ's model (full review-and-approve first, configurable confidence thresholds, graduate to autonomy, audit trail) is consistent with the controls above, but nothing public shows how it works. If Rhythm uses agents for enrichment, they should write through the same proposal, confidence and provenance path as any provider. --- ### 6. Sources, access dates, and pages not fetched All accessed 2026-10-22. #### 6.1 Sources read (page fetched and summarised) AMS and platform vendors - iMIS: https://documentation.advsol.com/imis/docs/managing-multiple-duplicates (Verified) - iMIS: https://documentation.advsol.com/imis/docs/merging-individuals-and-organizations (Verified) - iMIS: https://documentation.advsol.com/imis/v3.0/docs/managing-duplicate-contacts (Verified; older version) - iMIS: https://documentation.advsol.com/imis/docs/reporting-on-potential-duplicate-accounts-and-merged-accounts (Verified) - iMIS developer forum: https://developer.imis.com/discuss/68b6e97fae5fe7cfa88e0920 (Reported) - iMIS Zapier post: https://blog.imis.com/zapier-for-imis (Verified page; Marketing for value) - ASI Datascout announcement: https://news.advsol.com/datascout-strategic-partner (Verified page; Marketing) - CSI iValidate: https://www.csiinc.com/?p=2355 (Marketing) - Causeis Smart Duplicates: https://www.causeis.com/smartduplicates (Marketing) - GrowthZone press: https://www.growthzone.com/news-press/growthzone-accelerates-momentum ; https://www.growthzone.com/news-press/growthzone-membersuite-join-forces (Verified) - GrowthZone integration page: https://integration.growthzone.com/?p=57108 (Verified) - netFORUM Enterprise: https://documentation.mip.com/netforum-enterprise/2017.1/Content/CRM/Individuals/Combine_and_Merge_Duplicate_Records.htm ; https://documentation.mip.com/netforum-enterprise/2017.1/Content/SystemOptions/AddressCorrection.htm (Verified; 2017 version) - Fonteva: https://fonteva.com/?p=12312 (Fonteva 15 article, Verified absence of AI/dup content) - Nimble AMS help: https://help.nimbleams.com/help/live/duplicate-account-management ; https://help.nimbleams.com/help/live/configure-matching-and-duplicate-account-rules ; https://help.nimbleams.com/help/live/troubleshoot-duplicate-account-management (Verified) - Nimble Intelligence: https://www.communitybrands.com/company/news/community-brands-introduces-nimble-intelligence/ (Marketing) - Personify360 help: https://resource1.personifycorp.com/PersonifyOnlineHelp/7.7.0/mergedProjects/Customers/Duplicate_Customers_Overview.htm (Verified, partial) - Momentive: https://momentivesoftware.com/press-releases/personify-acquisition/ ; https://momentivesoftware.com/press-releases/momentiveiq-launch/ ; https://momentivesoftware.com/solutions/momentiveiq/ ; https://momentivesoftware.com/solutions/momentiveiq/ai-agent/ ; https://momentivesoftware.com/momentiveiq/ ; https://momentivesoftware.com/solutions/momentiveiq/ai-analytics/ (Verified existence of claims; Marketing for value) - Wild Apricot wishlist: https://forums.wildapricot.com/forums/308932-wishlist/suggestions/8827018-automate-finding-duplicate-contact-member-records (Verified; forum) - Naylor: https://www.naylor.com/naylor-member-data-platform/ (Verified page content) - Tessitura: https://www.tessitura.com/items/articles/news/2026/loqate ; https://www.tessitura.com/Items/Vendors/Loqate (Verified) Nonprofit CRMs and Salesforce - Blackbaud: https://www.blackbaud.com/nonprofit-direct-marketing/data-enrichment ; https://www.blackbaud.com/?p=3072 (Marketing) - Blackbaud help: https://webfiles-sc1.blackbaud.com/files/support/helpfiles/rex/content/data.html ; https://webfiles-sc1.blackbaud.com/files/support/helpfiles/rex/content/data-identify-duplicates.html ; https://webfiles-sc1.blackbaud.com/files/support/helpfiles/rex/content/bb-duplicate-records.html ; https://webfiles-sc1.blackbaud.com/files/support/helpfiles/crm/uk/40/Content/ADMPhoneFinderProcess.html (Verified; the last is Blackbaud CRM, not NXT) - Bloomerang: https://bloomerang.com/product/integrations-data-management ; https://help.bloomerang.com/en/articles/12632612-truegivers-national-change-of-address-ncoa-basics ; https://help.bloomerang.com/en/articles/12632564-report-on-national-change-of-address-ncoa-updates ; https://help.bloomerang.com/en/articles/12632743-review-and-merge-duplicate-constituents ; https://bloomerang.com/news/new-feature-real-time-duplicate-constituent-management (Verified) - Salesforce Trailhead: https://trailhead.salesforce.com/content/learn/modules/sales_admin_duplicate_management/sales_admin_duplicate_management_unit_2 ; https://trailhead.salesforce.com/content/learn/projects/explore-data-cloud-core-functionality/unify-your-data (Verified; training content, not reference docs) - ZoomInfo: https://tech-docs-library.zoominfo.com/enrich-for-salesforce-integration-guide.pdf (Verified; read locally from the saved PDF) #### 6.2 Sources relied on only through search-result summaries (Reported; page not opened or blocked) - Blackbaud Raiser's Edge NXT release note on Address Finder every 90 days and the data-processing acknowledgement: https://community.blackbaud.com/blogs/13/5744 (fetch returned only a title). - Blackbaud community threads on EmailFinder, AlumniFinder, NCOA exceptions: https://community.blackbaud.com/discussion/comment/249414 ; https://community.blackbaud.com/discussion/comment/282095/ ; https://community.blackbaud.com/discussion/comment/202050 ; https://community.blackbaud.com/discussion/comment/217868/ (fetch returned titles only). - Blackbaud case studies: https://www.blackbaud.com/files/pdf/directmarketing/CaseStudy_Houston_Community_College_Foundation.pdf ; https://www.blackbaud.com/?p=17236 - GrowthZone helpdesk (403 to fetch): https://helpdesk.growthzone.com/hc/en-us/articles/45934904318747-Managing-Merging-Duplicates ; https://helpdesk.growthzone.com/hc/en-us/articles/45935938310043-Import-Contacts ; https://supportwiki.growthzone.com/Contact_Management (connection refused) - MemberClicks: https://help.memberclicks.com/hc/en-us/articles/16319885298701-Merging-duplicate-member-profiles (403) - Wild Apricot: https://gethelp.wildapricot.com/en/articles/129-merging-contacts (fetch returned heading only) - Aptify (login required): https://aptifykb.atlassian.net/wiki/spaces/PK55/pages/15417383271/Integrating+With+Third-Party+Address+Verification+Systems ; https://aptifykb.atlassian.net/wiki/spaces/A6PK/pages/21935068686/About+the+Baseline+Persons+and+Companies+Sample+Duplicate+Objects - Novi AMS: https://help.noviams.com/en/articles/413615-merge-members-in-novi-ams (404 on direct fetch) - Impexium (ALTA SOP): https://purview.alta.org/p/merging-duplicate-records ; Preqin and CB Insights profiles - Tessitura forum threads on duplicates (communitytest.tessitura.com) ; Experian-Tessitura page https://tessitura.com/items/vendors/experian - Neon One Academy: https://academy.neonone.com/managing-data-in-neon-crm (403) - CSI iMerge and iDMS: https://docs.csiinc.com/imerge ; https://docs.csiinc.com/idms/ (search result only) - Ownership and deal news: https://thenonprofittimes.com/?p=212271 ; https://www.businesswire.com/news/home/20240709922273/en/ ; https://www.nonprofitpro.com/article/momentive-software-acquires-personify/ ; https://www.ropesgray.com/en/news-and-events/news/2026/01/ropes-gray-advised-on-momentive-softwares-acquisition-of-personify ; https://www.privsource.com/acquisitions/deal/vXSEVj (GrowthZone, Lead Edge 2023) ; https://madisonparkgrp.com/transactions/membersuite-an-arrowroot-capital-and-revolution-ventures-portfolio-company-has-been-acquired-by-growthzone - Associations Now job-board tracking: https://associationsnow.com/2021/09/membership-pro-tip-a-no-cost-retention-strategy/ - Salesforce Data.com retirement: https://gzconsulting.org/2018/04/17/sfdc-unplugging-data-com-connect/ ; https://clearbit.com/blog/how-to-transition-from-data-com ; https://developer.salesforce.com/docs/atlas.en-us.object_reference.meta/api/sforce_api_objects_leadcleaninfo.htm - Breeze Intelligence/Clearbit (pricing and credit conflicts): https://www.zeliq.com/blog/hubspot-data-enrichment ; https://marketbetter.ai/blog/clearbit-pricing-breakdown-2026/ - FullEnrich personal email: https://help.fullenrich.com/en/articles/9688091-data-provided-by-fullenrich ; Clay community: https://community.clay.com/x/support/pi4pnk74xecl/can-i-enrich-my-contacts-using-personal-emails - ZoomInfo cost range (biased vendor comparison): https://pipeline.zoominfo.com/sales/data-enrichment-tools #### 6.3 Pages I could not fetch or could not read (and why) | URL or topic | Result | | ------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | https://help.imis.com/Classic/Desktop/Customers/Customers_Overview.htm | 404 | | https://webfiles-sc1.blackbaud.com/files/support/helpfiles/rex/content/data-scorecard.html | 404 | | GrowthZone helpdesk "Managing Merging Duplicates" (Zendesk) | 403; kb.article 2003 URL 404 | | https://supportwiki.growthzone.com/Contact_Management | connection refused | | MemberClicks merge article | 403 | | Wild Apricot merge article | returned heading only (two attempts) | | Aptify Confluence KB pages | login redirect | | Novi AMS merge article | 404 | | https://academy.neonone.com/... (two URLs) | 403 | | Blackbaud community pages (several) | returned titles only (JavaScript shell) | | Blackbaud Data Health Center PDF (2011) | compressed PDF unreadable by the fetch tool; skipped as outdated | | https://help.salesforce.com/s/articleView?id=sales.managing_duplicates_overview.htm | CSS error page | | https://tessitura.com/services/partnerships/address-services | 404 (partner page reached through other URLs) | | https://bloomerang.com/product/data-health-agent and https://bloomerang.com/?p=40505 | 404 / no TrueGivers content; a Bloomerang page lists a "Data Health Agent" ("surfaces duplicates, gaps, and inconsistencies") which I could not open | #### 6.4 Leads not researched because the web-search budget ran out - MemberSuite, YourMembership and Fonteva primary documentation (help centres, release notes). - Impexium and Glue Up vendor documentation; Raklet; Higher Logic enrichment features. - Neon One duplicate and address-validation help articles. - Salesforce Help (help.salesforce.com) reference pages on duplicate management and Data Cloud identity resolution (only Trailhead training content was read). - AppExchange listings that pair enrichment tools (ZoomInfo, D&B, Melissa, DemandTools, Cloudingo, DataGroomr) with Fonteva or Nimble AMS. - Momentive release notes for netFORUM, Nimble AMS and YourMembership that would show whether MomentiveIQ enrichment is generally available and which products it supports. - Blackbaud pricing for NXT enrichment packages and the Address Finder "old address retained?" question. - Bloomerang "Data Health Hub" / "Data Health Agent" pages. - Any independent customer review of MomentiveIQ. ## 24. W2 evidence: how enterprise CRM/ERP platforms do data enrichment ## W2 evidence: how enterprise CRM/ERP platforms do data enrichment Research date: 2026-10-22 (all "accessed" dates below are 2026-10-22). Purpose: evidence for the "Rhythm 2.0 Data Enrichment" architecture proposal. Scope: Salesforce, HubSpot, Oracle NetSuite, Microsoft Dynamics 365 / Customer Insights, Zoho CRM, plus a short pass over Attio, Clay, Pipedrive, Freshsales, Copper, folk, Twenty, and the "agentic enrichment" pattern. ### 0. How to read this report #### 0.1 Claim labels | Label | Meaning | | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Verified** | Read on a primary vendor page (product documentation, help centre, vendor-run developer docs, vendor-hosted source code) during this research. | | **Reported** | Stated by a third party (consultancy, blog, review site, community post, Microsoft Message Center mirror), or taken from a search-tool excerpt of a vendor page that I could not open myself. Treat as a lead, not as proof. | | **Marketing** | Vendor sales or product-marketing page, or vendor-authored comparison. Describes intent, not verified behaviour. | | **Unavailable** | I looked and could not obtain the fact (page blocked, not found, or documentation silent). Silence in documentation is stated as "documentation silent", which is not the same as "feature absent". | | **Inferred** | My own reasoning for Rhythm. Never evidence. | #### 0.2 Limits of this research (read before relying on any cell) 1. Salesforce Help (`help.salesforce.com`) returned only a JavaScript loading shell, and `developer.salesforce.com` returned HTTP 403. Everything Salesforce says about reconciliation-rule semantics, Field Audit Trail and Lightning Data therefore comes from Trailhead (Salesforce's own training site, which I could read), search-tool excerpts of Help pages, and third-party write-ups. These are labelled Reported where they are not Trailhead. 2. The web-search quota for this session ran out near the end. I did not get to: HubSpot's Data Privacy FAQ "Enrichment Products" section, the HubSpot enrichment opt-out help page (404 at the guessed URL), Salesforce Data 360 consent enforcement behaviour, Salesforce's current Lightning Data/AppExchange status in 2026, and any Dataverse-native enrichment feature. These are marked Unavailable. 3. Pricing figures change often. Where sources conflict I show both and say so. 4. I did not test any product. Nothing here is observed behaviour of a running tenant. #### 0.3 One-paragraph answer No enterprise CRM or ERP in this set has a rich, native, first-party enrichment dataset of its own. Every one of them either (a) licenses a third-party dataset and brokers it (HubSpot: its own commercial dataset built on Clearbit plus third-party providers and public web; Zoho: public web crawling; Pipedrive: Surfe; Attio: unnamed providers; Microsoft: Microsoft search-derived affinities plus Experian/LiveRamp/Azure Maps), or (b) leaves enrichment to a marketplace partner (Salesforce, NetSuite). The interesting architecture is not where the data comes from. It is how the platforms keep enrichment from corrupting the system of record: separate output tables (Microsoft), provider-prefixed sidecar fields with a raw payload (Twenty), per-property overwrite rules (HubSpot), "manual values always win" (Attio, HubSpot, Pipedrive), survivorship rules plus a unified-link object (Salesforce, Microsoft), suggested-versus-automatic modes with citations (Microsoft Sales, Attio, Salesforce Agentforce), and success-only billing (Twenty, Clay, folk, Pipedrive, Attio). The weakest area across the board is provenance at field level and consent for enrichment of individuals; HubSpot's July 2026 terms-of-service reversal is the clearest market signal on the latter. --- ### 1. Salesforce (Sales Cloud + Data 360 + Agentforce + AppExchange) #### 1.0 Naming (current as of late 2026) - Data Cloud is now called **Data 360** (renamed at Dreamforce, October 2025). **Reported** (inceptasolutions.com via search excerpt, [S16]). Salesforce's own Trailhead and pricing pages already use "Data 360" and "Data 360 Unification" ([S1], [S5]) which **Verified** the current name in use. - Einstein-branded AI features are now presented under **Agentforce**; Agentforce Sales includes a **Prospecting agent** ([S9], [S10]). **Verified** (Trailhead Spring '26 release highlights) for the existence of "Agentforce Prospecting", **Marketing** for what it does. - **Data.com / Lightning Data**: Data.com Clean and Prospector were retired (no renewals after 2019-07-31, retired 2020-07-31, contact data removed 2021-02-01) and replaced via partnership with Dun & Bradstreet (Optimizer, Hoovers). **Reported** (D&B migration guide via search excerpt, [S13]). "Lightning Data" was the AppExchange packaging of data-service providers (each package = a custom object, an external object and a Data Integration Rule). **Reported** ([S13]). Whether Lightning Data is still offered in 2026 is **Unavailable**: the most recent item I found was a vendor help article dated 2025-12-06 (HG Insights) that still describes it, **Reported**. #### 1.1 Q1. What enrichment is native? - **No native firmographic or contact dataset.** Salesforce core CRM has no built-in provider data after Data.com's retirement. **Reported** (consistent across [S13], [S15]; [S15] is a ZoomInfo-authored guide, so competitor view). - **Data Integration Rules** (the Data.com Clean legacy mechanism) remain the platform mechanism by which a data-service package writes into records. **Reported** ([S13]). - **Data 360 identity resolution / unification** is native but is _matching and survivorship_, not enrichment. It creates a Unified Individual (or account) profile from multiple source records. **Verified** (Trailhead [S1]). - **Agentforce Prospecting** "enriches your existing Salesforce data and researches across existing and new accounts" and produces a prioritised prospect list in CRM and Slack. **Verified** that the sentence appears in Salesforce's Spring '26 release highlights ([S9]); Trailhead names no third-party providers and describes no write rules. **Marketing** for the rest ([S10]). - Data 360's own pricing page does not list enrichment as a product line; it only says unstructured-data processing, harmonisation/unification and similar actions are usage-metered. **Verified** ([S5]). #### 1.2 Q2. What depends on external providers? - AppExchange data providers: ZoomInfo, Dun & Bradstreet (Optimizer / Hoovers / D&B Connect), Demandbase, LinkedIn Sales Navigator, others. That these exist as AppExchange integrations is **Reported** ([S15] and several competitor listicles in search results). Pricing and coverage claims in those sources are **Marketing**. - ZoomInfo-authored page says its "GTM.AI" data is a built-in source for the Agentforce Prospecting agent once the customer connects their ZoomInfo entitlement in the Salesforce setup panel, delivered "via API and Model Context Protocol (MCP)", with "no separate enrichment pass". **Marketing** (vendor-authored, [S12]); it does not say whether the agent writes enriched values back to CRM fields. - Salesforce's own sales page says prospect research uses "any third-party data sources" and that account intelligence pulls from "web, third-party sources, conversations". **Marketing** ([S10]). - Clearbit: no longer sold as a standalone subscription (**Verified**, Clearbit's help centre/changelog via [H14]/[H16]); a competitor blog claims Clearbit stopped working with Salesforce, **Reported/Marketing** and unconfirmed by any Salesforce or HubSpot official statement I could open. #### 1.3 Q3 and Q4. When does enrichment happen, and in which modes? | Mechanism | Timing | Mode | Label | | ------------------------------------------------------ | ------------------------------------------------------------------------------------------- | -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | | Data Integration Rule (Lightning Data / D&B Optimizer) | Periodic, at a frequency set by the data-service provider; also user-initiated record check | Automatic scheduled + manual | **Reported** ([S13]); I could not confirm the manual "Check for New Data" action (documentation excerpt silent) | | Data 360 identity resolution | Scheduled/automatic ("Run jobs automatically" toggle); manual "Run Ruleset" | Batch over the whole ruleset; re-evaluates incoming data | **Verified** (Trailhead, [S1]); a third-party guide says it "runs on a scheduled basis" without frequency, **Reported** ([S4]) | | Agentforce Prospecting | Continuous ("24/7"), agent-driven | Autonomous or suggestive | **Marketing** ([S9], [S10]) | | Duplicate Rules / Matching Rules | On save (create/edit); alert or block | Synchronous prevention | **Verified** (Trailhead, [S7]) | Trailhead recommends turning **off** "Run jobs automatically" on a Data 360 ruleset to reduce cost by limiting jobs, which tells you the platform's own authors consider auto-runs a cost lever. **Verified** ([S1]). #### 1.4 Q5. Conflicting field values Three separate mechanisms, none of which is "enrichment-aware": 1. **Data Integration Rules**: "Except for geocode rules, data integration rules never overwrite your data; data is added only to blank fields." Geocode rules overwrite. **Reported** (excerpt of Salesforce help content mirrored by gyde.ai, via search, [S13]). The first line is the fill-empty-only default. 2. **Data 360 reconciliation rules** (survivorship for single-value fields on the unified profile) **Verified** at concept level on Trailhead ([S1]): options are frequency, recency and source-object priority. More detail **Reported** from a search excerpt of the Help article "Identity Resolution Reconciliation Rules" ([S2]): - _Most Recent / Last Updated_: takes the value from the source record with the latest Last Modified Date on the primary DMO; only available if that field is mapped; identical timestamps fall back to alphabetical order. - _Most Frequent_: highest-count value; with "Ignore Empty Values" nulls are never chosen. Tie-break is not confirmed (a third-party says Last Modified Date; I could not confirm). - _Source Priority_: ordered list of source objects; with Ignore Empty Values it takes the highest-ranked non-null. - Rules are set at object level with field-level overrides. They do not apply to contact points (email, phone, address), which are all kept on the unified profile. - Salesforce recommends Source Priority for ID fields to keep them stable. 3. **Duplicate Rules / Matching Rules** (prevention): matching rules find potential duplicates (Exact or Jaro-Winkler fuzzy comparison per field); duplicate rules choose the action (block or alert, with a report of duplicates created after users bypass warnings). A duplicate rule can include up to 3 matching rules; each object can have up to 5 active matching rules. **Verified** ([S7]). Merge is not covered on that page. Agentforce Sales sales page describes "suggestive mode" (reps "review, edit, and approve updates") versus "autonomous mode" for opportunity updates. **Marketing** ([S10]). #### 1.5 Q6. Provenance - **Unified link objects** in Data 360: "source data is not changed", a link object connects a unified profile to its source records, "you can trace each profile element back to its origin". **Verified** ([S1]). This is the strongest provenance primitive in the set for _identity resolution_, but it traces which source records fed a profile, not which provider supplied a field. - **Field History Tracking** (standard): up to 20 fields per object; changes kept about 18 months in the UI and 24 months via API; **Field Audit Trail** is a paid add-on that archives to a big object (`FieldHistoryArchive`) and is API-only. Figures **Reported** (SalesforceBen/Calibermind via search, [S14]); a search excerpt of the developer doc says archived data is kept until deleted. The 20-versus-60-versus-200 field ceilings conflict between sources. - **Salesforce Unified Lineage** (Data 360): visual view of data movement from source to activation. **Verified** that Spring '26 release highlights list it ([S9]); no detail. - Nothing I found describes a per-field "this value came from provider X at time T with confidence C" on a CRM field. **Unavailable** (documentation silent). #### 1.6 Q7. Permissions, consent, privacy - Consent model (Salesforce Consent Data Model, usable with Data 360): four layers: general consent (Individual / Party Consent), engagement-channel consent (Contact Point Type Consent), contact-point consent (Contact Point Consent, per specific email/phone and per Data Use Purpose), and communication-subscription consent. Each **Data Use Purpose** has a "Can Data Subject Opt Out" flag and links to a **Legal Basis** record, which drives right-to-be-forgotten decisions. **Verified** ([S8]). The Individual object anchors all consent to one person. **Reported** (third-party excerpt, [S8b in section 8]). - The Trailhead page does **not** describe enforcement (e.g. "Don't Process" flags suppressing enrichment) and I did not find a Data 360 page that links consent to enrichment. **Unavailable.** - Data residency/hyperscaler hosting for Data 360: **Unavailable** (not researched; quota). #### 1.7 Q8. Pricing and metering - **Data 360 credits** (**Verified**, [S5]): Flex Credits $500 per 100,000 credits; or per-profile SKUs $240 per 1,000 profiles/year (Profiles) and $420 per 1,000 profiles/year (Enterprise Profiles, includes 2 Flex Credits per profile). Ingestion "is free, including zero-copy"; unification, unstructured processing, segmentation, queries and real-time use are usage-metered. A pricing calculator and rate card exist. - **Rate card** (**Reported**, third-party blog dated March 2026, [S6]; the blog says its table comes from Salesforce's pricing page and the Data 360 Platform Services Rate Sheet, which I did not open): profile unification 100,000 credits per 1,000,000 rows (batch or streaming); external ingestion 2,000 per million rows batch; segmentation 20 per million rows; internal Salesforce CRM pipeline free. At $500 per 100k that is about $500 to unify 1M rows. Sandboxes at a 20% discount (unsourced). Repeat runs count only new or modified source profiles (**Reported**, [S6] excerpt of a Salesforce use-case). - **Agentforce**: $500 per 100k Flex Credits; a standard action uses 20 Flex Credits (about $0.10 per action, my arithmetic from the page); Agentforce add-ons $125 per user per month unmetered; Agentforce user licence $5 per user per month plus metered credits. **Verified** ([S11]). - Third-party provider data is billed by the provider (ZoomInfo, D&B), not Salesforce. **Inferred** from how AppExchange listings work; not directly verified. - Contradictions: one source quotes a Data 360 Starter at $60,000/year, another $108,000/year for 1 million credits. **Reported**, unresolved; ignore. #### 1.8 Q9. Rhythm: adopt / avoid (all **Inferred**) Adopt: - A **reconciliation-rule vocabulary** with exactly three strategies (source priority, most recent, most frequent), settable per object with per-field override, and with an "ignore empty values" switch. It is small, well understood, and covers what survivorship needs. - A **unified link / lineage object** that is separate from the golden record and never mutates source rows. - **Fill-empty-only as the default** for any provider writing to a master field (Data Integration Rule behaviour). - **Legal Basis and Data Use Purpose** as first-class records so enrichment of an individual can name its lawful basis. Avoid: - Salesforce's split between "matching" (Duplicate Rules, synchronous), "unification" (Data 360, batch, per-row credits) and "enrichment" (AppExchange, per-vendor semantics). Three engines with three rule languages is the cost of retrofit. Rhythm should have one rule vocabulary used by all three. - Per-row, per-run metering of matching. The $500-per-million-rows unify price makes re-running a ruleset an economic decision, and Trailhead's own advice to disable auto-runs shows the user cost. Rhythm's matching should be cheap enough to run on every write. --- ### 2. HubSpot (Data Hub / "data enrichment", formerly Breeze Intelligence, formerly Clearbit) #### 2.0 Naming and history - Dec 2023 HubSpot acquired Clearbit; "Breeze Intelligence" unveiled at INBOUND 2024; free Clearbit tools sunset 2025-04-30; Clearbit Logo API sunset 2025-12-01; Breeze Intelligence "effectively replaces" the free platforms with enrichment of contact and company records (40+ attributes, continuously refreshed), buyer intent, market sizing and form shortening. **Verified** (Clearbit/HubSpot blog dated 2025-02-13, [H14]). Clearbit's help centre says it is "no longer offering new standalone Clearbit subscriptions" (**Reported**, search excerpt, [H16]). - **Rename**: HubSpot's current knowledge base calls the feature "data enrichment" (Data Management > Data Enrichment). One consultancy states "Data enrichment (formerly Breeze Intelligence)" and that other Breeze items were renamed too (Breeze Copilot > Breeze Assistant, Breeze Agents > Agent Hub), but it explicitly found no formal HubSpot announcement. **Reported** ([H12]). Another source says the Breeze name still applies. I treat the product name as unsettled; "data enrichment" is the safe name. #### 2.1 Q1. What is native? All of the following are first-party HubSpot features (**Verified** unless marked): - **Contact and company enrichment** from "HubSpot's commercial dataset, third-party providers, and public internet information". Inputs: contacts need first name, last name, work email; companies need company domain. ([H1]) - **Property mapping** per object with per-property overwrite rules ([H2]). - **Enrichment Activity tab**: chronological log of enrichment events, with Source (e.g. Continuous, Manual), User ("HubSpot system" for automatic), record count and type, grouped into 3-hour blocks ([H2]). - **Form shortening** (progressive profiling: hides fields HubSpot already knows for a visitor from a known company). **Verified** only as a feature that "collects no data from your use" ([H1]); the "no credits since 2025-09-03" claim is **Reported** ([H12]). - **Buyer intent** (company-level web-visit identification and signals). **Verified** that the change-source taxonomy contains "Buyer-Intent" ([H4]); details of how it identifies companies (IP-to-company) are **Reported** ([H13]). - **Data quality tools / command center**: duplicate detection ("Duplicates are detected by Breeze"), formatting-issue detection with rule-based auto-fix, property insights, anomaly alerts, enrichment-gap scan with "enrich all records in a selected segment". **Verified** ([H5]). - **Conversational enrichment** (fills contact data from email conversations, flags out-of-office returns and job departures, creates contacts mentioned in threads). **Verified**, public beta ([H7]). - **Data agent / smart properties** (AI-generated property values from web research, company website, existing property data or call transcripts, per-prompt-per-record). **Verified** ([H8]). Also AI actions in workflows. - Data Hub marketing page: "missing fields are filled from email conversations, meeting notes, and verified third-party sources ... without anyone lifting a finger". **Marketing** ([H6]). #### 2.2 Q2. External providers - Which providers sit behind "third-party providers" is **Unavailable** (HubSpot does not name them in the pages I read). Clearbit (owned by HubSpot) is the heritage source. **Reported** ([H13]). - Quality claim: data "is weighed and scored against sources, and a human quality assurance team reviews it." **Marketing** by a vendor help page ([H1]); no accuracy metric. - **Data contribution back**: "Limited business contact data from verified enrichments may be contributed back to the commercial dataset. Continuous-only enrichment adds no data to that dataset." **Verified** ([H1]). This is the mechanism behind the July 2026 controversy (see Q7). #### 2.3 Q3 and Q4. When, and which modes All **Verified** ([H1], [H2], [H3]) unless marked. | Mode | Trigger | Scope | Overwrite behaviour | | --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | | Automatic ("Automatically enrich new records") | Record created by form, integration or manually. Off by default; Super Admin enables for contacts and companies separately. | New records | Fills only blank properties; "doesn't overwrite values set by a user or another system" | | Continuous ("Continuously enrich existing records") | Monthly, when new data is available | Previously enriched records | Updates properties previously filled by enrichment; **stops for a property once a user or another system edits it** | | Recently engaged | Record's Last engagement date updates | Contacts and companies with email/meeting/activity | Per mapping rules | | Manual, single record | "Enrich record" in the Intelligence tab; user ticks the properties | One record | User chooses fill or overwrite | | Bulk, index page | "Enrich records" | Maximum 100 records at a time | Per mapping rules | | Bulk, segment | "Enrich segment" | Segment members | Per mapping rules | | Import | "Enrich records" checkbox during import | Imported records | Not specified | | Workflow | "Enrich record" action | Enrolled records | Can overwrite regardless of mapping rules | | Conversational enrichment | Email content | Contacts | Bypasses mapping rules (**Verified**, [H7]) | Third-party warning: stacking automatic enrichment on form submits, creation, imports and updates can drain credits faster than expected. **Reported** ([search excerpt of a practitioner blog; see H13]). Note this predates HubSpot making standard enrichment credit-free (see Q8). Eligibility: the revopshq analysis says enrichment "skips records without a business email or company domain." **Reported** ([H12]). #### 2.4 Q5. Conflict handling (the most complete in this set) - **Mapping tab, per property, three overwrite rules** (**Verified**, [H2]): _Fill empty values only_ (HubSpot suggests it for first name and last name "to protect manually-entered contact names"); _Fill empty values and overwrite existing values_ (suggested for job title or seniority); _Do not fill any values_ (blocks enrichment for that property). Changes apply to future enrichment only. - **Overwrite incorrect enrichment values** (setting, only with continuous enrichment on): replaces previously enriched values with **nulls** when the source data is inaccurate, and does not touch values set manually, by workflows or by integrations. **Verified** ([H2]). This is the only "provider retraction" semantic I found anywhere. - **Exceptions that bypass the rules**: the workflow "Enrich record" action (when set to overwrite) and conversational enrichment. **Verified** ([H2], [H7]). The conversational-enrichment page describes no review or approval step. **Verified** (silence), a design smell: an AI path that can write regardless of the governance the admin configured. - Duplicates: Manage Duplicates tab; "Review" merges and lets the user choose which properties carry over; "Reject" keeps both and the pair does not resurface; alert fires when duplicates exceed a custom daily limit checked every 24 hours (Data Hub Professional and Enterprise only for alerts and formatting automation). **Verified** ([H5]). #### 2.5 Q6. Provenance - **Property history** shows old value, new value and a **Source** column; the same list populates the **Record Source** property. Change sources include CRM UI, CRM Bulk Action, Batch Update (API), Import, Integration, Data Integration, Workflow, Custom Agent, HubSpot AI, Buyer-Intent, Data Quality (duplicate management) and **HubSpot Company Insights**. **Verified** ([H4]). The page lists no distinct "data enrichment" change source other than HubSpot Company Insights (the page links to the data enrichment guide); and it does not document retention or API access (`propertiesWithHistory`). I treat "what exact source label an enrichment write carries" as **Unavailable**. - **Enrichment Activity tab** (see Q1) records who or what ran an enrichment, count and type, but not a per-field value-level lineage. **Verified** ([H2]). - A consultancy suggests a separate "Data Source" field for audit; that is a workaround, which tells you the native model is not sufficient. **Reported**. #### 2.6 Q7. Permissions, consent, privacy - **Permissions**: Super Admin configures enrichment settings and user access; enriching records needs "Data enrichment access" (a Data Management permission) or Super Admin. **Verified** ([H1], [H2]). - **Turning off**: disable automatic and continuous settings and remove "Data enrichment access"; Super Admins can still enrich manually; existing enriched data stays. **Verified** ([H1]). - **Individual opt-out**: an opt-out form removes a person from the commercial dataset (personal data only, not account data); the CRM contact gets an "enrichment opt out" property label and Super Admins are notified. A privacy request form exists for access or deletion. **Verified** ([H1]). - **Customer obligations**: customers must comply with applicable laws including "providing notices and opt-out mechanisms" and the page points to the "Enrichment Products" section of HubSpot's Data Privacy FAQs, which I could not read. **Verified** (page text); FAQ **Unavailable**. The page does not name GDPR. A third-party guide argues enrichment is a separate GDPR processing activity because data is not collected from the individual and recommends documenting lawful basis and updating privacy notices. **Reported**. - **Email deliverability data**: turning off the AI Model Training switch opts out of HubSpot using your data for deliverability. **Verified** ([H2]). The "Opt out of free company name enrichment" setting stops the default Company name from being populated on new company records. **Verified** ([H2]). - **July 2026 terms-of-service episode** (**Reported**, CMSWire [H10] and Concept Ltd [H11]; HubSpot's own announcement not opened): on 2026-07-01 HubSpot announced that from 2026-08-04 enrichment-derived data (business contact details, employer information, deliverability signals) could be shared with other customers, with customers **opted in by default** and required to opt out before the date to stay out of sharing and AI training; opting out meant losing access to others' contributed data. On 2026-07-05 HubSpot cancelled it after backlash; its chief product and technology officer said they "will not move forward", and committed that any future enrichment capability using customer data will be "fully and transparently opt-in". HubSpot has not said whether a modified version returns. A HubSpot Community thread (HTTP 403 to me; **Reported** via search excerpt) says opt-out applies go-forward and does not remove data already contributed. - **Inferred significance**: this is the strongest 2026 market evidence that customers (here, businesses managing their contacts) treat "your CRM data feeds a shared enrichment pool" as a trust breach. Associations are even more sensitive because their members are individuals who joined for the association, not for a data broker. - AI settings (Super Admin): generative AI, CRM data, conversation data on by default; files data off; AI model training on unless the account holds Sensitive Data. **Reported** ([H12]). #### 2.7 Q8. Pricing and metering - **Standard data enrichment does not use HubSpot Credits.** **Verified** ([H1]: "Data enrichment does not use HubSpot Credits. Smart properties do"). Enrichment is available with Starter, Professional or Enterprise tiers of Marketing, Sales, Service, Data and Content Hub (Smart CRM and Revenue Hub: Professional or Enterprise). **Verified** ([H1]). One consultancy reports a source conflict on the Smart CRM Starter tier. **Reported** ([H12]). - **Metered features** (**Reported**, [H12], as of 2026-09-27): HubSpot Credits are included per month by tier (500, 3,000 or 5,000 for Starter, Professional, Enterprise; 500, 5,000 or 10,000 with Data Hub or the Customer Platform bundle); overage $0.010 per credit or $10 per 1,000. Smart properties: 10 credits per prompt per record, **filled or not**. Workflow AI actions: 10 credits each. Buyer intent: 10 credits per company tracked monthly; 50 per custom signal (beta). Agents: 1 to 1,000 credits per unit of work. Conversational enrichment: free during public beta (**Verified**, [H7]). - **Legacy standalone Breeze Intelligence credit packs** (100 credits $30; 1,000 $150; 10,000 $700; one credit per enriched record, no rollover). **Reported** (HubSpot-hosted undated guide via a third-party summary, [H13]); and HubSpot "transitioned legacy Breeze Intelligence credit packs into the broader HubSpot Credits framework in June 2025" (**Reported**, a single source). Treat as historical. One third-party review site (2026-09-27) still quotes different standalone prices ($45 to $50 per 100 credits); these are **Reported** and inconsistent. - Bulk dialog shows record count and available credits before a smart-property run. **Verified** ([search excerpt of H9]); I could not open the page itself, so **Reported**. #### 2.8 Q9. Rhythm: adopt / avoid (all **Inferred**) Adopt: - The **per-property, three-value overwrite rule** (fill empty / overwrite / never) configured in a mapping screen, defaulting to fill empty. - **Continuous refresh that stops touching a property once a person or another system edits it.** This is the cleanest statement of "manual edits win" that I found, and it needs provenance per field to implement. - A **"retract" semantic**: when the provider later says a value is wrong, null it only if it is still provider-owned. - An **enrichment activity log** (who/what ran it, when, how many records, which type). - **Contact-level opt-out marker** that suppresses future enrichment and notifies an admin. - **Opt-in default**: automatic enrichment off until an admin switches it on, separately for people and organizations. Avoid: - Writers that **bypass the mapping rules** (workflow overwrite action, conversational enrichment). Every writer, human, workflow or AI, should pass through the same policy gate. - **Data contribution back to a shared pool**, even as a default-on term. Rhythm tenants are associations; never do this without per-tenant, explicit, revocable opt-in, and probably not at all. - Charging a credit for an AI prompt "filled or not" (smart properties): it punishes the customer for a miss and is the reverse of the success-only norm. - Making provenance a free-text "Data Source" field the admin must add. --- ### 3. Oracle NetSuite #### 3.1 Q1. Native enrichment - **None found.** I found no NetSuite-native contact or company enrichment feature, and no SuiteApp from ZoomInfo, D&B, Clearbit or Apollo in the NetSuite marketplace. **Unavailable** (negative evidence: absence in several searches; I could not browse the SuiteApp marketplace itself). - **Oracle DataFox** (acquired 2018-10) was the Oracle enrichment asset; it is described as sunset in 2023. **Reported** (TrustRadius listing, [N3]). Oracle's own April 2021 Cloud Readiness note describes "DataFox for Eloqua (Account Enrichment)", an Eloqua (marketing) feature, not NetSuite. **Reported** (search excerpt, [N4]). I found no NetSuite integration of DataFox. - NetSuite's own data-hygiene features are **duplicate detection and record merge** only (below). #### 3.2 Q2. External providers - Enrichment is reached through **third-party integration platforms** (Stacksync two-way sync, Workato, Zapier connectors between NetSuite and ZoomInfo). **Reported** ([N5]); vendor pages are **Marketing**. ZoomInfo's own comparison page lists NetSuite among supported CRMs. **Marketing**. - D&B on NetSuite: **Unavailable**. #### 3.3 Q3/Q4. Timing and modes - Not applicable natively. Via connectors: scheduled or trigger-based sync (**Marketing**). - **Duplicate detection** (Verified, Oracle NetSuite online help, [N1]): enable under Setup > Company > Enable Features, then configure criteria at Setup > Company > Duplicate Detection. After saving preferences NetSuite runs an **initial search** across existing records (can take long on big accounts) and until it finishes there are no alerts or merges. Record types: customer, contact, partner, vendor; default criterion: email address. Flags show as a notice with a link to a resolution page; bulk review via Lists > Mass Update > Entity Duplicate Resolution; permission "Duplicate Entity Management" gates the manage page. **The page does not say whether alerts fire on CSV import or API saves** (documentation silent). Two third-party sources claim UI-only; one says Latin/Cyrillic-only. **Reported**, unreconciled. #### 3.4 Q5. Conflict handling - **Merge** (**Verified**, [N2]): for each target field, if blank the source value is copied in; if populated the target value is kept (not updated). The source customer record is **deleted**. Merging cannot be reversed. In OneWorld, subsidiary and VAT/tax-registration constraints apply (two Oracle pages disagree slightly on how strict the subsidiary rule is; I report both as written). NetSuite recommends restricting merge rights because audit rules often discourage merging transaction-bearing records ([N1]). - This is "fill-empty, keep-primary" survivorship with no choice of winner per field. #### 3.5 Q6. Provenance - After a merge, the target's System Information subtab shows a System Note with Context "Duplicate Resolution". **Verified** ([N2]). System Notes in general are NetSuite's audit trail (documentation on retention not read). No source-of-value concept exists. **Unavailable** beyond that. #### 3.6 Q7. Permissions, consent - Duplicate Entity Management permission; recommendation to restrict merge rights to Administrator or specific roles. **Verified** ([N1]). - No enrichment consent features (none to describe). Consent for marketing is outside this scope. #### 3.7 Q8. Pricing - No native metering. Third-party connectors billed by the connector and data vendor. **Unavailable** (not researched). #### 3.8 Q9. Rhythm: adopt / avoid (**Inferred**) - Adopt: the **explicit initial-search-before-alerts step** (build the match index before enforcing) and the **permission to merge as a separate privilege**. - Avoid: **irreversible merges that delete the source record** ("merged customer records can't be reversed"). Rhythm should keep a tombstoned source with an undo window, because wrong merges of member records have real consequences (dues, voting eligibility, history). - NetSuite shows the ERP pattern: do not own enrichment; expose a clean integration surface. For Rhythm that reinforces a provider-adapter boundary rather than a built-in dataset. --- ### 4. Microsoft Dynamics 365 (Customer Insights - Data, Sales, Dataverse) #### 4.0 Naming - "Dynamics 365 Customer Insights - Data" (unification, enrichment, segments) is the successor to the 2019-2020 Customer Insights; "Customer Insights - Journeys" is the marketing orchestration product. **Verified** (Learn pages [M1], [M8]). - Copilot, agents and "Copilot for Sales" are the AI surface inside Dynamics 365 Sales. **Verified** (release plan [M13]). #### 4.1 Q1. Native enrichment **Customer Insights - Data** (**Verified**, [M1], [M2]): the Enrichment hub's Discover tab lists: | Enrichment | Provider | Status in docs | | ------------------- | ---------------------- | ------------------------ | | Brands | Microsoft | GA (no "preview" marker) | | Interests | Microsoft | GA | | Enhanced addresses | Microsoft | GA | | Demographics | Experian | Preview; US only | | Identity (AbiliTec) | LiveRamp | Preview; US only | | Location data | Microsoft Azure Maps | Preview | | SFTP custom data | Customer-supplied file | Preview | - Brands/Interests: Microsoft predicts affinity (four levels, 0-100 score) and share of voice from online **search-volume** data of people with similar demographics (age, gender, location); needs at least date of birth or gender, plus country and city/state or postal code; up to five brands or interests per enrichment; six countries. **Verified** ([M3]). Note: the output is a **statistical inference about a demographic segment**, not a fact about the person, and Microsoft's page says so ("based on data about people with demographics similar to your customers"). - "Microsoft enrichments are included with a Customer Insights - Data subscription." **Verified** ([M1]). - Output is stored in **separate enrichment tables** (e.g. `BrandAffinityFromMicrosoft`, `InterestAffinityFromMicrosoft`) linked to the unified profile by CustomerId, "preserving the original profile data." **Verified** ([M1], [M3]). - **Data-source enrichment before unification**: Enhanced addresses and LiveRamp AbiliTec IDs can be applied to a source table to improve match quality. **Verified** ([M7]). **Dynamics 365 Sales** (**Verified** existence; **Marketing** for intent): - 2026 release wave 1 overview says Sales "continuously enriches data" and that "capabilities that automatically validate and enrich your data" help lead qualification ([M13], archived as of October 2026). - **Automatic CRM Record Linking and Suggested Updates** (Message Center MC1420564, published 2026-07-08, GA 2026-08-07): detects accounts, contacts and leads mentioned in emails and meetings; suggests field updates such as job title, phone, industry, employee count; identifies new stakeholders and suggests creating or linking Contact/Lead records; suggestions are presented for review and each **cites the interaction it came from**. **Reported** (Message Center text via mc.merill.net mirror, [M14]; the Microsoft Learn page itself I did not find). - **AI-based data enrichment for opportunities** (MC1254454, published 2026-03-17, GA 2026-04-17): analyses sellers' emails, flags gaps, suggests updates. Admins define which opportunities are enriched and choose **automatic updates or manual review-and-approval**; sellers read the context, **accept or reject** (manual) or **revert** changes the agent made (automatic). **Reported** ([M15], Message Center mirror). - **LinkedIn Sales Navigator** integration: requires a Microsoft Relationship Sales subscription or a standalone Sales Navigator Advanced/Advanced Plus licence; works with customer engagement apps, not a bare Dataverse org. **Reported** (search excerpt of Learn, [M17]). Microsoft does not list it as an enrichment of Customer Insights - Data. - **Dataverse / Power Platform** native enrichment: **Unavailable** (not found, and not specifically researched before search quota ran out). Dataverse's own hygiene feature is duplicate detection rules (below). #### 4.2 Q2. External providers - Experian (consumer demographics: household size, income; US only), LiveRamp (identity graph), HERE Technologies (address/geo; mentioned in Microsoft's overview as needing "separate provider subscriptions"), Leadspace (B2B company and contact data), Azure Maps, SFTP file. **Verified** ([M1], [M2], [M4], [M5], [M6]). - **Leadspace**: still documented but labelled preview, "for customers previously onboarded to B2B environments. New customer requests for B2B mode aren't being accepted." **Verified** ([M5], page dated 2026-07-10). It no longer appears on the Discover tile list in [M2], though the progress-details text still lists Leadspace and HERE as "supported" for status tracking. Inferred reading: Microsoft has frozen the B2B route and is not actively expanding B2B enrichment. - A 2020 Leadspace partnership press report said its licence is priced by records under management. **Reported** (martechvibe, search excerpt). #### 4.3 Q3/Q4. Timing and modes - **Enrichment runs on demand or on the system refresh schedule** ("Run" a selection; or Settings > System > Schedule); inactive enrichments are skipped by scheduled refresh; progress/status view for Experian, Leadspace, HERE, SFTP, Azure Maps with ability to cancel a job. **Verified** ([M2]). - Enrichment can be scoped to a **segment** (only those profiles are sent); several enrichments of the same type can use one connection; some have per-type limits shown on the tile. **Verified** ([M1]). - No per-record on-create trigger in Customer Insights - Data. Unification re-runs on the same schedule. - Sales suggested updates run continuously off email/meeting signals (**Reported**). #### 4.4 Q5. Conflict handling Two layers, both **Verified**: **Unification (merge) rules** ([M11]): - A merged column chooses the winner by **Importance** (default; first non-null in source-rank order), **Most recent** or **Least recent** (needs a date or numeric field in every participating table). - **Combine a group of fields** so related fields travel together. Microsoft's own example: without grouping, a matched person's address mixes Loyalty-table street with Online-table suite, because the winner is chosen per field and a null in one source lets the next source fill it. Grouping chooses one source's whole address. - Fields can be renamed, separated, excluded, reordered. - Households and custom **clusters** can be generated from rules (same last name + address), adding a shared cluster ID. - **CustomerId** is stable across runs except on merge (one of the prior IDs survives; `PreviousCustomerId` lists the others, semicolon-separated) or on split (the new profile's `PreviousCustomerId` points to its origin) ([M9]). Optional stable-key configuration using up to five immutable inputs such as a government ID (not email/phone/address). **Match rules** ([M10]): - Pairs of tables in a **match order**; the first is primary; rules are ordered, later rules tried only if earlier fail; conditions inside a rule are ANDed. - **Normalisation options**: numerals, symbols, lowercase, phone (country-code-aware), **name** (500+ variants, e.g. debby -> deborah), **address** (street -> st), **organization** (about 50 noise words such as co/corp/ltd), Unicode to ASCII, whitespace, **alias mapping** (custom list that must be treated as equal), **custom bypass** (values that must never match, e.g. dummy phone). - **Precision**: basic Low 30 / Medium 60 / High 80 / Exact 100, or a custom percentage. - **Exceptions** (post-match veto, e.g. twins with different first names), and uploaded override tables: **Always match**, **Never match**, **Bypass**, **Alias**. - Deduplication rules choose the best row within a table first; non-winner "alternate rows" still participate in matching. **Enrichment vs master**: enrichment output never overwrites unified profile fields; it lands in separate tables joined by CustomerId, and the user decides what to use in segments ([M1]). This is a "no conflict by construction" design, at the cost of two places to look. **Dataverse duplicate detection rules** ([M12], Learn, **Verified**): default rules exist for accounts and contacts only; criteria operators include Exact Match, Same First Characters, Same Last Characters; options for case sensitivity, ignoring blank values, excluding inactive records; a matchcode is generated per record on publish; at most **five published rules per base record type**; bulk system jobs find duplicates; merge is a separate action. #### 4.5 Q6. Provenance - Enrichment tables are named by provider and attribute family (`...FromMicrosoft`), carry the enrichment configuration (name, connection) and last-refresh time; the results page shows number of customers **enriched by field** (coverage). **Verified** ([M2], [M4]). - CustomerId lineage via `PreviousCustomerId`. **Verified** ([M9]). - Suggested updates **cite the originating email or meeting**. **Reported** ([M14]). - No field-level "which source supplied this unified value" shown in the pages I read; the merge docs describe rank rules but not a per-value stamp. **Unavailable.** #### 4.6 Q7. Permissions, consent, privacy - **Connections**: only administrators create enrichment connections and they "give consent to data transfers"; administrators and contributors can use them; admins can remove a connection at any time and must first detach, delete or deactivate dependent enrichments. Contributors can create or edit enrichments (Contributor or Administrator role needed). Secrets can live in the customer's own Key Vault. **Verified** ([M2], [M8]). - **Privacy statement** shown on each connection: enabling transmission "allow[s] transfer of data (including personal data) outside of Customer Insights. Microsoft transfers such data at your instruction, but you're responsible for obtaining consent to use any data and ensuring that the third party meets any privacy or security obligations you might have." **Verified** ([M8]). The legal burden is explicitly the customer's. - Experian match inputs require at least one of name+address, phone or email; Leadspace uses company name/website/location. **Verified** ([M4], [M5]). - Residency, retention of enrichment tables, and data-subject deletion handling: **Unavailable** (not in pages read). #### 4.7 Q8. Pricing and metering - Microsoft enrichments included with the subscription; Experian, HERE, Leadspace, LiveRamp require separate provider subscriptions bought from the provider. **Verified** ([M1], [M4], [M5], [M6]). - Customer Insights - Data is licensed by **tenant per month with included capacity**: base and attach offers include 100,000 Unified People and 10,000 Interacted People; extra capacity in packs. **Reported** (search excerpts of Microsoft's pricing page and consultancies, [M16]). Dollar figures conflict (base $1,500 or $1,700 per tenant per month; attach $1,000; extra pack $2,000 per 100,000). **Reported**, unresolved. - Sales Navigator licensed separately (see above). Copilot/agent consumption: **Unavailable**. #### 4.8 Q9. Rhythm: adopt / avoid (**Inferred**) Adopt (this is the platform with the most reusable architecture): - **Enrichment results in their own tables keyed to the golden record**, with the original left untouched. Apply to the master through an explicit, auditable "promote" step. - **Connection object with admin-only creation, explicit consent acknowledgement and a delete-with-dependents flow.** Provider credentials and data-transfer consent belong to a tenant administrator, not to a developer. - **Match-rule toolkit**: normalise-then-compare, ordered rules, exceptions, always/never/bypass/alias override tables. The "never match" and "bypass" lists solve the two failure modes that hurt in membership data (shared inboxes like info@, and dummy values). - **Group-of-fields survivorship** for addresses and for organisation blocks, so the winner is a whole address, not per-field cherry-picking. - **Stable ID with explicit merge/split lineage** (`PreviousCustomerId`). - **Enrich only a segment** and run on a schedule or on demand. - **Suggested-versus-automatic switch with accept/reject and revert**, plus a **citation to the source interaction**, from Dynamics 365 Sales. - **Coverage-per-field metric** on every enrichment run. Avoid: - Having provider data live only in a "second place" that users must remember to consult; give it a first-class display (provenance-badged) on the record. - Depending on a single B2B vendor path (Leadspace B2B closed to new customers; Clearbit folded into HubSpot): see conclusions. - Customer-only legal burden presented as a click-through; Rhythm can do better by recording the lawful basis per enrichment configuration. --- ### 5. Zoho CRM #### 5.1 Q1. Native enrichment (Zia Data Enrichment) - **Verified** (Zoho CRM help, [Z1], [Z2]): "Zia Data Enrichment" can be configured per module (leads, contacts and accounts preset to organizational data; any module can be configured). **One configuration per module**; changing its data type means delete and recreate. Company fields: names, addresses, CEO or decision maker, email and phone, industry, business type/size/model, headquarters country, locations, social links, website, employee count, annual revenue, years in business, about, products, logo. A separate "People Data Enrichment" article exists for personal data and I did not read it. **Unavailable**. - Available for all paid editions per the help page (**Verified**), but third-party sources say Enterprise/Ultimate only (**Reported**, [Z5]). Contradiction, unresolved. #### 5.2 Q2. External providers - The help page says Zia "crawls the web" and "searches across multiple databases"; **no sources are named**. **Verified** (documentation silent on providers, [Z1]). A third-party says it is "a public-web lookup, not a verified contact database", with no phone verification and freshness "nowhere near real-time". **Reported** ([Z5], vendor blog dated 2026-07-07; vendor sells a competing outbound tool, treat with caution). - Related product evidence: Zoho Bigin's "Data Enrichment topping" is **powered by WebAmigo** using publicly available information; fetched fields appear alongside existing data and the user chooses whether to update. **Verified** (Zoho community post, undated; [Z4]). That is Bigin (SMB sibling), not CRM. #### 5.3 Q3/Q4. Timing and modes - The official CRM help documents **manual only**: on create, click "Enrich data" after entering a primary field (email, website or company name); on edit, "Enrich data" in the record's menu; hover shows last-enriched time. Automatic and bulk modes are **not documented** there. **Verified** (documentation silent, [Z1], [Z2]). - A third-party guide says you designate **trigger fields** and Zia enriches automatically when a trigger field is filled on create or update. **Reported** ([Z5]). A 2026 Zoho community post on "new enhancements in Zia" does not mention automatic or bulk enrichment either. **Verified** (silent, [Z3]). Conflict between "manual only" (help) and "trigger fields" (third party) is unresolved; I trust the help page for what is documented. #### 5.4 Q5. Conflict handling - On enrich, a window lets the user **tick the fields to populate**, then Update; Zia fills only checked fields; fields mapped to "None" are skipped; mapping to custom fields is allowed. **Verified** ([Z1]). Whether existing values are overwritten is **Unavailable** (documentation silent). So it is "suggested, per-field accept" by UI, not by policy. - **Duplicates**: _Find & Merge Duplicates_ (from a record: up to six criteria fields, choose the master and per-field values; "Show Conflicts Only" checkbox; up to three records merged at once) and _De-duplicate_ (from the list: up to three fields; exact-copy duplicates auto-merge; differing ones need a manual choice). Merged duplicates are "deleted permanently and the action cannot be reverted"; merged record takes the oldest created time; activities and notes move to the master; edit/delete automation triggers do not run for auto-merges. **Verified** ([Z6]). A Zoho community article says both tools are admin-initiated UI actions and do not fire automatically on import/API (**Reported**). #### 5.5 Q6. Provenance - **Enrichment History**: a log with user, record, date/time and field updates that "helps distinguish enriched data from manual updates"; filter by module, user, time; **cannot be exported**; logs for fields set to "don't show" or deleted are removed. **Verified** ([Z1], [Z3]). The retention-by-field-visibility behaviour is a footgun: hiding a field erases its provenance. - No per-field source badge on records documented. **Unavailable**. #### 5.6 Q7. Permissions, consent - "View results" profile permission needed to enrich; "Manage configuration" for history/usage; enrichment can be disabled per profile (already-enriched records stay). **Verified** ([Z1]). - Consent/privacy: **documentation silent** ([Z1]). **Unavailable**. #### 5.7 Q8. Pricing and metering - Edition-based **monthly limits** with a usage tab (limit, average monthly consumption, consumption by module, adoption trend). **Verified** ([Z1], [Z3]). - A Zoho community post (undated) gives: **Enterprise about 500 records per user licence, up to 50,000 per month; Ultimate 2,000 per user licence, up to 50,000 per month**, org-wide, not per module. **Verified** as Zoho-published community text ([Z3]), but undated and not on the help page. - Bigin topping: 500 enrichments per user per month, capped at 5,000; contacts and companies counted separately; **only successful enrichments count**. **Verified** ([Z4]). - List prices (Standard $14, Professional $23, Enterprise $40, Ultimate $52 per user per month, annual billing). **Reported** ([Z5]). #### 5.8 Q9. Rhythm: adopt / avoid (**Inferred**) Adopt: - **One enrichment configuration per object, mapping each provider output to a target field with an explicit "None"**, and a **user-visible field picker** before applying. - **Org-wide usage tab** broken down by module with a monthly limit. - **Permission split**: "may enrich" versus "may configure and see history". Avoid: - Provenance that disappears when a field is hidden or deleted. - A history log that cannot be exported. - Unreversible merges (shared with NetSuite). - Two modes of configuration with documentation that cannot say whether values are overwritten; Rhythm's policy must be explicit in the UI and in the audit record. --- ### 6. Other platforms with architectural ideas (briefer) #### 6.1 Attio (AI-native CRM) - **Native auto-enrichment on all plans, attributes tiered by plan.** Runs "within seconds" of adding a record; people need an email, companies need a domain; enriched cells show lilac with a sparkle icon in the header; most enriched attributes are editable and **manual values are never overwritten by later enrichment**; refresh is "when Attio has new data" (no cadence); company attributes include name, logo, description, categories, location, estimated ARR, employee range, funding raised, socials, foundation date, team; person attributes include name, picture, description, location, socials, company, job title. Providers are "third-party providers" (unnamed). **Enriched data cannot be exported**; data you enter yourself can be. Public email domains (Gmail etc.) are skipped when linking person to company. Plan tiers: Free = name, description, logo, category, social, location; Plus adds employee count and ARR; Pro adds funding raised. **Verified** ([A1], [A4]). The "cannot export" rule is a licensing constraint passed to the customer. - **AI attributes** (**Verified**, [A2]): Summarize, **Web agent**, Prompt completion, Classify. **Manual only** (single cell, bulk selection or whole view; automation only via workflow AI blocks). **Web agent costs 10 credits per record; others 1**; bulk runs show the credit cost for confirmation; if the AI returns no value the cell stays blank and you are not charged (exception: Classify with no fallback). Web agent shows **confidence** (green/yellow/red dot), **reasoning**, and **web sources**. The March 2026 changelog added confidence, reasoning, citations and bulk enrichment ([A3], **Verified**, 2026-03-24). - Credits by plan (**Verified**, [A4]): workspace credits for AI features and workflows: Free 250 (cannot buy more); Plus 1,500; Pro 10,000; Enterprise custom; per-seat Ask Attio credits 100/500/1,000/2,500. - Third-party enrichment apps (e.g. Apollo Enrichment) exist as marketplace integrations. **Reported** (search excerpt). - **Inferred takeaways**: show a visible badge on enriched values; keep customer-entered values sacrosanct; separate "deterministic provider lookup" (auto, cheap) from "AI web research" (manual, metered, with confidence and citations). #### 6.2 Clay (enrichment as a product: waterfalls) - **Waterfall**: providers chained in a fixed order; each record goes to the first provider; only records without a confident result fall to the next; "stop at the first confident result"; billed only for the lookup that returns a match; a validation step and a **match-confidence threshold** govern what counts as "found"; first step for work email can be a free "Infer Email" guess before any paid provider. **Verified** (Clay's own guide, [C2]); coverage numbers (94%, 40/63/80% stacked, "more than 100 email providers") are **Marketing**, vendor-run benchmarks. - **Two meters**: **Actions** (platform orchestration, one per record per enrichment, AI run or sync; "a few tenths of a penny") and **Data Credits** (third-party data and AI from the marketplace; "a few pennies"; typically 6-20 Data Credits per fully enriched record). **Bring your own key**: removes Data Credits but still consumes an Action. Refunds when a provider refunds Clay for invalid data. Plans (monthly): Launch 15,000 Actions and 2,500-10,000 Data Credits; Growth 40,000 Actions and 6,000-100,000 Data Credits; Enterprise 100,000+. Actions expire; monthly Data Credits roll over up to 2x; annual 15%; top-ups carry a 30% premium (50% on legacy plans) with caps. **Verified** ([C1]); third-party sources disagree on some figures (**Reported**). - Whether failed lookups consume Actions: Clay's own docs conflict (the main Actions doc counts every enrichment; the FAQ says only successful). **Verified** that the docs conflict. - **Inferred takeaways**: model "enrichment attempt", "provider hit", and "validated" as distinct states; **bill on the hit**; allow BYO provider key; show a cost estimate before a bulk run; keep orchestration cost separate from data cost. #### 6.3 Pipedrive (Smart Contact Data / Data Enrichment) - **Provider**: Surfe (Growster SAS) looks up publicly available information based on the contact's email domain, company website or company LinkedIn; Pipedrive "does not source, scrape, or own the data." Pipedrive's marketing page reportedly names a different provider (RightBound); the support page I read names only Surfe. **Verified** (support page, [P1]); the marketing inconsistency is **Reported**. - **Empty-only**: "Only empty fields are enriched. Existing information won't be overwritten." Fields already filled are not charged. **Verified** ([P1]). - **Modes**: manual from the detail view ("enrich data" pill or per-field link) and **bulk up to 100 people/organisations** from the Contacts list after reviewing suggested fields; automatic-on-create is **not documented**. **Verified** ([P1]). - **Credits per company per month**: organisation 1, email 2, phone 5; refresh monthly; top-ups available; **bulk-run credits are reimbursed** when data was already filled or the enrichment failed. **Verified** ([P1]). - **Plans**: organisation enrichment Premium; contact (organisation and people) Ultimate; person enrichment on Premium via add-on. **Verified** ([P1]). - **Privacy**: Surfe acts as an independent controller for its sourcing (with a Legitimate Interest Assessment and an opt-out mechanism) and as a processor for third parties; the Pipedrive customer is the controller for the data it processes and must define a legal basis, inform data subjects and honour opt-outs. The page does not describe collecting consent from data subjects. **Verified** ([P1]). This is the clearest statement of the three-party GDPR split I found. #### 6.4 Freshsales (Freshworks CRM) - Help article: Freshsales "automatically enriches leads, contacts and accounts" with "social and publicly listed information"; plan list Sprout, Blossom, Garden, Estate, Forest. **Verified** ([FS1]), page last modified 2022 per search excerpt, so plan names may be outdated. No providers, modes, overwrite rules or credits stated. **Unavailable**. A separate account article (search excerpt) says account auto-enrichment from a website adds display picture, address, social profiles and employee count on Garden and Estate plans. **Reported**. "Powered by Freddy AI" claims are third-party. **Reported**. #### 6.5 Copper - Copper's own page says it "automatically enriches contact profiles with up-to-date company information, social profiles, and more" and lets you update from LinkedIn. **Marketing** ([CP1], search excerpt). No official help article found. **Unavailable**. Third-party add-ons (Findymail, Surfe) are marketed for Copper. **Marketing**. #### 6.6 folk - **Verified** (folk help centre, [F1]): email enrichment via a **waterfall across Apollo.io, People Data Labs, Clearbit, Datagma, Prospeo and DropContact**; phone via People Data Labs, Wiza and Prospeo; "providers do not exploit or resell any data", folk copies the data into your account. Only people are enriched (company enrichment goes through a Research assistant). Triggers: from a group (select or "Enrich all", up to 500 at a time), a contact record, or the browser extension. **Credits**: Standard 500 per month, Premium 1,000; email 1 credit, phone 4; each successfully enriched field is deducted, failed enrichments are refunded; allowance renews on subscription anniversary. **Overwrite**: "Enriched data is added even when a field already has a value, but it does not replace the existing data" (additive multi-value). Enriched fields marked with a sparkle icon. Success rate "about 50-60%". Folk says it provides a DPA for enrichment and follows GDPR standards. **Marketing** (product page excerpt, [F1b]). - **Inferred**: the additive approach (new email/phone added as an additional value, flagged, not substituting) is a good fit for multi-value contact points. #### 6.7 Twenty (open-source CRM): the most inspectable design Source read from `github.com/twentyhq/twenty`, branch `main`, via the GitHub API on 2026-10-22 (commit not pinned). All **Verified** (vendor-hosted source) unless marked. - **Two enrichment paths exist.** 1. **Server module `company-enrichment`**: enriches the _workspace creator's company_ by work-email domain using the People Data Labs (PDL) client; feature-flagged ("no API key" means disabled and returns "unavailable"); only the workspace creator and only work (non-public) domains; **throttled per workspace** through a throttle service; records an **attempt** per domain so it does not re-spend; distinguishes outcomes `matched`, `transientError`, `unavailable`, permanent error, `skipped`. Files: `packages/twenty-server/src/engine/core-modules/company-enrichment/services/company-enrichment.service.ts` and siblings. 2. **A first-party "app" `people-data-labs`** (`packages/twenty-apps/public/people-data-labs`): "enrich any Person or Company in one click", runs from the command menu, as a **workflow step**, or from **AI chat**; "+30 rich data fields"; "Smart matching ... only writes confident matches; set the minimum likelihood in the app settings". - **Provider-namespaced sidecar fields**: the app creates `pdl*` custom fields (person: `pdlLikelihood`, `pdlRawPayload`, `pdlLastEnrichedAt`, `pdlEnrichmentStatus`, `pdlSeniority`, `pdlSkills`, `pdlJobTitleClass` and many more; company: `pdlId`, `pdlRawPayload`, `pdlEnrichmentStatus`, `pdlLastEnrichedAt`, `pdlTotalFunding` and more). On a match it writes `{ writable standard fields, ...pdl fields, pdlLikelihood, pdlRawPayload, pdlLastEnrichedAt, pdlEnrichmentStatus: 'MATCHED' }`. The provider's **whole raw payload** is stored; the "standard" CRM fields (name, emails, phones, job title, LinkedIn link) are written only through a `pickWritableStandard` step that applies per-field "is empty" checks. - **Overwrite policy is an explicit tri-state**: `UpdateFieldsOption = 'Yes and overwrite' | "Yes and don't overwrite" | 'No'`. In code: `overrideExistingValues` boolean, `shouldPersist` boolean (a dry run returns the mapped data without persisting). - **Confidence thresholds**: `minLikelihood` for strong identifiers (LinkedIn, email, domain) and a **separate, higher `weakIdentifierMinLikelihood` for name-based matches**; per-call or environment-variable configurable. - **Batching and cost control**: batch size 100 per provider call; deduplicated record IDs; an in-run cache of company IDs by match key; stops the run on a provider access error. - **Billing**: README states "Pay only for matches, in Twenty credits; not found and skipped records are free"; **person match $0.15, company match $0.075**. Code: `chargeMatchedEnrichments` calls `chargeCredits` with `matchedCount * creditsPerMatchMicro`, returning early when there are zero matches. - Open issue history shows operational fragility: #19184 "Fix CRM re-enrichment: phone number not populating + bulk enrich broken" (closed 2026-03-31) and #27547 "Retrieve available contact photos when email sync creates People" (closed 2026-10-07). **Verified** (GitHub issue titles). - **Inferred takeaways**: this is the closest public analogue to a clean enrichment architecture: provider-prefixed fields + raw payload + status + likelihood + timestamp; separation of "stored provider facts" from "applied to master fields"; dry-run; per-workspace throttle; attempt ledger; success-only billing via a platform credit API. #### 6.8 "Agentic enrichment" (cross-vendor pattern) | Vendor | Feature | Writes without review? | Evidence trail | Label | | ---------- | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | ----------------------------------------------------- | ----------------------- | | Salesforce | Agentforce Prospecting; Sales agents "suggestive" vs "autonomous" mode | Admin-selectable; autonomous mode updates opportunity fields automatically | Not described | Marketing ([S9], [S10]) | | Microsoft | Suggested Updates (accounts/contacts/leads); AI enrichment for opportunities | Admin chooses automatic or manual; sellers accept/reject or revert | Cites source email/meeting | Reported ([M14], [M15]) | | HubSpot | Data agent / smart properties; conversational enrichment | Smart properties preview then "Apply changes"; conversational enrichment has no documented review | Not described for data agent; none for conversational | Verified ([H7], [H8]) | | Attio | Web agent attribute | Manual trigger; value shown with confidence | Confidence + reasoning + sources | Verified ([A2], [A3]) | | Clay | AI runs inside tables (Actions + Data Credits) | N/A (workspace is staging) | Per-cell | Verified ([C1]) | Pattern: the platforms that attach **confidence, reasoning and citations to each AI-sourced value** (Attio; Microsoft for email-derived suggestions) are the only ones where a person can audit an AI write. HubSpot's own page says to "avoid entering sensitive information" into data-agent prompts ([H8], **Verified**). --- ### 7. (a) Comparison matrix Legend: V = Verified, R = Reported, M = Marketing, U = Unavailable. Cells are terse by design; detail is in sections 1-6. #### 7.1 Questions 1-4 (what, who, when, modes) | Platform | Q1 Native enrichment | Q2 External providers | Q3 When | Q4 Modes | | ------------ | ------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------- | | Salesforce | No native dataset; Data 360 unification (matching/survivorship); Agentforce Prospecting (V name, M behaviour) | AppExchange: ZoomInfo, D&B, Demandbase, LinkedIn (R/M); Clearbit not standalone (V) | Data Integration Rules periodic (R); Data 360 scheduled or manual run (V); duplicate rules on save (V) | Auto scheduled + manual; agent suggestive/autonomous (M) | | HubSpot | Contact+company enrichment, form shortening, buyer intent, dup mgmt, conversational + AI (V) | Commercial dataset + unnamed third parties + public web (V); Clearbit heritage (V) | On create, monthly continuous, on engagement, on import, on workflow, on demand (V) | Automatic (off by default), continuous, manual, bulk (100/index), segment, import, workflow (V) | | NetSuite | None found (U); dup detection + merge only (V) | Via third-party connectors (R/M); DataFox sunset 2023 (R) | Dup alerts on save, after initial search (V); import behaviour silent (U) | Manual merge, mass-update resolution (V) | | Dynamics 365 | CI-Data: Brands, Interests, Enhanced addresses (V); Sales suggested updates (R) | Experian, LiveRamp, HERE, Leadspace (B2B closed), Azure Maps, SFTP (V) | On demand or system schedule; segment-scoped (V); email-driven continuous (R) | Manual run, scheduled refresh; suggested vs automatic (R) | | Zoho CRM | Zia Data Enrichment per module, org/personal (V) | Unnamed "web crawl/databases" (V silent); public web (R) | On create/edit by click (V); trigger fields on save (R, conflicts) | Manual documented (V); auto/bulk not documented (V silent) | | Attio | Auto enrichment + AI attributes (V) | Unnamed third-party providers (V); Apollo etc. via apps (R) | On record add, seconds (V); refresh when new data (V) | Auto (deterministic); AI manual only; bulk AI (V) | | Clay | Product is enrichment orchestration (V) | 100+ providers (M); 5 named email examples (V) | Table-driven, per-row run (V) | Manual, bulk, API/CRM sync (V) | | Pipedrive | Data Enrichment (V) | Surfe (V) | Manual and bulk (V); auto-on-create not documented (V silent) | Manual; bulk 100 (V) | | Freshsales | Auto enrichment of leads/contacts/accounts (V thin) | Unnamed "social and public" (V) | Not stated (U) | Automatic only described (V) | | Copper | Auto contact enrichment (M) | Unnamed; 3rd-party add-ons (M) | Unavailable | Unavailable | | folk | People enrichment waterfall (V) | Apollo, PDL, Clearbit, Datagma, Prospeo, DropContact, Wiza (V) | On demand; group up to 500 (V) | Manual and group (V) | | Twenty | PDL app + server company enrichment (V code) | People Data Labs (V) | Command menu, workflow step, AI chat; throttled (V) | Single and batch (100) with dry run (V) | #### 7.2 Questions 5-9 (conflicts, provenance, consent, pricing, Rhythm) | Platform | Q5 Conflicts | Q6 Provenance | Q7 Consent/privacy | Q8 Pricing | Q9 Rhythm takeaway (Inferred) | | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------- | | Salesforce | Blank-only for integration rules (R); reconciliation: Most Recent / Most Frequent / Source Priority, object-level + field override (V concept, R detail); dup rules block/alert (V) | Unified link objects, source untouched (V); field history 20 fields/18-24 mo (R); no provider-per-field stamp (U) | Individual + Data Use Purpose + Legal Basis, 4 consent layers (V); enforcement link to enrichment (U) | Flex $500/100k credits; unify ~100k credits/1M rows (R); Agentforce $0.10/action (V arithmetic) | Adopt reconciliation vocabulary + link object; avoid three engines and per-row metering of matching | | HubSpot | Per-property overwrite: fill-empty / overwrite / never; automatic fills blank only; continuous stops on edited property; retract nulls; some writers bypass (V) | Property history Source column, change-source list, Activity tab (V); enrichment-specific source label (U) | Super Admin config; per-user access; contact opt-out marker; individual opt-out form (V); July 2026 sharing reversal (R) | Standard enrichment credit-free (V); smart properties 10 credits/prompt/record (R); overage $10/1k (R) | Adopt policy model + manual-edits-win + opt-in + opt-out marker; avoid bypass writers, pooled data, fee-on-miss | | NetSuite | Merge: blank copied, populated kept, source deleted, irreversible (V) | System Note "Duplicate Resolution" (V) | Merge permission; no enrichment consent (V) | None native (U) | Adopt build-index-first, merge as privilege; avoid irreversible delete | | Dynamics 365 | Merge by Importance/Most recent/Least recent, field groups; match normalisation + exceptions + override lists; enrichment in separate tables (V); suggestions accept/reject/revert (R) | Separate enrichment tables, run status, per-field coverage, `PreviousCustomerId` (V); citations (R) | Admin-only connections with consent acknowledgement; customer owns lawful basis (V) | Microsoft enrichments included; partners separate (V); capacity packs (R) | Adopt side tables, connection object, match toolkit, group survivorship, suggest/auto/revert + citations | | Zoho CRM | User ticks fields; overwrite semantics silent (V/U); dedup manual master choice, irreversible (V) | Enrichment History distinguishes Zia vs manual; not exportable; erased when field hidden (V) | Permissions only; privacy silent (V/U) | Edition limits 500-2,000/user up to 50k/mo (V community) | Adopt per-object config, field picker, usage tab; avoid fragile history | | Attio | Manual values never overwritten (V) | Lilac cell + sparkle; AI confidence/reasoning/sources (V) | Not described (V silent); enriched data not exportable (V) | Plan-tiered attributes; AI 10 credits web agent; no charge on empty (V) | Adopt badges, confidence + citations, no-charge on miss; note licence limits on export | | Clay | N/A (staging workspace) | Per-cell results; validation step (V) | Not researched (U) | Actions + Data Credits; hit-only billing; BYO key (V) | Adopt waterfall + two meters + hit-only billing | | Pipedrive | Empty fields only (V) | Not described (U) | Three-party GDPR split; Surfe independent controller + opt-out (V) | 1/2/5 credits; reimbursed on fill/fail in bulk (V) | Adopt empty-only default + reimbursement + explicit controller split | | Freshsales | Unavailable | Unavailable | Unavailable | Unavailable | Nothing to adopt | | Copper | Unavailable | Unavailable | Unavailable | Unavailable | Nothing to adopt | | folk | Additive; never replaces (V) | Sparkle icon (V) | DPA claim (M) | 1 credit email, 4 phone; refund on fail (V) | Adopt additive multi-value handling | | Twenty | Tri-state overwrite; per-field empty checks (V code) | pdl* fields + raw payload + likelihood + timestamp + status (V code) | Not documented (U) | Per match: $0.15 person, $0.075 company; misses free (V README/code) | Adopt sidecar fields, thresholds, dry run, throttle, attempt ledger | --- ### 8. (b) Architectural conclusions for Rhythm (all Inferred, each tied to the evidence above) Rhythm context assumed (and not verified here): multi-tenant, each tenant an association; members are individuals and employer organisations; every tenant needs to be able to explain where a piece of member data came from; enrichment of individuals is sensitive. 1. **Store provider output separately from the record, then apply by policy.** Microsoft keeps enrichment in separate tables joined by CustomerId ([M1]); Twenty stores `pdl*` sidecar fields plus the full raw payload ([T-code]); Salesforce keeps source data untouched behind link objects ([S1]). Rhythm should persist every enrichment result as an immutable observation (provider, query inputs, match confidence, timestamp, raw payload, status) and let a separate, auditable apply step write master fields. 2. **Policy per field, three values, default "fill empty".** HubSpot's mapping tab (fill empty / overwrite / never, [H2]), Twenty's tri-state ([T-code]), Pipedrive's empty-only ([P1]) and Salesforce's blank-only integration rules ([S13]) converge. Make it a per-tenant, per-field, per-provider setting, with "never" as the safe value for sensitive fields. 3. **Manual edits always win, and later enrichment may not touch them.** Attio ([A1]) and HubSpot continuous enrichment ([H3]) both stop writing a property once a person or another system edits it. This requires per-field last-writer provenance (conclusion 4); without it the rule is unenforceable. 4. **Field-level provenance is the gap everyone leaves open. Do it properly.** HubSpot has a Source column ([H4]) but no provider-per-value; Salesforce has link objects and field history but not provider per field; Zoho's history is deleted when a field is hidden ([Z1]); Microsoft cites interactions only for email-derived suggestions ([M14]). Rhythm should record on every applied value: provider, observation ID, applied-by (policy, user, workflow, agent), policy version, timestamp and confidence. Keep it queryable and exportable. 5. **Make confidence a stored, thresholded property, with a stricter threshold for weak identifiers.** Twenty uses `minLikelihood` and a higher `weakIdentifierMinLikelihood` for name-based matches ([T-code]); Clay uses a confidence threshold to decide "found" ([C2]); Microsoft exposes match precision percentages ([M10]). For member data, name-only matches should default to "suggest, never auto-apply". 6. **Offer a suggest / auto-apply switch, per field class, with accept, reject and revert, and an evidence link.** Microsoft's admin choice of automatic versus manual approval with revert ([M15]) and the citation to the source interaction ([M14]), plus Attio's confidence-reasoning-citations ([A2], [A3]), are the best patterns found. HubSpot's conversational enrichment, which can write regardless of the mapping rules and has no documented review ([H7]), is the counter-example. 7. **Provide four trigger modes, all opt-in per tenant, all through one policy gate.** On record create (off by default; HubSpot [H2]), on demand per record, bulk with a visible cap and cost preview (HubSpot 100, Pipedrive 100, folk 500, Twenty batch 100), and scheduled refresh of previously enriched records (HubSpot monthly) that only touches provider-owned values. Cap bulk size, show the estimate first, and never let a workflow or agent bypass the policy gate. 8. **Bill on the hit, not the attempt; keep orchestration cost distinct from data cost; show the estimate before bulk.** Twenty (misses free), Clay (hit-only; Actions vs Data Credits; BYO key), folk (failures refunded), Pipedrive (reimbursed in bulk when filled or failed), Attio (no charge on empty). Avoid HubSpot smart-property billing "filled or not" ([H12]) and Salesforce's per-million-row unify credits ([S6]). Rhythm should record a metered event only when a provider returns a usable, applied-or-staged result, and keep a per-tenant monthly cap. 9. **Provider adapter boundary, not a built-in dataset; ordered waterfall for later.** None of the CRMs owns primary B2B data (HubSpot's dataset is purchased/assembled; Zoho crawls; Pipedrive and Attio resell). Provider volatility is real: Data.com retired (2020), DataFox sunset (2023, R), Clearbit absorbed (2025), Leadspace B2B closed to new customers ([M5]), Pipedrive's marketing and support pages name different providers. Define a provider interface (inputs required, fields returned, confidence, licence terms, cost). Start with one or two providers. Add waterfall ordering (cheapest trustworthy first, stop at first confident hit, per Clay [C2]) only when coverage data justifies it. 10. **Store each provider's licence terms with its data.** Attio's enriched data cannot be exported ([A1]), which shows provider terms can restrict what the customer may do with data. Rhythm's tenant data export, member data portability and reporting features must know which fields are provider-licensed. Keep a per-provider flag for retention, redistribution and export. 11. **Treat enrichment of individuals as a separate processing activity with a recorded lawful basis, a controller split, an opt-out and a suppression list.** Pipedrive documents the three-party split (provider independent controller with legitimate-interest assessment and opt-out; customer controller for their processing) ([P1]); HubSpot has a contact-level "enrichment opt out" marker and a dataset opt-out form ([H1]); Salesforce models Legal Basis and Data Use Purpose ([S8]); Microsoft puts the consent burden on the customer admin ([M8]). Rhythm should: default people-enrichment to off; require an admin to enable it per tenant with the lawful basis recorded; keep a do-not-enrich flag on members that suppresses all providers; log the opt-out; and honour deletion by tombstoning the observation. Check that CRM consent ledger and enrichment suppression use one shared record (Inferred; project memory says consent lives in the CRM service). 12. **Never pool tenant data into a shared enrichment dataset.** HubSpot's attempt to share enrichment-derived data across customers with default-on consent was reversed within four days after backlash, with a commitment to "fully and transparently opt-in" ([H10], R). Members of associations are even less likely to accept it. Write the non-pooling rule into the architecture and the contracts, and make any future sharing an explicitly separate, opt-in product. 13. **Separate enrichment from identity resolution, but use one rule vocabulary and keep merges reversible.** Salesforce's three engines, Zoho's manual merges, and NetSuite's irreversible delete-on-merge are the cautionary tales; Microsoft's normalise-compare-except pipeline with always-match, never-match, bypass and alias override tables, and its group-of-fields survivorship ([M10], [M11]), is the useful toolkit. Dedupe shared and dummy values (info@, 555-1212), keep stable IDs with merge/split lineage, and tombstone merged records with an undo window. 14. **Prefer organisation (domain-keyed) enrichment first; people enrichment second.** Company enrichment keys on a domain (HubSpot, Attio, Twenty), skips public email domains (Attio, Twenty), has better coverage and lighter privacy exposure; people enrichment needs name plus company and is where match quality, GDPR and error cost concentrate (folk reports 50-60% success; Twenty needs a stricter weak-identifier threshold). For associations, employer-organisation attributes (size, industry, location) deliver most of the value. 15. **Operational controls from day one.** Per-tenant throttle and idempotent attempt ledger so a known miss is not repurchased (Twenty throttle service and attempt record); a run log with initiator, count, type (HubSpot Activity tab); per-field coverage and match-rate reporting on every run (Microsoft "number of customers enriched by field"); a usage and limit view broken down by object (Zoho); a connection-removal flow that deactivates dependents (Microsoft); and a documented behaviour when a provider is down or out of credits (Twenty: explicit "unavailable" outcome). --- ### 9. (c) Sources All pages accessed 2026-10-22. "Fetched" = I read the page via the fetch tool (content summarised by the tool; I did not read raw HTML). "Excerpt" = I only saw a search-tool excerpt, not the page. IDs are used in the report. #### 9.1 Salesforce - [S1] Trailhead, "Unify Your Data with Effective Identity Resolution Strategies". https://trailhead.salesforce.com/content/learn/projects/explore-data-cloud-core-functionality/unify-your-data . Fetched. Vendor training content, treated as Verified. - [S2] Salesforce Help, "Identity Resolution Reconciliation Rules". https://help.salesforce.com/s/articleView?language=en_US&id=sf.c360_a_reconciliation_rules.htm&type=5 . **Not fetchable** (JavaScript "Sorry to interrupt / CSS Error" shell). Content seen only as a search excerpt. Reported. - [S3] Salesforce Blogger, "Balance Precision and Consolidation with Better Identity Resolution Match Rules" (2025-06-11). https://www.salesforceblogger.com/2025/06/11/balance-precision-and-consolidation-with-better-identity-resolution-match-rules/ . Fetched. Reported (ruleset up to 10 match rules x 10 criteria; unified profiles from more than 50,000 sources discarded). - [S4] InfallibleTechie, "Salesforce Data 360 (Data Cloud) Identity Resolution" (2026-01). https://www.infallibletechie.com/2026/01/salesforce-data-360data-cloud-identity-resolution.html . Fetched. Reported. - [S5] Salesforce, Data 360 pricing. https://www.salesforce.com/data/pricing/ . Fetched. Verified. - [S6] Jitendra Zaa, "Salesforce Data 360 credit optimization guide (March 2026)". https://www.jitendrazaa.com/blog/salesforce/salesforce-data-360-credit-optimization-guide-march-2026/ . Fetched. Reported (rate table). - [S7] Trailhead, "Resolve and Prevent Duplicate Data in Salesforce", unit 2. https://trailhead.salesforce.com/en/content/learn/modules/sales_admin_duplicate_management/sales_admin_duplicate_management_unit_2 . Fetched. Verified. - [S8] Trailhead, "Design the Consent Data Model for General Consent". https://trailhead.salesforce.com/content/learn/modules/consent-management-with-salesforce/design-the-consent-data-model-for-general-consent . Fetched. Verified. ([S8b] = third-party excerpts about the Individual object, via search; Reported.) - [S9] Trailhead, "Spring '26 release highlights: Agentforce, Data 360, Agentforce Sales". https://trailhead.salesforce.com/content/learn/modules/spring-26-release-highlights/see-whats-new-agentforce-data-360-agentforce-sales . Fetched. Verified. - [S10] Salesforce, AI sales agent. https://www.salesforce.com/sales/ai-sales-agent/ . Fetched. Marketing. - [S11] Salesforce, Agentforce pricing. https://www.salesforce.com/agentforce/pricing/ . Fetched. Verified. - [S12] ZoomInfo (vendor-authored), "How to set up the Salesforce Prospecting Agent in 2026". https://pipeline.zoominfo.com/sales/salesforce-agentforce-prospecting-agent-zoominfo . Fetched. Marketing. - [S13] Data Integration Rules and Data.com retirement: gyde.ai mirror of Salesforce help (`https://gyde.ai/kb/activate- data- integration- rules`, excerpt only; my fetch of a normalised URL returned 404); D&B Dreamforce migration guide (`https://www.dnb.com/content/dam/english/business-trends/DNB_Dreamforce_6_Tips_for_Successful_DDC_Migration.pdf`, excerpt); HG Insights, "What is Lightning Data?" (`https://support.hginsights.com/articles/2639215939-what-is-lightning-data`, excerpt, updated 2025-12-06). All Reported. - [S14] Field history: SalesforceBen (`https://salesforceben.com/salesforce-field-history-tracking`), Calibermind (`https://docs.calibermind.com/onboarding/create-field-history-in-salesforce/`) and Salesforce developer doc "Field Audit Trail" (`https://developer.salesforce.com/docs/platform/field-history-retention/guide/field-audit-trail.html`, **HTTP 403 to my fetch**, excerpt only). Reported. - [S15] ZoomInfo (vendor-authored), "Salesforce Data Enrichment Guide". https://pipeline.zoominfo.com/operations/salesforce-data-enrichment . Excerpt only. Marketing. - [S16] Incepta Solutions, "Salesforce Data 360: What changed and what to fix first". https://inceptasolutions.com/salesforce-data-360/ . Excerpt only. Reported (rename Oct 2025). #### 9.2 HubSpot - [H1] HubSpot KB, "Get started with data enrichment". https://knowledge.hubspot.com/records/get-started-with-data-enrichment . Fetched. Verified. - [H2] HubSpot KB, "Manage data enrichment settings". https://knowledge.hubspot.com/records/manage-data-enrichment-settings . Fetched. Verified. - [H3] HubSpot KB, "Enrich your contact and company data". https://knowledge.hubspot.com/records/enrich-your-contact-and-company-data . Fetched. Verified. - [H4] HubSpot KB, "HubSpot's change sources". https://knowledge.hubspot.com/properties/hubspots-change-sources . Fetched. Verified. - [H5] HubSpot KB, "Use data quality tools". https://knowledge.hubspot.com/data-management/use-data-quality-tools . Fetched. Verified. - [H6] HubSpot, Data Hub data quality software page. https://www.hubspot.com/products/data-quality-software . Fetched. Marketing. - [H7] HubSpot KB, "Use conversational enrichment for emails". https://knowledge.hubspot.com/one-to-one-email/use-conversational-enrichment-for-emails . Fetched. Verified. - [H8] HubSpot KB, "Set up and use the data agent". https://knowledge.hubspot.com/data-management/use-data-agent . Fetched. Verified. - [H9] HubSpot KB, "Automatically populate records using smart properties". https://knowledge.hubspot.com/properties/create-smart-properties . Excerpt only. Reported. - [H10] CMSWire, "HubSpot Scraps Opt-Out Data Sharing Plan Days After Announcing It". https://www.cmswire.com/customer-experience/hubspot-reverses-customer-data-enrichment-plan-after-customer-backlash/ . Fetched. Reported. - [H11] Concept Ltd, "HubSpot's data enrichment changes: what you need to know". https://conceptltd.com/blog/hubspot-data-enrichment-changes . Fetched. Reported. - [H12] RevOps HQ, "HubSpot Breeze: components, credits, data requirements and governance" (as of 2026-09-27). https://revopshq.com/blog/hubspot-breeze . Fetched. Reported. - [H13] eesel.ai, "A deep dive into HubSpot's Breeze Intelligence data enrichment" (last edited 2026-10-14). https://www.eesel.ai/blog/breeze-intelligence-data-enrichment . Fetched. Reported. Also HubSpot-hosted Breeze Intelligence pricing PDF (`https://www.hubspot.com/hubfs/Agency/GTM%20-%20Product%20Resources%20(Justin)/Product%20Folders%20(EN)/HubSpot%20AI/Breeze%20Intelligence%20-%20Pricing%20Guide.pdf`, excerpt only; the sibling "Data Enrichment" PDF was unreadable binary). - [H14] Clearbit/HubSpot blog, "The Future of Clearbit's Free Tools" (2025-02-13). https://clearbit.com/blog/the-future-of-clearbits-free-tools . Fetched. Verified. - [H15] HubSpot Community, "Clarification needed: AI model training vs enrichment data sharing". https://community.hubspot.com/t/clarification-needed-ai-model-training-vs-enrichment-data-sharing/151789 . **HTTP 403 to fetch**; excerpt only. Reported. - [H16] Clearbit Help Center, "How can I buy Clearbit?". https://help.clearbit.com/hc/en-us/articles/20717192156695-How-Can-I-Buy-Clearbit . Excerpt only. Reported. #### 9.3 Oracle NetSuite - [N1] Oracle NetSuite Help, "Setting Up Duplicate Detection". https://docs.oracle.com/en/cloud/saas/netsuite/ns-online-help/section_N258211.html . Fetched. Verified. - [N2] Oracle NetSuite Help, "Merging Customer Records". https://docs.oracle.com/en/cloud/saas/netsuite/ns-online-help/section_1509542900.html . Fetched. Verified. - [N3] TrustRadius, DataFox listing. https://www.trustradius.com/products/datafox/details . Excerpt only. Reported. - [N4] Oracle Cloud Readiness, "DataFox for Eloqua (Account Enrichment)". https://www.oracle.com/webfolder/technetwork/tutorials/tutorial/cloud/data/releases/Apr21/21apr-datafox-nfs.htm . Excerpt only. Reported. - [N5] Stacksync (`https://www.stacksync.com/integrations/netsuite-and-zoominfo`), Workato (`https://www.workato.com/integrations/netsuite_rest~zoominfo`), Zapier (`https://zapier.com/apps/netsuite/integrations/zoominfo`). Excerpts only. Marketing. #### 9.4 Microsoft - [M1] Data enrichment overview (Customer Insights - Data), updated 2026-09-03. https://learn.microsoft.com/en-us/dynamics365/customer-insights/data/enrichment-hub . Fetched. Verified. - [M2] Enrich unified data (manage enrichments), 2026-09-03. https://learn.microsoft.com/en-us/dynamics365/customer-insights/data/enrichment-manage . Fetched. Verified. - [M3] Brands and interests from Microsoft, 2026-09-03. https://learn.microsoft.com/en-us/dynamics365/customer-insights/data/enrichment-microsoft . Fetched. Verified. - [M4] Demographics from Experian (preview), 2026-07-14. https://learn.microsoft.com/en-us/dynamics365/customer-insights/data/enrichment-experian . Fetched. Verified. - [M5] Leadspace (preview), 2026-07-11. https://learn.microsoft.com/en-us/dynamics365/customer-insights/data/b2b/enrichment-leadspace . Fetched. Verified. - [M6] LiveRamp AbiliTec (preview), 2026-07-14. https://learn.microsoft.com/en-us/dynamics365/customer-insights/data/enrichment-liveramp . Fetched. Verified. - [M7] Enrichment for data sources, 2026-09-03. https://learn.microsoft.com/en-us/dynamics365/customer-insights/data/data-sources-enrichment . Fetched. Verified. - [M8] Connections overview, 2025-10-01. https://learn.microsoft.com/en-us/dynamics365/customer-insights/data/connections . Fetched. Verified. - [M9] Data unification overview, 2025-10-01. https://learn.microsoft.com/en-us/dynamics365/customer-insights/data/data-unification . Fetched. Verified. - [M10] Matching rules for data unification, 2026-09-11. https://learn.microsoft.com/en-us/dynamics365/customer-insights/data/data-unification-match-tables . Fetched. Verified. - [M11] Unify customer columns (merge rules), 2025-10-01. https://learn.microsoft.com/en-us/dynamics365/customer-insights/data/data-unification-merge-tables . Fetched. Verified. - [M12] Set up duplicate detection rules (Power Platform), 2026-08-14. https://learn.microsoft.com/en-us/power-platform/admin/set-up-duplicate-detection-rules-keep-data-clean . Fetched. Verified. - [M13] Dynamics 365 Sales 2026 release wave 1 overview (archived October 2026). https://learn.microsoft.com/en-us/dynamics365/release-plan/2026wave1/sales/dynamics365-sales/ . Fetched. Marketing. - [M14] Message Center MC1420564 (mirror), "Automatic CRM record linking and suggested updates". https://mc.merill.net/message/MC1420564 . Fetched. Reported. - [M15] Message Center MC1254454 (mirror), "AI-based data enrichment for opportunities". https://mc.merill.net/message/MC1254454 . Fetched. Reported. - [M16] Customer Insights pricing: o365hq (`https://o365hq.com/blog/dynamics-365-customer-insights-pricing-and-licensing-journeys-data-and-attach`), Rand Group (`https://www.randgroup.com/dynamics-365-customer-insights-pricing/`). Excerpts only. Reported. Microsoft's pricing URL (`https://dynamics.microsoft.com/marketing/pricing/`) returned a redirect to the Dynamics 365 home page that I did not follow; **not read**. - [M17] LinkedIn Sales Navigator for Dynamics 365 Sales: Learn "Install and enable LinkedIn Sales Navigator" (`https://learn.microsoft.com/en-au/dynamics365/linkedin/install-sales-navigator`). Excerpt only. Reported. #### 9.5 Zoho - [Z1] Zoho CRM Help, "Data Enrichment" (classic). https://help.zoho.com/portal/en/kb/crm/zia-artificial-intelligence/data-enrichment/articles/zia-data-enrichment . Fetched. Verified. - [Z2] Zoho CRM Help, "Data Enrichment" (NextGen). https://help.zoho.com/portal/en/kb/crm-nextgen/zia-artificial-intelligence/data-enrichment-1/articles/nextgen-zia-data-enrichment . Fetched. Verified. - [Z3] Zoho Community, "New enhancements in Zia" (undated). https://help.zoho.com/portal/fr/community/topic/new-enhancements-in-zia . Fetched. Verified as Zoho-published community text. - [Z4] Zoho Community, "Enrich your contact and company details automatically using the Data Enrichment topping" (Bigin; undated). https://help.zoho.com/portal/fr/community/topic/enrich-your-contact-and-company-details-automatically-using-the-data-enrichment-topping?page=1 . Fetched. Verified (Bigin, not CRM). - [Z5] Salesgear, "Zoho CRM Data Enrichment in 2026: What Zia does (and where it stops)" (2026-07-07). https://www.salesgear.io/blog/zoho-crm-data-enrichment-guide/ . Fetched. Reported (vendor with competing product). - [Z6] Zoho CRM Help, "Merging duplicate records". https://help.zoho.com/portal/en/kb/crm-v2-1/manage-crm-data/duplication-management/articles/crm-v2-merge-duplicate-record . Fetched. Verified. #### 9.6 Others - [A1] Attio Help, "Enriched data". https://attio.com/help/reference/managing-your-data/enriched-data . Fetched. Verified. - [A2] Attio Help, "AI attributes". https://attio.com/help/reference/attio-ai/ai-attributes . Fetched. Verified. - [A3] Attio changelog, "Web research agent" (2026-03-24). https://attio.com/changelog/2026/web-research-agent . Fetched. Verified. - [A4] Attio Help, "Attio plans and features". https://attio.com/help/reference/workspace-settings-billing/attio-plans-and-features . Fetched. Verified. - [C1] Clay Docs, "Actions and Data Credits". https://university.clay.com/docs/actions-data-credits . Fetched. Verified. - [C2] Clay, "The Complete Guide to Waterfall Enrichment (2026)". https://www.clay.com/guides/waterfall-enrichment . Fetched. Verified for method; Marketing for coverage claims. - [P1] Pipedrive Support, "Data enrichment". https://support.pipedrive.com/en/article/data-enrichment . Fetched. Verified. - [F1] folk help centre, "Data enrichment". https://intercom.help/folk-app/en/articles/6804416-data-enrichment . Fetched. Verified. [F1b] folk product page (`https://www.folk.app/it/fr/products/contact-enrichment`), excerpt only. Marketing. - [FS1] Freshsales support, "Does Freshsales enrich lead, contact and account information?". https://support.freshsales.io/support/solutions/articles/217625-does-freshsales-enrich-lead-contact-and-account-information . Fetched. Verified (thin). - [CP1] Copper, "Manage contacts" page (`https://try.copper.com/manage-contacts`). Excerpt only. Marketing. - [T-code] Twenty source on GitHub (`main`, read via `gh api`, 2026-10-22): `packages/twenty-apps/public/people-data-labs/README.md`, `.../SETUP.md`, `.../src/types/update-fields-option.ts`, `.../src/types/single-enrich-input.ts`, `.../src/logic-functions/utils/{run-batch-enrichment,run-single-enrichment,resolve-min-likelihoods,build-person-matched-data,charge-matched-enrichments}.ts`, `.../src/fields/{person,company}/pdl-*.field.ts`; `packages/twenty-server/src/engine/core-modules/company-enrichment/services/company-enrichment.service.ts` and sibling modules. https://github.com/twentyhq/twenty . Issue titles: https://github.com/twentyhq/twenty/issues/19184 and https://github.com/twentyhq/twenty/issues/27547 . Verified (source code is the primary evidence; behaviour at runtime not tested). #### 9.7 Pages I could not fetch or read (and what that leaves open) | Page | Result | What stays open | | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ | ------------------------------------------------------------------------------------ | | help.salesforce.com identity resolution and reconciliation articles | JavaScript shell only | Exact reconciliation semantics, ruleset limits, schedule (used Trailhead + excerpts) | | developer.salesforce.com (Field Audit Trail, privacy data model, Tooling `CleanRule`) | HTTP 403 | Field history limits; consent entity names; Data Integration Rule options | | gyde.ai Data Integration Rules (normalised URL) | HTTP 404 | Direct read of the "never overwrite except geocodes" rule (excerpt only) | | community.hubspot.com enrichment thread | HTTP 403 | Official HubSpot staff wording on opt-out and contributed data | | knowledge.hubspot.com `property-history-and-source`, `enrich-records-using-breeze-intelligence`, `opt-out-of-hubspot-data-enrichment` (guessed URLs) | HTTP 404 | Opt-out help page text; exact enrichment source label in property history | | HubSpot Data Privacy FAQ ("Enrichment Products") | Not reached (search quota exhausted) | HubSpot's stated lawful basis and GDPR posture for enrichment | | HubSpot "Breeze Intelligence - Data Enrichment" PDF | Unreadable binary | Nothing material (a 2024 Canva sales sheet) | | handsontek.net Message Center write-up | HTTP 403 | Used the merill.net mirror instead | | microsoft.com Customer Insights pricing | Redirect to home, not followed | Authoritative price and capacity numbers | | zoho.com `/crm/help/zia/enrichment.html` | HTTP 404 | Zoho's own edition table of enrichment limits | | attio.com `/help/reference/managing-your-data/enrichment` and `/enriching-your-data` (guessed) | HTTP 404 | None; found the correct "Enriched data" page | | NetSuite SuiteApp marketplace; D&B Connect NetSuite listing | Not reachable | Whether any vendor SuiteApp for enrichment exists | | Salesforce AppExchange listings and 2026 Lightning Data status | Not reached | Current availability of Lightning Data | | Dataverse / Power Platform native enrichment | Not researched | Whether Power Platform ships any enrichment connector set | | Salesforce Data 360 consent enforcement and residency | Not reached | Whether Data Use Purpose or Individual flags suppress enrichment automatically | | Clay data-protection posture; Clay AI (Claygent) docs | Not researched | Clay's own privacy model | #### 9.8 Conflicts left unresolved (so nobody mistakes them for findings) - Zoho: help page says all paid editions; third-party says Enterprise and Ultimate only. Help page documents only manual enrichment; third-party says trigger-field automatic enrichment. - HubSpot: product name "Breeze Intelligence" versus "data enrichment"; HubSpot knowledge-base tiers versus product-catalog tiers for Smart CRM; whether "Data enrichment access" spends credits (RevOps HQ notes the two HubSpot pages disagree). - Salesforce: Field History field ceilings (20, 60, 200); Data 360 Starter price ($60k versus $108k); Most Frequent tie-break. - Microsoft: Customer Insights - Data list price ($1,500 versus $1,700 base). - Clay: whether a failed lookup consumes an Action (Clay's own docs differ). - NetSuite: two Oracle pages describe the OneWorld subsidiary merge restriction with different strictness. - Pipedrive: provider named as Surfe (support) versus RightBound (marketing, per search excerpt). ## 25. W3 Evidence: Enrichment Provider Landscape for an Association Management System ## W3 Evidence: Enrichment Provider Landscape for an Association Management System Proposal: "Rhythm 2.0 Data Enrichment" Research date: 2026-10-22 (all pages accessed 2026-10-22 unless a row says otherwise) Author: market-research agent (evidence only; no code changed; this is the only file written) ### 0. How to read this report #### 0.1 Why this report exists Rhythm 2.0 is an association management system (AMS). Its contacts are association members: many use personal email addresses (Gmail, Outlook, iCloud), belong to several organizations, hold certifications or licenses, and have layered affiliations. The proposal asks whether Rhythm should offer data enrichment (photo, employer, title, social links, company logo and firmographics, job-change signals, email validity) and, if so, from which providers and under what contract shape. Almost every commercial enrichment vendor was built for B2B sales prospecting (work email in, company and title out). That mismatch drives most of the findings below. #### 0.2 Evidence labels (applied to every claim) - **Verified**: read on a primary vendor or regulator document (API docs, pricing page, terms page, regulator decision, court-adjacent press release) in this session. - **Reported**: stated by a secondary source (comparison blog, review site, news article, search-result summary of a primary page that I could not open directly). Often a competitor with a commercial stake. Treat as a lead, not a fact. - **Marketing**: a vendor's own promotional claim with no method or audit (match rates, "accuracy", "compliant"). - **Unavailable**: I could not obtain the information (page blocked, rendered by JavaScript, requires sales contact, or not published). - **Inferred**: my reasoning from the other evidence. Only used in the analysis sections and where marked. Short tags used in tables: **[V]** Verified, **[R]** Reported, **[M]** Marketing, **[U]** Unavailable, **[I]** Inferred. Source keys such as [S12] point to the full list in section 10. #### 0.3 Method limits you should know about 1. The fetch tool returns a summary produced by a small model, not the raw page. Numbers marked Verified were read in that summary of the primary page; re-check any number before it goes into a contract or budget. 2. The research session's web-search budget ran out part-way through (the tool reported a per-turn limit). Providers researched only by a single search or not at all are listed in section 10.8. Gaps are marked **[U]**, not filled with guesses. 3. Two vendor pages (Diffbot pricing, Firecrawl pricing) contained text addressed to AI agents (instructions to fetch onboarding files). That text was ignored; it is noted here only so a reader knows those pages are not clean sources. 4. Legal statements below are a reading of public text, not legal advice. Anything about resale, caching or GDPR needs counsel review against the actual signed contract. #### 0.4 Corrections to assumptions in the research brief | Assumption in the brief | What the evidence says | | -------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | FullContact is "under Allstate/Dentsu" | Ziff Davis acquired FullContact's assets in Q4 2024 [R, S30, S29]. No Allstate or Dentsu ownership found. | | FullContact has a "Verify API" | No separately named Verify API found. Docs list Enrich, Resolve, Acumen and Identity Streme; "verification" is described as a use case of Identity Streme [R, S26, S27]. | | Clearbit standalone API still sold | Unconfirmed. Free Logo API shut down 1 Dec 2025 [R, S38, S39]. Enrichment is sold as HubSpot Breeze Intelligence; whether new standalone API keys are issued could not be verified from HubSpot [R, S40]. | | Proxycurl "shut down?" | Yes. Proxycurl shut down on 4 July 2025 as a condition of a settlement with LinkedIn [V, S63]; LinkedIn announced the case resolved on 28 July 2025 [V, S64]. Successor: NinjaPear (competitive-intelligence data). | | TowerData is a separate vendor | TowerData and FreshAddress merged in 2021 and rebranded AtData [V, S111]; Experian acquired AtData in February 2026 [R, S112; Experian press page appeared in results]. | | ZoomInfo "Copilot" | The 2026 API product is branded GTM.AI (REST at api.zoominfo.com/gtm), open to all customers since 27 July 2026 [R, S35]. The Nasdaq ticker changed from ZI to GTM in May 2025 (not a company rename) [R, S36]. | | Perplexity Sonar API | Current Perplexity docs price a Search API ($5 per 1,000 requests; $1 per 1,000 fast) and an Agent API; `perplexity/sonar` appears as one model at $0.25 input / $2.50 output per 1M tokens [V, S76, S77]. One third-party source says Sonar is supported only until 27 Sept 2026 [R, S82; single source, unconfirmed]. | ### 1. Executive summary (findings, not recommendations) 1. **No provider publishes a verified match rate for "personal email only" inputs.** The only quantified evidence is indirect: People Data Labs (PDL) documents that an email-only match "rarely" scores above 6 of 10 and that name plus location "rarely" scores above 4, with a score of 2 implying roughly a 10 to 30 percent chance the record is the right person [V, S2]. Vendor blogs quote 12 to 28 percent personal-to-work conversions and 55 to 75 percent single-vendor match rates; these are vendor marketing with unstated methods [M/R]. 2. **Most B2B vendors cannot or will not serve personal emails.** Cognism states it holds no personal email addresses [M, S47]. Apollo does not reveal personal emails for people in GDPR regions and defaults `reveal_personal_emails` to false [V, S31]. Apollo's terms cover only "Business Contact Information" in a professional context [V, S33]. 3. **PDL and FullContact are the two documented sources built around personal identifiers.** PDL accepts any email, a name with a supporting field, or a phone number, and returns `personal_emails`, `certifications`, `education`, `experience` and `interests` fields [V, S2, S5]. FullContact's identity graph accepts hashed email, phone, name plus postal address and returns consumer plus professional insight bundles (US-focused) [V/R, S21, S22, S29]. 4. **Reselling or embedding enrichment results to an AMS customer's end users is the exception, not the norm.** Apollo prohibits sublicensing and third-party API access [V, S33]. Brandfetch requires written consent for use "on behalf of or for the benefit of third parties" and an agreement for sublicensing [V, S83, S84]. LinkedIn's API terms forbid storing member content beyond service need and forbid commingling with third-party data [V, S69]. PDL's own page says a Data License allows use "in any way" compliant with its Acceptable Data Use Policy, but is silent on retention and redistribution [V, S12]; a reproduction of PDL's terms on a partner page says customers may not resell or sublicense [R, S17]. The real answer sits in each signed contract; none of the self-serve terms read were written for an OEM that feeds its own customers. 5. **LinkedIn-derived data is the main compliance hazard.** LinkedIn sued Proxycurl (Jan 2025), which shut down in July 2025 under a settlement [V, S63, S64]; sued ProAPIs in Oct 2025 [R, S65]; won a 2022 consent judgment against hiQ with $500,000 damages and data deletion [R, S66]. France's data regulator fined Kaspr EUR 240,000 in December 2024 for collecting contact details of LinkedIn users who had restricted visibility [V, S62]. Vendors that sell LinkedIn-sourced person data (several "API" vendors) carry supplier-side risk that flows to a licensee. 6. **Free and cheap building blocks exist for the non-person parts.** Gravatar (free API, SHA-256 email hash, avatar requests not rate-limited) [V, S86, S87]; GLEIF LEI data (CC0 per secondary sources) [R, S98]; Wikidata (CC0) [V, S96]; Companies House (600 requests per 5 minutes) [V, S99]; SEC EDGAR (10 requests per second) [R, S100]; Logo.dev (free tier 500K logo requests per month, store anything returned) [V, S85]; Brandfetch (free tier, but 30-day cache and no third-party use without agreement) [V, S83, S84]. 7. **Email verification is commodity and cheap** ($0.002 to about $0.02 per check) but verifying a webmail address proves deliverability, not identity. Hunter's own docs give webmail and disposable addresses a fixed score of 50 [V, S53]. 8. **AI web research is now priced in cents and is available as an API** (Exa search from $4 per 1,000; Parallel Task Lite $5 per 1,000; Anthropic and OpenAI web search $10 per 1,000 searches plus tokens) [V, S72, S75, S80, S81]. Anthropic's doc requires citations to be shown when API output is displayed to end users, and says web search is not available on Amazon Bedrock [V, S80]. ### 2. Person and identity providers (detail blocks) Each block: what it is, operations, matching inputs, fields, match quality, limits, pricing, licensing, privacy. Every line carries a tag and a source key. #### 2.1 People Data Labs (PDL) - **Category**: B2B-plus-consumer person and company data API and data licensor. [V, S1] - **Ownership/status**: independent, San Francisco; registered California data broker since 06/16/2020 [R, S16 (search-result view of the state registry)]. SOC 2 Type 2 and ISO 27001 stated on its own pricing and home pages [V, S6, S20]. - **Operations (Verified from docs index [S15])**: Person Enrichment (one-to-one match against "nearly three billion" profiles [V, S1 (size is a vendor claim)]); Bulk Person Enrichment (up to 100 per request) [V, S11]; Preview Enrichment (which fields are non-null); Person Identify (broad inputs, up to 20 ranked candidates, 1 credit per call regardless of results) [V, S7]; Person Search; Company Enrichment, Bulk Company Enrichment, Company Search; IP, Job Title and Skill enrichment; Person Changelog API (updates, additions, deletions, merges, opt-outs between versions) [V, S13]; Webhooks; Data License feeds [V, S12]. - **Job-change monitoring**: webhooks fire on record changes for chosen filters such as `job_change`, `work_email`, `location`; batches of up to 1,000 IDs sent about monthly, aligned with data releases; enterprise API and Data License customers only; no automatic retry [V, S10]. This is monthly batch, not real time. - **Matching inputs [V, S2]**: `profile` (social URL), `lid` (LinkedIn ID), `email` ("an email the person has used"), `email_hash` (SHA-256 or MD5), `phone` (must begin with + and country code), `name` or first/last, `birth_date`, `location` parts (postal code assumed US if no country), `company`, `school`, `pdl_id`. - **Minimum to match [V, S2]**: one identifier (`profile`, `email`, `phone`, `email_hash`, `lid`, `pdl_id`) OR a name plus at least one of company, school, location, address parts, birth date. Name-only is not accepted. - **Likelihood score [V, S2, S3]**: integer 1 to 10, logarithmic; `min_likelihood` default 2; 2 is roughly a 10 to 30 percent chance of being the requested person; guidance: use 6 or higher when accuracy matters; name only usually scores 2 to 5; name plus location rarely above 4; email only rarely above 6. `include_if_matched` lists which inputs matched [V, S2]. `required` lets you bill only responses that contain given fields [V, S2]. - **Fields [V, S5]**: `personal_emails`, `recommended_personal_email`, `work_email`, `emails`/`possible_emails` (typed current_professional, disposable, personal, professional [V, S19]); `profiles` with LinkedIn, Facebook, Twitter, GitHub URLs; `experience`, `job_history`; `education`; `certifications` (name, organization, dates); `skills`; `interests`; location current and history; birth date/year; `sex`. The fields page as read lists **no photo or profile-picture field** [V, S5; absence in the page I read, re-check the full schema]. Contact fields are gated: on the free plan emails, phones and street addresses return only true/false; Pro unlocks values [V, S5]. - **Coverage evidence [V, S8]** (Email Slice, data version 36.0, 592,188,754 records that have at least one email): `personal_emails` populated on 64.1 percent, `experience` 45.3, `linkedin_url` 34.4, `education` 26.6, `mobile_phone` 15.9, `work_email` 14.0, `github_url` 0.5. These are fill rates inside the email-bearing slice, not match rates for a new input list. Non-US coverage: not published [U]. - **Caution on email use [V, S9]**: PDL tells customers to use only `recommended_personal_email` or `work_email` for outreach, not `emails`/`personal_emails`, because PDL does not verify deliverability; validate at least quarterly. PDL does not explain how personal emails are sourced [V, S9]. - **Rate limits [V, S4]**: free 100 requests per minute, paying 1,000 per minute; 60-second fixed window; per API key; headers expose remaining credits [V, S14]. - **Billing [V, S4, S11]**: per match (HTTP 200 charged, 404 not charged); bulk charges each 200 as if sent individually. - **Pricing [V, S6]**: Free "Testing" 100 credits per month, contact data obfuscated; Pro from $98 per month for 350 credits, annual saves 20 percent, volume discounts; Enterprise custom with annual contract, extra bundles, data license feeds, SSO. That implies $0.28 per credit at the entry tier [I from V: 98/350]. A third-party reproduction of PDL's self-serve table lists $0.28, $0.265 and $0.25 per credit across higher tiers and $0.10 per company credit [R, S18 (primary blocked with 403), S18b]. One credit per successful person, company or IP enrichment [R, S18b]. - **Licensing**: Data License page: customer may use data on premise "in any way" compliant with the Acceptable Data Use Policy; retention after expiry and redistribution not addressed on that page [V, S12]. The Services Subscription Agreement and Acceptable Data Use Policy are at privacy.peopledatalabs.com but render only a "Privacy Center" shell to the fetch tool [U, S20]. A partner-platform page reproducing PDL terms says customers may not "resell, sublicense, distribute or otherwise provide access" to PDL services or derived data to third parties [R, S17]. A comparison site claims PDL terms permit embedding in a redistributed product [M, S19b; conflicts with S17, unresolved]. - **Privacy/litigation**: California registered data broker; opt-out form at peopledatalabs.com/opt-out-form; deletion of PDL's copy does not remove copies already delivered to customers [R, S16, S16b]. A 2023 press analysis of breaches from 2012 to 2021 attributed 40 percent of the total (179 million leaked records) to PDL; details of the incident were not verified [R, S16b]. No GDPR statement or EU data-residency page found [U]. Privacy-law implication: PDL's own docs show records are opt-out-able and changelog includes opt-outs, so a licensee that caches results must process opt-outs [I from V S13]. #### 2.2 FullContact (Ziff Davis) - **Category**: consumer identity resolution and enrichment (US-centred); marketer and fraud use cases. [R, S29] - **Ownership/status**: acquired by Ziff Davis, Q4 2024 (CB Insights says Oct 2024; Wikipedia says Q4) [R, S30]; operates inside Ziff Davis's martech division [R, S29]. Not Allstate or Dentsu. - **Products [R, S29, V S26]**: Enrich (person insights), Resolve (persistent PersonID from fragmented identifiers), Acumen (anonymous visitor identification; published $99 and $300 per month tiers), Identity Streme (first-party ID mapping: enrichment, dedupe, verification, amplification). Changelog lists `identity.mapResolve` (map plus resolve in one call) [V, S27 (titles only, undated)]. - **Graph size**: docs say 240M US profiles, 30 million+ updates a day, consumer data updated quarterly, MAIDs weekly, graph rebuilt biweekly [V, S21 (vendor claims)]; a competitor review says 248M profiles, 900+ attributes [R, S29]. - **Inputs [V, S22]**: `email(s)` (clear, MD5, SHA-256), `phone(s)` (country code for international), `profiles` (URLs, IDs or usernames; Facebook and Instagram not supported), `name` (full or given plus family), `location` (address lines, city, region, postal code), `maids`, `recordId`, `personId`, `li_nonid`. **Name is only queryable together with a postal address**, and the reverse. Incomplete combinations are ignored if a valid email is also sent. Up to 25 of each identifier type per request. - **Fields [V, S23, S24, S25]**: Insights Bundles enabled per plan. Social bundle: `profiles.{network}.url`, username, bio, followers; `photos[]` with label "avatar"; `urls[]`. Core Professional: `details.employment` (name, title, domain), `profiles.linkedin.url`, gender, emails, phones. Other bundles listed in the docs index: Core Segmentation, Interests, Individual, Demographic, Professional, Consumer Email, Employment History, Premium Phone, Email Hash Activity, Email Risk, MAIDs, Premium Address, Vertical [V, S26]. Education and certifications: not shown in the pages read [U]. - **Confidence**: multi-field doc describes no confidence or likelihood field [V, S22]; the bundles doc says a match requires "high enough confidence" to link to a Person ID [V, S25]. Match rates: none published [U]. - **Personal email**: docs do not specifically address it [V, S29 absence]. Inferred from architecture: an email-keyed consumer graph is the one design here where a personal email is the natural key [I]. - **Latency/limits**: vendor claims 40 ms to 150 ms [M, S29]; rate limits not in pages read [U]. Free developer key with 1,000 matches [R, S29]. - **Pricing**: "pay for matches, not lookups; if we don't return data, you don't pay" [V, S28 (marketing line on official page)]; no numbers; Enrich and Resolve require sales [R, S29]; bundles priced under Plan and Billing [V, S25]. - **Licensing**: caching, storage and redistribution terms not in docs read [U]. Batch enrichment by contacting sales [V, S25]. - **Privacy**: data is consumer-level (public records, voter registrations, property, surveys, purchase data, professional data per docs) [V, S21]. This is exactly the category subject to CCPA data-broker and state-law scrutiny; FullContact's California data broker registration was not checked [U]. #### 2.3 Apollo.io - **Category**: B2B prospecting database with API. [V, S31] - **Operations [V, S31, S32]**: People Enrichment (`POST /api/v1/people/match`), Bulk People Enrichment (up to 10), Organization Enrichment, Bulk Organization Enrichment, People and Organization Search, job postings, news search, waterfall email and phone enrichment, webhooks for phone and waterfall results (HTTPS required; `poll_only` alternative). - **Inputs [V, S31]**: `first_name`+`last_name` or `name`; `email` or hashed email (MD5/SHA-256); `domain` or `organization_name`; `linkedin_url`; Apollo `id`. Name alone may return 200 with no record enriched. - **Fields [V, S31]**: name, title, headline, LinkedIn URL, photo URL, email and `email_status`, location, `employment_history`, seniority, departments; organization firmographics (industry, employee estimate, revenue, funding, technologies). Education and certifications: not listed [U]. - **Confidence [V, S31]**: `match_confidence` = high, medium, low, none; credit charging depends on it (no demographic credit when none). - **Personal email [V, S31]**: `reveal_personal_emails` defaults to false; "Apollo won't reveal personal emails for people in GDPR-compliant regions"; setting true may consume credits. - **Credits [V, S32]**: people enrichment 1 to 9 credits (1 for demographics or email, plus 8 if a mobile number is returned); organization enrichment 1 per organization; waterfall email typically 1 to 4 (sometimes 20+), phone 8 to 25 (sometimes 45+). Zero credits when nothing credit-consuming is found. - **Rate limits [V, S31, S32]**: example 429 shows 600 calls per hour for the endpoint; real limits vary by endpoint and plan, viewable in Settings. - **Pricing [V, S34]**: pricing page lists plan names (Free, Basic, Professional, Custom) but gave no dollar figures in the fetch; per-credit price not published there [U]. API access: page states API on Custom; docs imply broader [V S34 vs S32 unresolved]. - **Licensing [V, S33]**: license is "solely for your internal business purposes"; may not resell, sublicense or provide third parties access to any API; may not distribute Business Contact Information to third parties; may not use Business Contact Information to train AI/ML models (internal, non-commercial exception); "perpetual" licence to keep data already incorporated into your records but tied to the subscription term (ambiguous); termination: you may keep BCI already incorporated, subject to restrictions. Personal emails not expressly addressed; terms are scoped to professional-context business contact info. - **Privacy/litigation**: no Apollo-specific lawsuit found in the search that was run [U]. LinkedIn "removes Apollo" premise in brief: not supported by any result [U]. #### 2.4 ZoomInfo (GTM.AI developer platform) - **Category**: largest B2B contact and company database. [R] - **API [R, S35]**: GTM.AI REST API `api.zoominfo.com/gtm`; OAuth 2.0 (authorization code with PKCE, "required for partner apps"; or client credentials); scopes for company, contact, intent, news, scoops; contact and company enrich up to 25 records per call; search endpoints free but return no emails or phones; legacy API deprecated with no shutdown date; MCP server enriches 10 records per call and ZoomInfo prohibits using MCP data for AI model training. - **Credits [R, S35]**: 1 data credit per new contact or company record; no match means no charge; re-enriching within 12 months free on enterprise contracts; self-serve data credits $0.38 each at the $1,000 spend tier; AI credits $0.05 flat; new accounts get 1,000 data and 1,000 AI credits; page conflicts on credit expiry. - **Rate limits [R, S35]**: Builder 5/s, 10,800/h, 129,600/day; Standard 25/s; Scaling 35/s. - **Match inputs**: not listed in the source I could read [U]. - **Personal email**: ZoomInfo is a B2B contact database; no personal-email enrichment evidence found [I]. - **OEM / partner / embedding**: no ZoomInfo OEM program text found. The "PKCE required for partner apps" line shows a partner-app route exists [R, S35]. Storage, caching and redistribution terms "sit in the customer contract" [R, S35]. API access is "not included in base plans" and one unverified forum figure puts enterprise API at $50K per year [R/U]. - **Related**: ZoomInfo owns NeverBounce (email verification) [R, S37]. #### 2.5 Clearbit and HubSpot Breeze Intelligence - **Status [R]**: HubSpot acquired Clearbit (Nov 2023); Clearbit Connect and the free tools ended by April 2025; the free Logo API (`logo.clearbit.com`) was scheduled to shut down 1 Dec 2025 and HubSpot recommended Logo.dev [R, S38]; Brandfetch's docs say it did shut down in Dec 2025 [R, S39]. - **Standalone Enrichment API**: one vendor blog says it is "a sunset product in 2026" and that classic endpoints survive only in legacy integrations; it cites other blogs and sells a competing product [R, S40]. HubSpot's Clearbit page does not mention API access or pricing [V, S42]. HubSpot's own changelog URL for the sunset returned 404 to the fetch tool [U, S43]. **Whether a new customer can still buy a standalone Clearbit API key: Unavailable.** - **Breeze Intelligence pricing [V, S41]** (HubSpot-hosted pricing guide, document date 6 Sept 2024, so possibly stale): 100 credits $30 per month ($0.30), 1,000 credits $150 ($0.15), 10,000 credits $700 ($0.07); one credit = one enriched company or contact record; credits reset monthly, no rollover; one-time bulk enrichment $5,000 for up to 1 million records. A late-2026 blog quotes different numbers ($45 per month for 100 credits) and says it requires a paid HubSpot subscription [R, S40b]. - **Fit**: works inside HubSpot only (reported) [R]; no API mentions in the pricing guide [V, S41]. A reseller or an AMS cannot use it as an embedded data source [I]. #### 2.6 Clay - **Category**: workflow and waterfall orchestration over 150+ data providers plus AI research agent (Claygent). Clay is a buyer of other vendors' data, so data rights come from each underlying provider. [V, S44] - **Pricing [V, S44]**: Free $0 (100 Data Credits, 500 Actions per month); Launch from $167 per month annual ($185 FAQ) with 2,500 Data Credits and 15,000 Actions; Growth from $446 per month annual ($495 FAQ) with 6,000 Data Credits and 40,000 Actions; Enterprise custom, annual. Data Credits from $0.05 each; Actions under $0.01. No result means no charge. Top-ups at a 30 percent premium on Launch and Growth (a search summary says 50 percent; conflict) [V/R, S44, S45]. The page's own sections conflict on Launch and Growth prices. - **API**: HTTP API integrations on Growth; Clay API on Enterprise; bring-your-own-key runs cost one Action [V, S44]. Claygent included on all plans; about 80 percent of models cost a flat Data Credits count; token-heavy models billed on token use [V, S44]. - **Use for Rhythm**: a back-office experimentation tool for staff, not an embeddable provider; contract is a SaaS subscription with enterprise custom CSA and DPA [V, S44]. OEM terms: not found [U]. #### 2.7 Cognism - **Category**: B2B contact data with EMEA emphasis. [M] - **API [V, S46]**: docs index lists Search (companies, persons), Count, Dictionaries, rate-limit and credits pages; credits meter contact-detail reveals and search tokens meter returned records. **No "Enrich API" section appears in the index** (brief assumed one) [V, S46 absence]. - **Personal email**: states it holds no personal email addresses and delivers business emails only; checks mobile numbers against do-not-call lists (13 European lists per ZoomInfo-hosted review; other reviews say 15+) [M/R, S47]. - **Pricing**: unpublished; reviews cite about $15,000 per year starting contracts and $20,000 to $25,000 for top tier [R, S47]. - **Fit**: B2B work-email only; not suited to personal-email members [I]. #### 2.8 Lusha - **Pricing [V, S48]**: Free 40 credits per month; Starter $49.90 per month ($37.45 annual) for 400 credits; Pro $69.90 ($52.45) for 600; Premium $399.90 ($299.95) for 3,400; Scale custom. Verified email reveal 1 credit; phone reveal 5 credits. One credit balance covers workspace, extension, API and MCP; the page does not say which plans include API [V, S48]. (Phone credit cost reportedly fell from 10 to 5 in spring 2026 [R, S50].) - **API [V, S49; R, S50]**: Person Enrichment and Prospecting endpoints; docs landing page gives no inputs, fields or limits [U]. One review says billing is per row returned and an empty call is not charged [R, S50]. API plan gating contradictory across sources [R, S50]. - **Fit**: browser-extension heritage; B2B. [I] #### 2.9 RocketReach - Pricing page lists "personal email" as included in plans, API access "varies by plan", and export credits; no numbers in the fetch [V, S51]. Third-party figures conflict widely ($39/$79/$169 monthly vs $69/$119/$209) [R, S52]. - API docs returned 403 [U]. Personal-email inclusion is the only distinguishing feature worth following up; direction of lookup (name and employer to email versus personal email to profile) is **Unavailable**. #### 2.10 Hunter.io - **Operations [V, S53]**: Email Finder (domain or company plus name or LinkedIn handle; `provenance` found vs inferred; status valid, accept_all, unknown), Email Verifier (valid, invalid, accept_all, webmail, disposable, unknown; up to 20 seconds; 202 then poll), **Combined Enrichment** (`/combined/find`: email in, person and company out; 404 if none), Company Enrichment by domain, Discover (companies by filters or natural-language query; free call). - **Rate limits [V, S53]**: Finder, Combined, Company Enrichment 15 per second and 500 per minute; Verifier 10 per second and 300 per minute; Discover 5 per second and 50 per minute. - **Scores [V, S53]**: Verifier `score` is deliverability; **webmail and disposable addresses get a fixed value of 50**; Domain Search `confidence` is an estimated probability based on sources. - **Privacy [V, S53]**: HTTP 451 `claimed_email` means the person asked that their data not be processed; Combined Enrichment omits phone numbers people asked not to share. - **Pricing [V, S54]**: Free 50 credits; Starter $49 per month for 2,000 credits ($34 annual); Growth $149 for 10,000 ($104 annual); Scale $299 for 25,000 ($209 annual). Each verification costs 0.5 credit; no verification allowance separate from shared credits. Data Platform (API-only bulk): example $6,500 for 1,000 search credits plus 200,000 verification credits. API listed on all plans. - **Fit**: company-domain driven; a personal email (gmail.com) gives no company, so Finder/Combined cannot infer an employer [I]. ### 3. Person and identity providers (condensed table) Columns: provider | what it is | personal-email fit | pricing | licensing, legal, status | tags. | Provider | Operations and inputs | Personal-email fit | Pricing | Licensing / legal / status | | ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Dropcontact | Algorithmic enrichment (no purchased database, per reviews); name plus company in, verified business email and company data out [R, S55] | Low; designed around professional email construction [I] | About EUR 24 per month for 1,000 credits per one source; EUR 79 per month Starter per another; tiers reportedly consolidated Aug 2026; official page not fetched [R/U, S55] | GDPR-oriented design corroborated across reviews [R]; match rate claims 60 to 70 percent overall [R]; "acquired" premise: nothing found [U] | | Datagma | Real-time enrichment; email, LinkedIn URL inputs [R, S56] | Not documented [U] | Plans $39 to $249 per month; email 1 credit, mobile 30 credits per a secondhand quote; sources disagree on rollover [R, S56] | Terms [U] | | Enrich.so (enrichmentapi) | Person plus company from email lookup, waterfall-style per vendor pages [M, S57] | Claims "full person + company from email" [M]; no per-lookup credit cost found [U] | Growth $49 per month for 100,000 credits ($0.0005 per credit); Scale $149 for 500,000; Pro $499 for 2.5M; free 100 credits [M, S57] | Self-published comparison pages; terms [U] | | Crustdata | Person Enrich, Company Enrich, Search, live (real-time web) endpoints, **Person Watchers** [V, S58, S59] | Needs LinkedIn URL for watchers [V, S59]; enrich inputs not in the page read [U] | $20 free credits on signup [R, S60]; person base 1 credit, 5 for real-time per one page; credit dollar price [U]. Watcher cost per changed person delivered: 5 credits at 30-day freshness, 10 at 14, 20 at 7, 50 at 3, 150 at 1 day; baseline run free; up to 10,000 people per watch; 10 requests per minute on `/watch/person` [V, S59] | Data sourcing and compliance statement not in pages read [U]. Vendor claims 800M+ people [M]. LinkedIn-derived risk [I from S63, S64] | | Prospeo | Email finder, LinkedIn URL enrichment, MCP server [R, S61] | Work-email oriented [R] | $39 to $49 per month entry; credit costs conflict (0.5 to 2 credits per email, 10 per mobile) [R, S61]; reviews say credits may be charged on no-match [R] | Terms [U] | | Kaspr | Chrome extension revealing contacts of LinkedIn profiles; database about 160M contacts [V, S62] | Not applicable; LinkedIn-driven [V] | API pricing not found [U] | **CNIL fined Kaspr EUR 240,000 on 5 Dec 2024** (legal basis, retention, transparency, access rights); ordered to stop collecting from people who limited visibility and to delete or notify [V, S62] | | Seamless.ai | Sales database, Chrome extension | B2B [I] | Quote-only; one third-party estimate $44K to $90K per year for 25 users [R, S62b] | Not researched further [U] | | Lead411, LeadIQ, UpLead, Snov.io, Wiza | B2B contact tools | B2B [I] | UpLead Essentials $99 per month for 170 credits (monthly billing); others [U] | Not researched further after search budget ended [U] | | Proxycurl / Nubela | LinkedIn profile API | n/a | n/a | **Shut down 4 July 2025** to comply with a settlement with LinkedIn [V, S63]; LinkedIn says it "successfully resolved" the suit (alleged scraping, fraud, trademark misuse) [V, S64]; a permanent injunction requiring deletion of unauthorized LinkedIn data and notice to customers is reported by a search summary but was not on LinkedIn's page [R/V, S64]. Successor NinjaPear [V, S63] | | Coresignal | Base and Multi-source employee, company, job-posting records via API [R, S67] | Needs LinkedIn-style profile identifiers [I] | Credits across endpoints; 7-day trial with 2,000 credits per its docs; plans $49 to $5,000 per month per third parties; per-record credits vary by source (10 vs 20 vs 2) [R, S67] | LinkedIn-sourcing and ToS question unanswered [U] | | The Companies API | Company enrichment by domain; 50M-company database [R, S68] | Company only | Startup $95 per month for 50,000 credits; Scaleup $295 for 250,000; Enterprise $595 for 500,000; extra credits $2.50, $1.75, $1.50 per 1,000; 50 to 1,000 requests per second [R, S68]; credits per domain enrichment not stated [U] | Terms [U] | | UserGems / Champify | Job-change and champion tracking for sales | Tracks business job moves; CRM-integration product [I] | Champify Core $2,000 per month (15,000 tracked contacts), Pro $3,000 (40,000), Enterprise $6,000 (150,000); UserGems unpublished, estimated $15K to $100K+ per year [R, S71]; no API pricing found [U] | Annual contracts typical [R] | | Persana, Common Room | GTM/AI prospecting; community and signal platform | Not researched | [U] | [U] | | LinkedIn (official APIs) | See section 6 | Member-consent only [I] | n/a | See section 6 | | Pipl | Identity search for fraud, KYC and investigations (insurance, finance, law enforcement) [R, S117] | High in principle (email, social handles, phones) but sold to investigators; not marketed for CRM [I] | Review sites list "from $0.10" per feature; no rate card [R, S117] | Use-purpose restrictions likely strict; terms [U] | | Windfall (wealth) | Household net worth match; weekly rebuild [M, S118] | Matches by identity data [I] | Quote only [R, S118] | Wealth screening of members is a sensitive purpose requiring policy review [I] | | iWave / DonorSearch (nonprofit screening) | Philanthropic capacity screening; iWave from $4,150 per user-year (G2) [R, S118]; DonorSearch reports integration with 55+ CRMs, 20 with direct API access; EverTrue acquired DonorSearch Sept 2025 (unverified) [R, S118] | Built for nonprofit donor files (name plus address) [I] | See left | Not an enrichment API for membership data in the AMS sense [I] | ### 3A. Capability, input and ownership matrices (one row per provider) Legend: Y = documented on a primary page [V]; y = reported by a secondary source [R]; n = documented as not offered; ? = not found [U]. "Bulk" = batch endpoint or file service. "Monitor" = continuous change notification. #### 3A.1 Operations offered | Provider | Person enrich | Company enrich | Identity resolve | Verify | Search | Job-change monitor | Bulk | Webhook / continuous | | ----------------- | -------------------------- | -------------------- | ----------------------- | ----------------------- | --------------------------- | --------------------------------- | ------------------------------- | ---------------------------------- | | PDL | Y | Y | Y (Identify) | n | Y | Y (monthly webhook, enterprise) | Y (100 per call) | Y (monthly) | | FullContact | Y | y (company endpoint) | Y (Resolve) | y (via Identity Streme) | ? | ? | Y (contact sales) | ? | | Apollo | Y | Y | n | Y (email status) | Y | ? | Y (10 per call) | Y (phone/waterfall result webhook) | | ZoomInfo | y | y | ? | n | y (free search) | y (intent add-on, not job change) | y (25 per call) | ? | | Clearbit / Breeze | y (in HubSpot) | y (in HubSpot) | n | n | n | n | y (one-time bulk $5,000 per 1M) | n | | Clay | Y (via vendors) | Y (via vendors) | n | Y (via vendors) | Y | y (signals on Growth) | Y | Y (webhooks on Growth) | | Cognism | y | y | n | n | Y (Search API) | ? | ? | ? | | Lusha | Y | ? | n | n | Y (Prospecting) | ? | ? | ? | | RocketReach | y | y | n | y | y | ? | y | ? | | Hunter | Y (Combined) | Y | n | Y | Y (Discover, Domain Search) | n | y | n | | Dropcontact | y | y | n | y | n | n | y | ? | | Datagma | y | y | n | y | ? | ? | ? | ? | | Enrich.so | y | y | ? | ? | ? | ? | ? | ? | | Crustdata | Y | Y | n | n | Y | Y (Person Watcher) | Y (10,000 per watch) | Y (webhook, Slack, email) | | Prospeo | y | ? | n | y | ? | ? | y | ? | | Coresignal | y | y | n | n | y | ? | y | ? | | The Companies API | n | y | n | n | y | n | ? | ? | | Diffbot | Y (KG enhance) | Y | y | n | Y (DQL) | ? | Y | ? | | Exa | Y (Agent, Websets) | Y | n | n | Y | Y (Monitors) | Y | Y (Monitors) | | Parallel | Y (Task API) | Y | n | n | Y | Y (Monitor API) | Y | Y | | Melissa | Y (Personator) | n | Y (Personator Identity) | Y (address, email) | n | Y (Move Update) | Y | n | | AtData | ? | n | Y (identity layer) | Y (email) | n | y (activity signals) | y | ? | | Versium | y | y (C2B) | y | ? | n | n | y | ? | | Gravatar | Y (avatar, public profile) | n | n | n | n | n | n | n | | Brandfetch | n | Y (brand data) | n | n | Y (Brand Search) | n | y (Custom: batch) | n | | Logo.dev | n | Y (logo, brand) | n | n | Y (Search) | n | n | Y (webhooks on paid plans) | #### 3A.2 Matching inputs accepted | Provider | Any email | Personal email explicitly handled | Name + company | Name + location | LinkedIn URL | Phone | Domain | Minimum stated | | ----------- | ------------------- | ------------------------------------------------------------ | ------------------------------- | -------------------------------- | ------------ | ----------------- | --------------------------------- | ------------------------------------------------------ | | PDL | Y | Y (`personal_emails`, `recommended_personal_email` returned) | Y | Y | Y | Y (+country code) | Y (company field accepts website) | one identifier, or name + one supporting field [V, S2] | | FullContact | Y (clear or hashed) | y (consumer graph) | ? | Y (name requires postal address) | Y (profiles) | Y | n | valid identifier or name + address pair [V, S22] | | Apollo | Y | n (personal emails withheld in GDPR regions) | Y (with domain or org name) | n | Y | n | Y | name alone may return no record [V, S31] | | ZoomInfo | ? | ? | ? | ? | ? | ? | ? | not listed in source read [U] | | Hunter | Y (Combined) | n (webmail score fixed at 50) | Y (Finder needs domain/company) | n | Y (handle) | n | Y | domain or company for Finder [V, S53] | | Lusha | ? | ? | ? | ? | ? | ? | ? | docs landing page gives none [U] | | Crustdata | ? | ? | ? | ? | Y (watchers) | ? | ? | LinkedIn URL for watchers [V, S59] | | Diffbot | ? | ? | y | ? | ? | ? | y | [U] | | Melissa | y | y (consumer email verification/append) | n | Y (name + address) | n | y | n | name + address, phone, email or address key [R, S116] | #### 3A.3 Ownership, privacy posture, known issues | Provider | Owner / status (late 2026) | Privacy posture evidence | Litigation or compliance issues found | | --------------------------------------------------------------------------------------- | --------------------------------------------- | -------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | PDL | Independent | CA registered data broker (2020); opt-out form; ISO 27001, SOC 2 Type 2 [R/V] | None found beyond a 2023 breach-tally mention [R] | | FullContact | Ziff Davis (Q4 2024) [R] | Consumer data from public records, voter files, purchases, per its docs [V, S21] | None found; CA registration unchecked [U] | | Apollo | Independent | GDPR-region personal-email carve-out [V] | None found [U] | | ZoomInfo | Public company; ticker GTM since May 2025 [R] | Terms in contract [U] | None found in this search [U] | | Clearbit | HubSpot (since Nov 2023) [R] | | Free tools wound down 2025 [R] | | Clay | Independent | Enterprise DPA available [V, S44] | | | Cognism | Independent | Do-not-call screening, no personal emails (claims) [M] | | | Lusha, RocketReach, Dropcontact, Datagma, Prospeo, Seamless, LeadIQ, UpLead, Snov, Wiza | Not verified | Mostly [U] | Dropcontact is the only one with a corroborated no-database GDPR design [R] | | Kaspr | Independent | | CNIL fine EUR 240,000 (Dec 2024) [V] | | Proxycurl / Nubela | Closed 4 July 2025; NinjaPear successor | | LinkedIn lawsuit and settlement [V] | | Crustdata, Coresignal | Independent | Sourcing statements not found [U] | LinkedIn-sourcing exposure [I] | | AtData | Experian (Feb 2026) [R] | Email-activity network of billions of signals [M] | None found | | Melissa | Independent | | None found | | Brandfetch, Logo.dev | Independent | Public web logos; brand owners can request blocking [V, S84] | | | Gravatar | Automattic | Public profiles are public by member action [V, S89] | | ### 4. Consumer-data appenders (relevant to personal emails) | Provider | Ownership / status | What it does with a personal email | Pricing | Notes | | --------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | AtData (formerly TowerData plus FreshAddress) | Formed 2021 from TowerData and FreshAddress; **Experian acquired AtData in February 2026**; Experian says the deal adds 10 billion+ email addresses [V/R, S111, S112] | Email validation, identity resolution, "resolve and link profiles", consumer insights; claims data on 98 percent of US emails [M, S111]; one review says TowerData appended demographics, social profiles and purchasing indicators [R, S113] | Quote-only; a competitor's page cites $0.20 to $0.12 per match and $0.01 to $0.15 per API match, unverified [R, S113]; AtData's pricing page shows no prices [V, S119] | Deterministic activity signals (150 billion per month, per press) [R, S112]. Whether the email-to-name/postal append still ships under Experian: [U] | | Versium REACH | Independent data append (B2C and B2B) | B2C contact append; email, phone, address enrichment [R, S114] | AWS Marketplace listing: match credits $0.04 per unit [R]; Datarade: $300 per month or $3,600 per year for a B2C append licence [R]; reviewer estimate $0.05 to $0.075 per match with $125 minimum per file [R] | Self-reported phone match above 70 percent; no email-plus-name match rate found [M/U] | | Melissa | Long-standing address, identity and data-quality vendor | **Personator Consumer** and **Personator Enrich** verify and append consumer identity from name plus address, phone, email or address key [R, S116 (search summary; reference page 404)] | **Verified**: Personator Consumer with Move Update (the NCOA-class service) $12,300 per year for 1,000,000 records; Personator Enrich $14,300 per year for 1,000,000 records; PAYG credit bundles $40 for 10,000, $350 for 100,000, $3,200 for 1,000,000; global address 10 credits, US 3 credits per FAQ (card says 10; conflict) [V, S115]. Free global tier 250 records per month | Personator SSN support ended 2023; ethnic-code fields deprecated 2026-03-17 [V, S116 nav list]; useful for address validation and move updates, not employer or certification | | FullContact Resolve | See 2.2 | Hashed or clear email, phone, name plus address into PersonID | Sales | US consumer graph | | Experian, BriteVerify, FreshAddress | Experian's address and email products; BriteVerify (Validity) not researched | [U] | [U] | [U] | Takeaway (Inferred): appenders map email to postal address, phone, demographics and sometimes social handles. None document employer history, certifications or license data. They help with **address hygiene and identity confirmation**, not with professional affiliation. ### 5. Email verification (cost per check) | Provider | Published price | Source | Notes | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------- | ------------------------------------------------------- | | ZeroBounce | Pay-as-you-go from $39 for 2,000 credits ($0.0195 each); ONE subscription $99 per month for minimum 10,000 credits ($0.0099 each), rollover; free 100 credits per month; enterprise above 1M [V, S104] | S104 | Rate-limit numbers not stated on pricing page [U] | | Emailable | $38 for 5,000 credits ($0.0076); subscription $32.30 for 5,000 ($0.0065); packs to 1M; 250 free credits; unknown results refunded [V, S105] | S105 | Rate limit in API docs (not fetched) [U] | | Bouncer | 1,000 for $8 ($0.008); 5,000 for $35; 10,000 for $60; 50,000 for $250; 100,000 for $400 ($0.004); 250,000 for $750; 500,000 for $1,250; 1,000,000 for $2,000 ($0.002); 100 free credits [V, S106] | S106 | | | NeverBounce | About $0.008 pay-as-you-go per third-party guides ($8 per 1,000); owned by ZoomInfo; pricing page 403 to the fetch tool [R/U, S107] | S107 | Credit expiry disputed between sources [R] | | Kickbox | Third-party tracker: $10 per 1,000 down to about $3 per 1,000 at 1M ($2,999); official page blocked [R/U, S108] | S108 | | | MillionVerifier | Pricing page returned 404 [U] | | | | Mailgun Validate | Extra validations from $1.20 per 100 on Basic and Foundation ($0.012 each) and $0.80 per 100 on Scale ($0.008); Scale includes 5,000 per month [V, S109] | S109 | Bulk list upload exists; separate bulk price not shown | | SendGrid validation | Page fetched did not state a price [U, S110] | | | | Hunter Verifier | 0.5 credit per verification; about $0.012 each on Starter [I from V, S54] | S53, S54 | Webmail and disposable get a fixed score of 50 [V, S53] | Takeaway (Inferred): for roughly 50,000 contacts a full verification pass costs about $100 (Bouncer at volume) to $400 (Emailable or Mailgun at small packs). Verification returns deliverability status, so a personal Gmail returns "webmail" or "accept all"-style outcomes rather than proof that the mailbox belongs to the member. ### 6. LinkedIn and "LinkedIn-derived" data: what is permitted | Topic | Evidence | | ---------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Official API terms (storage) | "You must not capture, copy, cache, or store any Content" except for the duration necessary to provide the application's services to its users; derived, anonymized and aggregated versions still count as Content; profile data storage only where legally required and with the member's valid consent; refresh only when the member is actually using your application, not on an automated schedule [V, S69] | | Official API terms (combining and selling) | May not combine Content with other LinkedIn content (including non-official content); may not commingle Content with third-party data so users cannot attribute it; may not sell, rent, disclose or share Content with third parties; may not use for credit, employment or housing decisions [V, S69] | | Deletion duties | Delete all Content collected on behalf of a user when they ask or close their account; delete on LinkedIn's termination or request [V, S69] | | Enrichment/CRM | The page does not expressly address enrichment or CRM use; the limits above govern [V, S69] | | Profile pictures | Treated as Profile Data under the same storage limits [V, S69] | | User Agreement section 8.2 | Prohibits scraping tools, copying information from the services (including through search tools or data brokers) and selling or monetizing the services' data, and unauthorized automated access [V, S70] | | Enforcement | Proxycurl: suit Jan 2025, closed 4 July 2025 under settlement [V, S63, S64]; ProAPIs suit Oct 2025 (millions of fake accounts alleged), outcome not found [R, S65]; hiQ: district court held user-agreement scraping ban enforceable (Nov 2022), consent judgment with permanent injunction, data deletion and $500,000 (Dec 2022) [R, S66]; Ninth Circuit CFAA holdings still favour access to public pages, but contract and state-law claims remain [R, S66] | | Third-party scraper actors (Apify and similar) | Scraping public profiles is unlikely to be a CFAA crime but breaches the User Agreement; no source found on how Apify allocates legal responsibility to its users [R, S66b] | | Inferred position | The only clearly compliant LinkedIn route for an AMS is a **member-initiated, consent-based OAuth action** with the storage limits above. Buying LinkedIn-derived records from a data vendor does not shield the buyer from the vendor's supply-side exposure (see Proxycurl, Kaspr). [I] | ### 7. Organization data | Provider | What it gives | Pricing | Licensing and limits | Tags | | ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---- | | Brandfetch | Logo API, Brand API (logos, colours, fonts, description), Brand Search API, Brand Context and Transaction APIs | Free: 100 brand fetches one-time, Logo API 1M per month, Brand Search 500K per month. Growth: $99 per month on yearly billing ($1,188 per year) or $129 monthly, 2,500 to 10,000 brand fetches per month, Logo 5 to 20M, overage $0.10 per fetch. Custom: more than 10,000 fetches, batch lookups, custom legal terms. Qualified startups and nonprofits 40 percent off first 12 months [V, S83] | **Terms [V, S84]**: use for own business purposes; use for the benefit of third parties and sublicensing need prior written consent; customers may not "make the Services or any part of their infrastructure available to any third party"; cache licence maximum 30 days from retrieval and must re-call at least every 30 days or delete; no AI training without agreement; third-party brand assets remain property of their owners and the content licence does not extend to the logos themselves. Pricing page: sublicensing data to customers or including it in your API responses requires a specific agreement with sales; Brand API responses cacheable 30 days [V, S83] | V | | Logo.dev | Logo CDN, Brand, Brandmark, Search APIs | Community free: 500K logo CDN requests, $5 API credit; Startup $33 per month (1M requests); Pro $150 (5M); Enterprise custom. API: Brand $0.05, Brandmark $0.03, Logo $0.02, Search $0.00001 per request; repeats free for 24 hours. Attribution required only on free plan for commercial projects. **On every plan you may store anything returned and serve it from your own CDN** [V, S85] | Its full terms of service not read [U]; HubSpot's changelog names Logo.dev as the Clearbit logo replacement [R, S38] | V/R | | Clearbit Logo API | Free logo by domain | Shut down 1 Dec 2025 [R, S38, S39] | Do not rely on it | R | | Google favicons | Unofficial favicon endpoint | Not researched; undocumented as a product API [U] | | U | | GLEIF LEI | Legal entity reference data, parent and child relationships; LEI covers entities that have registered (mostly financial and regulated) | Free; published under CC0 1.0 per secondary sources; free Golden Copy on AWS Open Data [R, S98]; API page confirms entity and ownership data, fuzzy search; rate limit and terms text not obtained [U, S97] | Coverage limited to LEI holders; most associations and small employers have none [I] | R/I | | Wikidata | Entity IDs, websites, logos (Commons), industries | Free; **all structured data CC0** [V, S96]; API usage policy not on that page [U] | Coverage uneven; no reliability guarantee [I] | V | | SEC EDGAR | US public-company filings | Free; maximum 10 requests per second per user, must send a descriptive User-Agent [R, S100] | US public companies only | R | | Companies House (UK) | UK company registry | Free; 600 requests per five-minute window, 429 afterwards, bans for abuse [V, S99] | UK only | V | | OpenCorporates | Global company registry aggregator | Free open-data API: default 200 requests per month and 50 per day; paid accounts remove share-alike; plans reported: Essentials GBP 2,250 per year (500 calls per month), Starter GBP 6,600 (2,500), Basic GBP 12,000 (5,000), Enterprise custom [V S94, R S95] | **Share-alike Open Database Licence**: free accounts must release the incorporating product or database under an open licence with attribution [V, S94] | V/R | | Crunchbase API | Funding and company data | Full API requires Enterprise or Applications licence; no public price; reported median contract about $20,000 per year, "from $30,000" elsewhere; free API tier reportedly discontinued [R, S101] | Licence terms [U]; resale multiplies price [R] | R | | Dun & Bradstreet Direct+ | Identity Resolution / match with confidence codes (1 to 10 scale; fraud signals on candidates at 7+), enrichment data blocks, bulk Multi-Process API above 4,000 entities [V, S102] | Usage-based, tiered, quote-only [R, S102] | After non-renewal API access ends; already exported data stays but stops updating [R, S102] | V/R | | Apollo Organization Enrichment | See 2.3: 1 credit per organization [V, S32] | | Internal-use licence [V, S33] | V | | PDL Company Enrichment | One-to-one company match; Company Schema; fields not read [V, S15, S1b]; $0.10 per credit tier reported [R, S18b] | | See 2.1 | V/R | | Diffbot Knowledge Graph | Entity enhance (person and organization) and Knowledge Graph export | Free: 10,000 credits per month, 5 requests per minute; Startup $299 per month for 250,000 credits (overage $0.001); Plus $899 for 1,000,000 (overage $0.0009); KG enhance 1 entity = 25 credits, with refresh 100 credits; export 1 entity = 25 credits [V, S103] | Plans monthly, no long-term contract [V]; data licence terms [U]; page contained AI-directed text that was ignored | V | Takeaway (Inferred): for logos the only options with a documented permission to store results are Logo.dev (every plan) and Brandfetch (30-day cache, consent for third-party use). Logo files remain third-party trademarks regardless of vendor [V, S84 for Brandfetch statement]. For firmographics, registry sources (GLEIF, Wikidata, Companies House, EDGAR) are free and redistributable or low-friction but narrow; D&B and Crunchbase are enterprise-priced. ### 8. Photos, avatars and social | Source | What is available | Terms and limits | Tags | | ------------------------- | ----------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---- | | Gravatar | Avatar image by SHA-256 email hash; profile JSON/XML/vCard via REST API v3 at `api.gravatar.com/v3` [V, S86, S87] | Profile lookups: default 1,000 requests per hour with an API key, free higher limits on request; avatar image requests do not count toward limits; `d=404` returns 404 when no avatar; default rating returns only G-rated images; sizes 1 to 2048 px; profile lookups use the hash of the account's **primary** email, so secondary-email profile requests can fail while avatars succeed [V, S86, S87]. Free API use is governed by the WordPress.com "Guidelines for Responsible Use" [V, S86, S90] | | Gravatar guidelines | | Free API access but "excessive" calls not allowed (Automattic's sole discretion); do not use user information or Content to market products that compete with Automattic; you may cache or store content but must refresh at reasonable intervals and tell users how you store it; apps need a privacy policy; do not give data to governments for surveillance; guidelines cover the Gravatar API but give no Gravatar-specific caching rule [V, S88]. Gravatar's own privacy policy says public profile information is public and appears on sites that use Gravatar when the email matches [V, S89] | | Libravatar | Federated avatar service; MD5 or SHA-256 of lowercased email; size 1 to 512 [R, S91] | Terms not read [U] | | Unavatar | Aggregates avatars from Gravatar, GitHub, X and others by identifier | Terms list "CRMs" as fair use but prohibit "harvesting or enumerating user data from public identifiers" and "systematically resolving identifiers to build user databases"; avatars remain property of their owners; no rate limits stated [V, S92] | | GitHub avatars | Public avatars and profiles via API | GitHub's acceptable-use rules: API collection is not "scraping" and falls under API terms; personal data may only be used for the purpose the user authorized; no spam, no selling personal information including to recruiters; honor removal and do-not-contact requests [V, S93] | | X (Twitter) avatars | Not researched | [U] | | LinkedIn profile pictures | Profile Data under API terms; no storage without consent; scraped photos breach the User Agreement | [V, S69, S70] | | PDL | No photo field in the fields page read [V, S5] | | | FullContact | `photos[]` entries labelled "avatar" in the Social bundle [V, S23] | Display and caching rights [U] | | Apollo | Returns a photo URL [V, S31] | Internal-use licence; display to third parties not addressed [V, S33] | | Clearbit person photos | Not verified post-Breeze [U] | | Takeaway (Inferred): an avatar the member chose to publish through Gravatar is the lowest-friction photo source (public by the member's action; free; hash-based so no name is sent). All other sources carry either third-party-use limits or unclear display rights. ### 9. AI-native research options | Option | What it does | Price | Notes | | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Exa Search API | Neural/keyword web search with contents | Search up to 10 results per 1,000 requests: lite/instant $4, auto/fast $7; each extra result $1 per 1,000; deep $12 to $15; contents $1 per 1,000 pages per content type; answer $5 per 1,000; monitors $15 per 1,000; $10 free credits monthly [V, S72] | | | Exa Agent | Fixed effort per request: minimal $0.012, low $0.025, medium $0.10, high $0.50, xhigh $1.00; metered modes at $0.10 per compute unit, $0.005 per search, **$0.02 per email enrichment, $0.07 per phone enrichment** [V, S72] | Third-party provider calls billed separately (for example Fiber.ai at $0.02 per credit) [V, S72] | | Exa Websets | Find and verify lists of entities against criteria, then enrich | Starter $49 per month for 8,000 credits (10 enrichments per Webset); Pro $449 for 100,000 (50 enrichments); a matching result costs 10 credits, non-matching 0 [R, S74]; Websets API requires a paid plan and free caps a Webset at 25 verified results [V, S73]; criteria "should be verifiable from public web pages" [V, S73] | Contact data consumes credits; people entity not documented on that page [V, S73] | | Parallel | Search API, Task API (enrichment from structured input), FindAll, Monitor | Task API per 1,000: Lite $5, Base $10, Core $25, Pro $100, Ultra $300; Search $1 (fast) to $5 per 1,000; FindAll priced per match; Monitor $3 or $10 per 1,000; 5,000 free requests per month; up to $80 signup credit [V, S75] | The page states Task API is the enrichment option for "database enrichment" [V, S75] | | Perplexity | Search API $5 per 1,000 ($1 fast); Agent API token billing with `web_search` tool at $2.50 per 1,000 invocations; `perplexity/sonar` $0.25 input and $2.50 output per 1M tokens [V, S76, S77] | Older third-party price tables ($1 per 1M tokens, request fees $5 to $14 per 1,000) conflict with current docs; single source says Sonar sunset 27 Sept 2026 [R, S82] | | Tavily | Search, extract, crawl, research | Free 1,000 credits per month; pay as you go $0.008 per credit; Project $30 per month for 4,000 credits; startup program 100,000 free credits [V, S78] | Per-endpoint credit use not on page [U] | | Firecrawl | Scrape, crawl, map, search, agent (preview) | Free 1,000 credits; Hobby $16 per month (5,000); Standard $83 (100,000); Growth $333 (500,000); Scale $599 (1,000,000); scrape or crawl 1 credit per page, search 2 credits per 10 results; JSON extraction +4 credits per page [V, S79] | No social-site restrictions on the page; page contained AI-directed text that was ignored [U] | | Anthropic web search tool | Model-driven web search with citations; domain allow/block lists; `max_uses` cap | $10 per 1,000 searches plus token costs; failed searches not billed [V, S80] | Citations must be displayed when API output is shown to end users; **not available on Amazon Bedrock** (available on Claude API, Claude Platform on AWS, Microsoft Foundry and Google Cloud basic) [V, S80] | | OpenAI web search | Tool in the Responses API | $10 per 1,000 calls; tokens extra [V, S81] | | | Browse.ai | Not researched | [U] | | | Apify LinkedIn actors | Third-party scrapers | Legality see section 6 [R] | | Takeaway (Inferred): the "research this person" use case is technically inexpensive (a few cents per member) but returns unstructured claims about people with namesake-collision risk, so it needs human confirmation and citation display. There is no published evidence of match accuracy for people research in any of these products. ### 10. Source list All accessed 2026-10-22. "Search summary" means I saw only the search-result summary of that page, not the page itself. #### 10.1 People Data Labs - [S1] https://docs.peopledatalabs.com/docs/person-enrichment-api (Verified, docs overview); [S1b] https://docs.peopledatalabs.com/docs/company-enrichment-api.md - [S2] https://docs.peopledatalabs.com/docs/input-parameters-person-enrichment-api - [S3] https://docs.peopledatalabs.com/docs/output-response-person-enrichment-api - [S4] https://docs.peopledatalabs.com/docs/reference-person-enrichment-api - [S5] https://docs.peopledatalabs.com/docs/fields - [S6] https://www.peopledatalabs.com/pricing/person - [S7] https://docs.peopledatalabs.com/docs/identify-api - [S8] https://docs.peopledatalabs.com/docs/email-stats.md - [S9] https://docs.peopledatalabs.com/docs/email-data-for-outreach.md - [S10] https://docs.peopledatalabs.com/docs/webhooks.md - [S11] https://docs.peopledatalabs.com/docs/bulk-enrichment-api.md - [S12] https://docs.peopledatalabs.com/docs/data-license.md - [S13] https://docs.peopledatalabs.com/docs/person-changelog-api.md - [S14] https://docs.peopledatalabs.com/docs/usage-limits.md - [S15] https://docs.peopledatalabs.com/llms.txt - [S16] https://oag.ca.gov/data-broker/registration/190662 (search summary); [S16b] https://joindeleteme.com/blog/peopledatalabs-opt-out-guide/ (secondary) - [S17] https://www.sayprimer.com/third-party-terms (search summary; reproduces PDL terms on a partner page) - [S18] https://support.peopledatalabs.com/hc/en-us/articles/25794271805211-Pricing (403 to fetch tool); [S18b] https://fullenrich.com/content/people-data-labs-pricing (third-party reproduction, search summary); [S19b] https://thegtmdirectory.com/compare/people-data-labs-vs-thecompaniesapi (search summary, comparison site) - [S19] https://docs.peopledatalabs.com/docs/email-types.md - [S20] https://www.peopledatalabs.com/ (home page legal links); policies at https://privacy.peopledatalabs.com/policies?name=services-subscription-agreement and ...name=acceptable-data-use-policy (JavaScript shell only) #### 10.2 FullContact - [S21] https://docs.fullcontact.com/docs/enrich-overview - [S22] https://docs.fullcontact.com/docs/multi-field-request.md - [S23] https://docs.fullcontact.com/docs/social.md - [S24] https://docs.fullcontact.com/docs/core-professional.md - [S25] https://docs.fullcontact.com/docs/person-insights-bundles.md - [S26] https://docs.fullcontact.com/llms.txt - [S27] https://docs.fullcontact.com/changelog - [S28] https://www.fullcontact.com/pricing/ - [S29] https://pipeline.zoominfo.com/sales/fullcontact-review (competitor-authored) - [S30] https://en.wikipedia.org/wiki/FullContact ; https://www.cbinsights.com/investor/fullcontact (search summary) #### 10.3 Apollo, ZoomInfo, Clearbit/HubSpot, Clay, Cognism, Lusha, RocketReach, Hunter - [S31] https://docs.apollo.io/reference/people-enrichment - [S32] https://docs.apollo.io/docs/api-pricing - [S33] https://www.apollo.io/terms - [S34] https://www.apollo.io/pricing - [S35] https://www.cleanlist.ai/blog/2026-10-16-zoominfo-api-guide (competitor-authored) - [S36] https://www.businesswire.com/news/home/20250512038027/en/ZoomInfo-Cements-Go-To-Market-Leadership-with-New-Nasdaq-Trading-Symbol-GTM-and-Launch-of-GTM-Studio (search summary) - [S37] https://emelia.io/hub/neverbounce-pricing (search summary) - [S38] https://community.hubspot.com/t/clearbit-logo-api-will-be-sunset-on-december-1-2025/134244 (search summary) - [S39] https://docs.brandfetch.com/comparisons/clearbit (search summary) - [S40] https://abmatic.ai/blog/clearbit-enrichment-api-2026 (competitor-authored); [S40b] https://www.zeliq.com/blog/hubspot-data-enrichment (search summary) - [S41] https://www.hubspot.com/hubfs/Agency/GTM%20-%20Product%20Resources%20(Justin)/Product%20Folders%20(EN)/HubSpot%20AI/Breeze%20Intelligence%20-%20Pricing%20Guide.pdf - [S42] https://www.hubspot.com/products/clearbit - [S43] https://developers.hubspot.com/changelog/upcoming-sunset-of-clearbits-free-logo-api (404 on direct fetch) - [S44] https://www.clay.com/pricing ; [S45] https://crustdata.com/reviews/clay (search summary) - [S46] https://docs.cognism.com/ ; [S47] https://www.salesrobot.co/blogs/cognism-review and https://pipeline.zoominfo.com/sales/cognism-review (search summaries) - [S48] https://www.lusha.com/pricing/ ; [S49] https://docs.lusha.com/ ; [S50] https://reply.io/blog/lusha-api/ (search summary) - [S51] https://rocketreach.co/pricing ; [S52] https://emelia.io/hub/rocketreach-preise (search summary) - [S53] https://www.hunter.io/api-documentation/v2 (first 100,000 of 292,608 characters read) ; [S54] https://hunter.io/pricing #### 10.4 Other person-data vendors, litigation, regulators - [S55] https://leadhaste.com/blog/dropcontact-pricing-2026 ; https://syncgtm.com/blog/dropcontact-review (search summaries) - [S56] https://www.saleshandy.com/blog/datagma-pricing/ (search summary) - [S57] https://enrich.so/compare/apollo (search summary) - [S58] https://docs.crustdata.com/ ; [S59] https://docs.crustdata.com/watcher-docs/person/entity.md ; [S60] https://crustdata.com/pricing2 (search summary) - [S61] https://fullenrich.com/content/prospeo-pricing ; https://www.clay.com/university/guide/prospeo-integration-overview (search summaries) - [S62] https://www.cnil.fr/en/data-scraping-kaspr-fined-eu240000 ; https://www.edpb.europa.eu/news/data-scraping-french-supervisory-authority-fined-kaspr-eu240-000_da ; [S62b] https://amplemarket.com/blog/how-much-does-seamless-ai-really-cost (search summary) - [S63] https://nubela.co/blog/goodbye-proxycurl/ - [S64] https://news.linkedin.com/2025/LinkedInWinsLegalBattleToProtectMemberData - [S65] https://adnews.com.au/news/linkedin-files-lawsuit-against-data-scraping-company (search summary) - [S66] https://www.proskauer.com/blog/hiq-and-linkedin-reach-proposed-settlement-in-landmark-scraping-case (search summary); [S66b] https://thunderbit.com/blog/is-scraping-linkedin-legal (search summary) - [S67] https://docs.coresignal.com/pricing (search summary) - [S68] https://www.thecompaniesapi.com/pricing (search summary) - [S69] https://www.linkedin.com/legal/l/api-terms-of-use ; [S70] https://www.linkedin.com/legal/user-agreement - [S71] https://marketbetter.ai/blog/usergems-pricing-2026/ ; https://marketbetter.ai/blog/champify-pricing-breakdown-2026/ (search summaries, competitor-authored) #### 10.5 AI research - [S72] https://exa.ai/pricing ; [S73] https://exa.ai/docs/websets/api/overview ; [S74] https://websets.exa.ai/billing (search summary) - [S75] https://parallel.ai/pricing - [S76] https://docs.perplexity.ai/docs/getting-started/pricing ; [S77] https://docs.perplexity.ai/docs/agent-api/models ; [S82] https://developer.puter.com/tutorials/perplexity-api-pricing/ (search summary) - [S78] https://www.tavily.com/pricing ; [S79] https://www.firecrawl.dev/pricing - [S80] https://platform.claude.com/docs/en/docs/agents-and-tools/tool-use/web-search-tool ; [S81] https://developers.openai.com/api/docs/pricing #### 10.6 Brand, photos, organizations - [S83] https://brandfetch.com/developers/pricing ; [S84] https://brandfetch.com/terms - [S85] https://www.logo.dev/pricing - [S86] https://docs.gravatar.com/rest/api-data/ ; [S87] https://docs.gravatar.com/general/images/ ; [S88] https://developer.wordpress.com/guidelines/ ; [S89] https://automattic.com/privacy/ ; [S90] https://docs.gravatar.com/api - [S91] https://wiki.libravatar.org/api/ (search summary) - [S92] https://unavatar.io/terms ; [S93] https://docs.github.com/en/site-policy/acceptable-use-policies/github-acceptable-use-policies - [S94] https://api.opencorporates.com/documentation/API-Reference ; [S95] https://opencorporates.com/pricing/ (search summary) - [S96] https://www.wikidata.org/wiki/Wikidata:Licensing - [S97] https://www.gleif.org/en/lei-data/gleif-api ; [S98] https://registry.opendata.aws/lei/ (search summary; CC0 claim secondary) - [S99] https://developer-specs.company-information.service.gov.uk/guides/rateLimiting - [S100] https://www.sec.gov/os/accessing-edgar-data (search summary) - [S101] https://pipeline.zoominfo.com/sales/crunchbase-api (search summary, competitor-authored) - [S102] https://directplus.documentation.dnb.com/html/pages/IdentityResolutionAPIs.html (search summary) - [S103] https://www.diffbot.com/pricing/ #### 10.7 Verification and appenders - [S104] https://www.zerobounce.net/email-validation-pricing/ ; [S105] https://emailable.com/pricing ; [S106] https://usebouncer.com/pricing/ ; [S107] https://emelia.io/hub/neverbounce-pricing ; [S108] https://pricingsaas.com/companies/kickbox ; [S109] https://www.mailgun.com/pricing/ ; [S110] https://www.twilio.com/en-us/sendgrid - [S111] https://atdata.com/atdata-history ; [S112] https://www.experianplc.com/newsroom/press-releases/2026/ (search result; title "Experian fortifies identity and fraud capabilities with acquisition of AtData") ; [S113] https://tomba.io/blog/atdata-pricing-reviews-pros-and-cons ; [S119] https://atdata.com/pricing - [S114] https://aws.amazon.com/marketplace/pp/prodview-lk7mzwm6jhl7k ; https://www.datarade.ai/data-products/contact-append-versium-reach-b2c-versium (search summaries) - [S115] https://www.melissa.com/pricing ; [S116] https://docs.melissa.com/personator/personator-consumer-web-service.html (404; nav only) - [S117] https://www.capterra.com/p/160486/pipl/reviews/158647/ (search summary) - [S118] https://dataro.io/blog/best-wealth-screening-tools-nonprofits ; https://windfalldata.com/solutions/wealth-screening ; https://g2.com/compare/donorsearch-vs-iwave (search summaries) #### 10.8 Unfetchable or unanswered (explicit) Pages that failed: PDL Services Subscription Agreement and Acceptable Data Use Policy (JavaScript shell); PDL support pricing article (403); PDL `/terms-of-service` and `/data-license-agreement` (404); FullContact platform pages (JavaScript shell, title only); RocketReach API docs (403); NeverBounce pricing (403); Kickbox (blocked by the tool); MillionVerifier pricing (404); HubSpot developer changelog for the Clearbit sunset (404); GLEIF access-and-use page (404); Melissa Personator reference (404); Wordpress/Gravatar-specific caching rule (not on page); Clay terms (404); SendGrid validation price (not on the page). Not researched or not answered after the search budget ended: Snov.io, Wiza, Seamless.ai (terms), Lead411, LeadIQ, UpLead (API), Persana, Common Room, UserGems and Champify API terms, Browse.ai, MillionVerifier, Google favicon terms, X avatar terms, Windfall and DonorSearch API terms, D&B and Crunchbase contract terms, Apollo and ZoomInfo OEM contracts, Exa, Parallel, Perplexity and Tavily terms of service on personal data, personal-email match rates for any vendor, non-US coverage for any vendor, certification and license data availability beyond PDL's `certifications` field. ### 11. Analysis (Inferred) Everything in this section is my reasoning from sections 2 to 10. Each judgement names the evidence it rests on. #### 11.A Who can enrich a professional known only by a personal email, name and maybe city or employer? **Realistic candidates** 1. **PDL**. It accepts an email "the person has used" and a name with company or location [V, S2]; it exposes `personal_emails`, `experience`, `education`, `certifications`, `skills`, `interests` [V, S5]; its email slice has `personal_emails` populated on 64.1 percent and `experience` on 45.3 percent [V, S8]. The catch is confidence: PDL's own guidance says an email-only match rarely exceeds likelihood 6 and name plus location rarely exceeds 4 [V, S2]. A system should send as many inputs as it has (email, name, employer, city, school) and gate display on likelihood, with staff review below a threshold. Pay-per-match billing means failed searches are free [V, S4]. 2. **FullContact (Enrich and Resolve)**. A hashed or clear email, phone, or name plus postal address resolves to a consumer-level person ID; Core Professional returns employer, title and LinkedIn URL; Social returns avatar and profile URLs [V, S22, S23, S24]. Evidence of US-only focus is Reported [S29]. No match rate or confidence field is documented [V, S22]. It is the clearest design fit for personal-email-keyed matching but also the highest data-broker exposure and a sales-led contract. 3. **AI research (Exa, Parallel Task, Anthropic or OpenAI search)** as a fallback that takes name, city and employer and returns cited public-web findings [V, S72, S75, S80]. Cheap (about $0.01 to $0.10 per member), no accuracy data, namesake risk. **Weak or B2B-work-email-only in practice** - **Cognism** (no personal emails, by its own statement [M, S47]), **ZoomInfo** (B2B database; no personal-email enrichment evidence [I]), **Apollo** (personal emails not revealed in GDPR regions; terms scoped to business contact information [V, S31, S33]), **Hunter** (needs a company domain; webmail gets fixed score 50 [V, S53]), **Lusha and RocketReach** (B2B reveal tools; RocketReach says personal email is included but direction of lookup unknown [V, S51]), **Dropcontact and Prospeo** (professional email construction [R]), **Clearbit/Breeze** (HubSpot-internal) [R]. - **LinkedIn-derived vendors (Crustdata, Coresignal, Nubela legacy)** need a LinkedIn URL as the key [V S59 for Crustdata watchers], and carry the supply-side risk in section 6. **Consumer-data appenders**: AtData (TowerData heritage), Versium, Melissa and FullContact Resolve map email to postal address, phone, demographics and sometimes social handles [R/V]. They answer "is this a real, current person and where do they live" and "has the address changed" (Melissa Move Update is priced at $12,300 per year per million records [V, S115]). They do not document employer, certifications or licenses. For an AMS this is useful for **address hygiene and duplicate detection**, not for affiliation enrichment. **Gap that no vendor fills**: certifications and licenses. Only PDL documents a `certifications` field (name, organization, dates) [V, S5]; no coverage figure exists [U]. For credential data the authoritative sources are the credentialing bodies and licensing boards themselves, which this research did not cover [U]. #### 11.B Resale, embedding and persistence rights Rhythm is a SaaS vendor; each association is a third party relative to Rhythm, and its members are data subjects. What the documents say about each question: | Provider | Can Rhythm display results to customer staff? | Persist indefinitely? | Re-supply to customers (OEM)? | Evidence | | ---------------------------------------------------- | ------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------- | ----------------------------------------------------------------------- | -------------- | | Apollo | Internal use only [V] | Keep what is already in your records, tied to the subscription term, ambiguous [V] | **No**: no sublicensing, no API access for third parties [V] | S33 | | PDL (self-serve) | Not stated; partner-page reproduction bars resale and sublicensing [R] | Not stated [U]; Data License "any way" within Acceptable Data Use Policy [V] | Needs a contract; claim of embedding rights is unverified marketing [M] | S12, S17, S19b | | FullContact | [U] | [U] | [U]; sales-led | S28, S25 | | ZoomInfo | Contract-defined [R] | Docs suggest caching IDs 12 months [R] | Partner-app route exists with PKCE [R]; terms [U] | S35 | | Brandfetch | Internal use for own business [V] | 30-day cache then must re-call [V] | **Needs sales agreement** (Custom plan "custom legal terms") [V] | S83, S84 | | Logo.dev | Yes (pricing says store anything returned, every plan) [V] | Yes per pricing page [V]; full ToS not read [U] | Not addressed on page [U] | S85 | | Gravatar | Yes, as the member's public avatar; link and privacy disclosure expected [V/R] | May cache; refresh at reasonable intervals; disclose [V] | Guidelines say no competing-product use [V] | S88 | | LinkedIn API | Only to the consenting member's own app experience [V] | No, except duration necessary [V] | **No** [V] | S69 | | Unavatar | CRM display is "fair use" [V] | | Bulk resolution of identifiers prohibited [V] | S92 | | GLEIF, Wikidata | Yes (CC0) [R/V] | Yes | Yes | S96, S98 | | OpenCorporates | Free tier requires open-licence release of the incorporating product [V] | | Paid accounts remove share-alike [V] | S94 | | Exa, Parallel, Tavily, Firecrawl, Perplexity, OpenAI | ToS not read [U] | | | | | Anthropic web search | Citations must be displayed to end users when output is shown [V] | | | S80 | | Clay | Staff tool; underlying vendor rights apply [I] | | | S44 | **Photos**: the displayable-photo question has no clean commercial answer. Gravatar is the least encumbered (member-published, public by the member's action [V, S89]). LinkedIn photos are Profile Data with storage limits [V, S69]. Unavatar forbids enumerating a user base [V, S92]. Reading across: do not store third-party photos by default; store a link plus hash and fetch for display [I]. **Per-customer contracts**: for providers whose terms bar third-party use (Apollo, Brandfetch below Custom, LinkedIn API), the compliant routes are (a) each association signs its own account and supplies its key (bring-your-own-key), (b) Rhythm negotiates an OEM or data-license agreement, or (c) the data is not sold by Rhythm at all. Option (a) keeps vendor terms between association and vendor; its feasibility depends on provider terms for API-key delegation, which none of the pages read address [U]. #### 11.C Shortlist for an AMS (starting position, not a decision) **Baseline (free or near-free)** - Gravatar avatars and profile by SHA-256 hash (free; 1,000 profile lookups per hour; avatar requests unmetered) [V, S86]. - Employer logos: Logo.dev (store-anything on every plan; free tier needs attribution for commercial use) as default; Brandfetch only if Rhythm signs a Custom agreement with sublicensing rights [V, S83, S84, S85]. - Registry layer: GLEIF (CC0), Wikidata (CC0), Companies House, EDGAR [V/R, S96, S98, S99, S100]; narrow coverage. - Email verification: any of Bouncer ($0.002 to $0.008), Emailable ($0.0065 to $0.0076), ZeroBounce ($0.0099 to $0.0195), Mailgun ($0.008 to $0.012) [V, S104 to S109]. Choose by contract terms on data retention (not researched [U]). **Premium person (compare for associations)** | Dimension | PDL | FullContact | Apollo | Clearbit / Breeze | | -------------------- | ------------------------------------------------- | ------------------------------------------ | -------------------------------------------------- | ------------------------------------------ | | Personal-email key | Yes, documented input [V] | Yes, designed for it (US) [V/R] | Weak; personal emails withheld in GDPR regions [V] | Not applicable outside HubSpot [R] | | Professional history | experience, education, certifications, skills [V] | employment bundle; education unknown [V/U] | employment history [V] | 100+ B2B points per HubSpot [V, S42] | | Confidence signal | likelihood 1 to 10 [V] | none documented [V] | match_confidence high to none [V] | n/a | | Billing basis | per match [V] | per match [V (marketing line)] | credits by data returned [V] | credit per enriched record [V] | | Published price | $98 per month for 350 credits entry [V] | none [U] | none fetched [U] | $0.07 to $0.30 per credit (2024) [V] | | Resale / OEM | contract [M/R/U] | contract [U] | prohibited [V] | not offered as an API to third parties [I] | | Non-US evidence | unpublished [U] | US-centred [R] | GDPR region carve-out [V] | [U] | My reading: PDL is the only one of the four that is simultaneously API-accessible, documents personal-email matching with a score, and publishes an entry price. Apollo's terms make it unsuitable for embedding; Breeze is HubSpot-bound; FullContact merits a quote conversation specifically on US member coverage and OEM rights. This is a hypothesis to test with a sample file, not a conclusion. **Organization**: Logo.dev (logos) plus PDL Company Enrichment or Apollo Organization Enrichment (Apollo is internal-use only) plus D&B Direct+ only if an association needs verified hierarchy (quote-only). Diffbot Knowledge Graph is a priced alternative at about $0.025 to $0.03 per entity [I from V, S103]. **Job change** - PDL webhooks: monthly batches, enterprise only [V, S10]. Cheapest by structure but slow. - Crustdata watchers: priced per changed person (5 credits at 30-day freshness), keyed on LinkedIn URLs; sourcing risk [V, S59; I]. - UserGems and Champify: sales-team tools at $24K to $100K+ per year [R, S71]; built for CRM-sales champion tracking, not membership. - Apollo: no job-change monitor documented in pages read [U]. - Inferred: for associations, job-change monitoring is useful for membership retention and chapter-transfer prompts, but the evidence supports only PDL (monthly, enterprise) and Crustdata (LinkedIn-based) as API products; none are priced for associations. **AI research**: Exa Search plus Contents ($0.007 plus $0.001 per page), or Parallel Task Lite/Base ($0.005 to $0.01 per request), feeding an LLM; or the Anthropic or OpenAI web-search tool ($0.01 per search plus tokens). Note the Anthropic tool is unavailable on Bedrock [V, S80]. Treat outputs as suggestions needing confirmation and citation. #### 11.D Unit economics (per record or per unit) Tags: **Published** = number on a vendor page I read (URL key); **Derived** = arithmetic on published numbers; **Reported** = from a secondary source; **Assumed** = my planning placeholder with no evidence. | Item | Per-unit cost | Basis | Tag | Source | | ----------------------------------------- | ------------------------------------------------------------------------------------------------ | ------------------------------------------------ | ------------------------------------------------------- | ------------ | | PDL person enrichment, entry tier | $0.28 per matched record | $98 per 350 credits; 404 not charged | Derived from Published | S6, S4 | | PDL person enrichment, higher tiers | $0.265 and $0.25 | third-party reproduction of PDL self-serve table | Reported | S18b | | PDL company enrichment | $0.10 per credit | same | Reported | S18b | | PDL Identify | 1 credit per call (up to 20 candidates) | docs | Published (credit count) | S7 | | PDL enterprise/data license | custom | | Published ("custom") | S6 | | Apollo people enrichment | 1 credit for email or demographics, +8 for mobile | docs | Published (credits); dollar value of a credit not found | S32 | | Apollo dollars per credit | not obtained | | Assumed placeholder only | | | FullContact Enrich | match-based, no price | | Unavailable | S28 | | ZoomInfo contact or company enrich | $0.38 per record at the $1,000 tier; lower at volume | third-party guide of ZoomInfo's self-serve page | Reported | S35 | | Clearbit / Breeze enrichment | $0.30, $0.15, $0.07 per record by tier; $5,000 one-time for up to 1M records (about $0.005 each) | HubSpot-hosted guide, Sept 2024 | Published (2024) / Derived | S41 | | Lusha email reveal | 1 credit = $0.094 to $0.125 at Starter | $37.45 to $49.90 per 400 credits | Derived from Published | S48 | | Lusha phone reveal | 5 credits | | Published | S48 | | Hunter email find or combined | about $0.0245 per credit at Starter; $0.0120 at Scale list | $49 per 2,000; $299 per 25,000 | Derived from Published | S54 | | Hunter verification | 0.5 credit = about $0.012 (Starter) to $0.006 (Scale) | | Derived from Published | S54 | | Clay data credit | from $0.05 each | | Published | S44 | | Crustdata watcher | 5 credits per changed person (30-day), up to 150 (daily); dollar per credit not found | docs | Published (credits only) | S59 | | Diffbot KG enhance | 25 credits = $0.025 to $0.03 (Startup $299 for 250,000 credits; overage $0.001) | | Derived from Published | S103 | | Companies API | $95 per 50,000 credits = $0.0019 per credit; credits per domain not stated | | Derived (credit); per-domain Unavailable | S68 | | Brandfetch brand fetch | $0.0099 to $0.052 per fetch inside Growth allowance; $0.10 overage; Logo API separate quota | $99 or $129 per 2,500 to 10,000 | Derived from Published | S83 | | Logo.dev logo | $0.02 per API logo; CDN logos free within plan quotas (500K free) | | Published | S85 | | Gravatar | $0 | | Published | S86 | | GLEIF, Wikidata | $0 | | Reported / Published | S98, S96 | | Email verification | $0.002 to $0.0195 per check | see section 5 | Published (ZeroBounce, Emailable, Bouncer, Mailgun) | S104 to S109 | | Melissa Personator Consumer + Move Update | $0.0123 per record at 1M per year | $12,300 per 1,000,000 | Derived from Published | S115 | | Melissa Personator Enrich | $0.0143 per record at 1M per year | $14,300 per 1,000,000 | Derived from Published | S115 | | Versium match credit | $0.04 | AWS Marketplace listing | Reported | S114 | | AtData append | $0.12 to $0.20 per match | competitor blog | Reported (weak) | S113 | | Exa search + contents | $0.007 per search + $0.001 per page | | Published | S72 | | Exa Agent contact enrichment | $0.02 email, $0.07 phone | | Published | S72 | | Parallel Task | $0.005 (Lite) to $0.10 (Pro) per request | | Published | S75 | | Anthropic / OpenAI web search | $0.01 per search plus tokens | | Published | S80, S81 | | Perplexity Search API | $0.005 per request ($0.001 fast) | | Published | S76 | | Champify / UserGems | $24K to $72K per year (Champify); $15K to $100K+ per year (UserGems est.) | | Reported | S71 | **Worked scenario (all inputs labelled)**: an association with 50,000 member contacts, enriched once, then 10 percent refreshed each year. - Assumed: 40 percent of records reach a usable match (no vendor-published figure; PDL guidance suggests personal-email-only inputs yield low scores [V, S2]). At PDL's $0.28 per matched record: 50,000 x 0.40 x $0.28 = $5,600 initial, about $560 per year to refresh 10 percent. At $0.25: $5,000. If the real match rate is 20 percent the initial cost halves, which is why pay-per-match matters. [I, arithmetic on Published price and Assumed rate] - Email verification of 50,000 at $0.002 to $0.0076: $100 to $380 [Derived from Published]. - Logos for 5,000 distinct employers at Logo.dev $0.02: $100 [Assumed count; Published unit]. Within a Startup plan's included API credits ($15 per month) plus CDN requests, much of this can be free. - Address hygiene via Melissa on 50,000 records at about $0.0123 to $0.0143: $615 to $715 per year at the 1M-record tier's unit price; the 1M-record annual minimum is $12,300, so a small file would use PAYG bundles ($350 per 100,000 credits, 3 credits per US address) [Derived/Published; minimum structure matters]. ### 12. Risks and open questions to resolve before architecture is fixed 1. **Contract shape before vendor choice.** Of the providers read, none published self-serve terms that clearly allow an AMS vendor to resell results to association customers. Obtain written OEM or data-license terms from PDL, FullContact and Logo.dev (and, if used, Brandfetch Custom) before building the product around any one. 2. **Personal-email match rate is unmeasured.** Ask each shortlisted vendor to run the same 1,000-record sample of anonymized or consented association members (personal email, name, city, employer) and report match rate, confidence distribution and wrong-person rate. This is the single most decisive missing datum. 3. **Opt-out propagation.** PDL's changelog carries opt-outs [V, S13]; Hunter returns 451 for claimed emails [V, S53]; Kaspr was ordered to delete [V, S62]. A cache in Rhythm needs a mechanism to honor vendor-side deletions and member-side objections. 4. **Supplier-side LinkedIn exposure.** Ask any vendor in the shortlist whether LinkedIn-derived fields are included and under what rights; Proxycurl's shutdown shows what happens to customers when a supplier loses [V, S63]. 5. **Certification and license data** has no documented provider other than PDL's thin field; direct integrations with credentialing bodies are outside this report [U]. 6. **Stale or conflicting prices** to re-check: Clay (page contradicts itself on Launch and Growth), Breeze (2024 PDF vs 2026 blogs), Apollo (no dollars fetched), RocketReach, Datagma, Dropcontact, Perplexity Sonar (old tables vs docs). 7. **Bedrock constraint.** If Rhythm's AI path is Bedrock, Anthropic's web search tool is not available there [V, S80]; a separate search API (Exa, Parallel, Tavily) would carry the research function. ## 26. W4 Evidence Report: Privacy Law and Data-Licensing Constraints on Rhythm 2.0 Data Enrichment ## W4 Evidence Report: Privacy Law and Data-Licensing Constraints on Rhythm 2.0 Data Enrichment Research date: 2026-10-22 (all "accessed" dates below are 2026-10-22). Prepared for: the "Rhythm 2.0 Data Enrichment" architecture proposal. Author: automated research pass (Claude). This is research, NOT legal advice. Nothing here has been reviewed by counsel. Every legal conclusion marked Inferred needs a lawyer's confirmation before it drives a contract, a product claim or a launch decision. --- ### 0. How to read this report #### 0.1 The proposal being tested Rhythm is a multi-tenant SaaS association-management system. Its tenants are associations (mostly nonprofits and trade or professional bodies). The people in the tenants' records are individual members, many of whom use personal email addresses. The proposal is that Rhythm calls third-party enrichment providers on the tenants' behalf (identity, professional profile, company data, profile photo, company logo, possibly an AI agent that browses the web) and stores the results on the member record. #### 0.2 Evidence labels - **Verified (primary)**: I read the statute, regulator document, court/agency release, or the provider's own terms/policy page during this session. Where I downloaded a PDF and read the raw text, the source list marks it `[raw]`. For all other "Verified" items the page was read through a fetch tool that summarises the page; section numbers and quotations are as the tool reported them and should be re-checked against the live page before anyone relies on them in a contract. - **Reported**: taken from a secondary source (law-firm note, trade press, vendor blog, third-party reproduction of a provider's terms, or a search-result summary), or from a primary page where two sources disagreed. Treat as a lead, not a finding. - **Inferred**: my reasoning or general legal knowledge, not confirmed by a source read in this session. This is where lawyers should look first. #### 0.3 Known limits of this pass (read before relying) 1. The web-search tool hit its per-turn budget part-way through. Searches done before that point returned useful summaries; topics I had not yet searched are flagged in Section 16 (for example CASL text, Meta v. Bright Data, the append vendors' own terms, People Data Labs' own license agreement). 2. Several primary pages could not be fetched (CNIL, Illinois General Assembly, California leginfo, PDL's own legal pages, Cognism, FTC press pages). Section 16 lists them. Where I fell back to a secondary source I say so. 3. Enforcement in this area is moving and is contested. Two of the most relevant 2024-2026 European decisions (Garante v. Lusha, Garante v. OpenAI) are shown on the Garante's own site as temporarily removed after Court of Rome rulings. Do not treat any single fine as settled law (Section 1.6). 4. Provider terms change often. ZoomInfo's 2021 and 2025 license terms differ on post-termination retention (Section 5.4). Every provider row below must be re-read at contract time and the version pinned. 5. The corrupt or irrelevant search hit "ppllabs.com Terms of Service" is a different company (PPL Labs), not People Data Labs. I excluded it. Nothing in this report about PDL comes from it. --- ### 1. Executive summary (the findings that most change the architecture) 1. **The member's identifier is itself disclosed to the provider.** Calling a provider with an email address is a disclosure of personal data by the tenant (controller) to a recipient. Several providers say outright that data you submit may be used to improve their own datasets (HubSpot Product Specific Terms 6.5; ZoomInfo 5.5/5.6; FullContact 4(a); Verified). So enrichment is not a one-way read: the tenant's member list leaks into a third party's graph. This must be disclosed to members as a recipient (GDPR Art. 13(1)(e) / 14(1)(e)) and it changes the lawful-basis analysis. 2. **Providers are independent controllers, Rhythm is the tenant's processor.** Hunter's DPA, Apollo's privacy policy and HubSpot's terms all say the provider is an independent controller for its own database. EDPB Guidelines 07/2020 para. 81 and GDPR Art. 28(10) say a processor that uses data for its own purposes becomes a controller. Rhythm must therefore never use tenant-initiated lookups or results for its own purposes (Section 11). 3. **A cross-tenant cache keyed by email is very likely unlawful and almost certainly a license breach.** It fails purpose limitation / processor-instruction rules (GDPR), mirrors the exact example the California regulations forbid (11 CCR 7050(a): a provider "cannot use one business's customer list to market for another business"), and breaches the no-resale / no-use-for-the-benefit-of-others clauses of ZoomInfo, Apollo, FullContact, Brandfetch, D&B and (as reported) PDL (Section 11). It would also turn Rhythm into a de facto data broker (Section 3.3). 4. **Article 14 transparency cannot be waved away.** The EDPB's July 2026 draft guidelines (para. 27) say the "disproportionate effort" exception "should not be routinely relied upon", and their Example 3 says that where contact details exist the controller must inform people individually. A tenant that already holds the member's email has no disproportionate-effort argument (Inferred from the example by analogy). The ICO says expressly that customers are unlikely to expect enrichment and it should not be "buried in your privacy information". 5. **Personal-email enrichment is the high-risk tier; business-email/company enrichment is the lower-risk tier.** Reasonable expectations (EDPB LI Guidelines 1/2024 paras. 52-54) and the narrow readings of "publicly available" (OPC, Canada) both cut against pulling consumer-identity data (photos, social profiles, home address) onto an association member record. Default that tier off. 6. **For some associations, membership itself is special-category data.** GDPR Art. 9(1) lists religious beliefs, political opinions and trade-union membership. Art. 9(2)(d) lets such associations process their members' data but forbids disclosing it outside the body without consent. Sending those members' identifiers to an external provider under the tenant's account reveals that the person is connected to that body (Inferred). Tenant-type gating is needed. 7. **Provider licenses are mostly "internal business use by the named licensee".** ZoomInfo (2.3), Apollo (3(d)(ii)-(iii)), Brandfetch (4(v), 2.6), D&B (3.2), Crunchbase (Enterprise), Hunter (one account, no sharing) all bar use for the benefit of others or "on behalf of another entity". FullContact is the exception: its Exhibit A expressly contemplates a Customer Application with Third Party End Users under flow-down written terms. Either each tenant brings its own key (BYOK, each tenant contracting directly) or Rhythm holds a negotiated OEM/reseller agreement. A single Rhythm-held self-serve key serving many tenants is the pattern most likely to breach. 8. **AI use of enrichment data is restricted by several providers.** Apollo 3(e)(v) bars training AI on contact data (except the customer's own internal, non-commercial models); FullContact 2(d)(1)(vi) bars training/fine-tuning but permits AI analysis for internal business purposes; Brandfetch 2.6 bars genAI training without a written agreement. Rhythm's export and AI features need a per-field "license class". 9. **Scraping-sourced data carries downstream baggage.** hiQ was ordered to delete LinkedIn-derived data including copies held by third parties; Proxycurl's injunction required it to send the order to its customers; CNIL fined KASPR (LinkedIn contact scraping) in December 2024; Garante fined Lusha EUR 2m in July 2026 (now suspended pending a Court of Rome opposition). Vet provider provenance and contract for deletion-notice flow-down. 10. **Photos: link or short-TTL proxy, never a durable copy shared across tenants.** Gravatar's docs permit saving profile data in your own database and its responsible-use guidelines ask you to refresh at reasonable intervals; Brandfetch caps cached logos at 30 days; LinkedIn forbids copying. A profile photo is personal data (and copyrighted by whoever took it) but not special-category data unless processed with face-geometry matching (GDPR Recital 51; BIPA turns on face geometry; EU AI Act Art. 5(1)(e) bans untargeted facial-image scraping). 11. **Opt-out and erasure must be modelled as a suppression record that outlives the payload.** CPPA's Delete Act rules (DROP, processing from 1 Aug 2026, every 45 days, including inferences) and the ICO's B2B guidance both expect a suppression list. Erasure of enrichment payloads must propagate to every cache; the minimal suppression identifier is the lawful exception to "delete everything". 12. **US state-law exposure is real but uneven.** CCPA does not cover most nonprofits (for-profit "business" definition), but Colorado, Delaware, New Jersey and Oregon laws have no general nonprofit exemption, Minnesota and Maryland exempt only narrow insurance-fraud nonprofits, and Virginia, Texas and Tennessee exempt nonprofits broadly. A multi-tenant platform should build to the strictest state it serves. Rhythm itself, a for-profit, is exposed on any data it holds for its own purposes. --- ### 2. Question 1: GDPR and UK GDPR #### 2.1 Roles (who is the controller of what) | Processing | Likely role | Basis | | ------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Association's member record, including the decision to enrich | Association = controller | GDPR Art. 4(7); EDPB Guidelines 07/2020 (Inferred application) | | Rhythm executing the lookup, storing results for the tenant | Rhythm = processor (Art. 28) if it acts only on documented tenant instructions | Art. 28(3)(a) Verified; EDPB 07/2020 para. 81 Verified [raw] | | Provider's own database and its use of submitted identifiers | Provider = independent controller | Hunter DPA 10.1-10.2 (Verified, via fetch); Apollo Privacy Policy (controller for Contributor Database; processor only when handling data "on a Customer's behalf") (Verified); HubSpot PST 6.5(iii) "each party is a Controller" (Verified); ZoomInfo LTC incorporates a Controller-to-Controller DPA (Verified) | | Rhythm using lookups or results for its own purposes (shared cache, product analytics, model training) | Rhythm = controller for that processing | Art. 28(10) Verified: a processor that determines purposes and means "shall be considered to be a controller in respect of that processing"; EDPB 07/2020 para. 81 Verified [raw] | EDPB 07/2020 para. 81 (raw text): acting "on behalf of" means "the processor may not carry out processing for its own purpose(s)", and a processor "going beyond the controller's instructions" is "considered a controller in respect of that processing and may be subject to sanctions". Where an enrichment provider receives the tenant's identifier and returns data, the EDPB's third-party analysis (Guidelines 07/2020 para. 89) is that the recipient "will then be considered a controller in its own right for the processing that it carries out for its own purposes". Verified [raw]. Hunter is an instructive hybrid (Verified, via fetch of Hunter's DPA): Hunter is "generally a processor" except for Profile Data (data it collects from public web pages), Customer Account Data and Customer Usage Data, where it is an independent controller. So even within one vendor the role depends on the data category. Rhythm's records of processing and sub-processor annex must classify each provider by data flow, not by vendor. #### 2.2 Lawful basis: legitimate interests versus consent - **Legitimate interests (Art. 6(1)(f)) is the usual basis, with a three-part test.** The EDPB's Guidelines 1/2024 (adopted 8 Oct 2024, "version for public consultation" in the PDF I read; I did not confirm whether a final version has since been adopted) require (1) a legitimate interest, (2) necessity, and (3) balancing. Verified [raw]. - **Reasonable expectations are decisive and information duties do not cure them.** LI Guidelines para. 53 [raw]: "the mere fulfilment of the information obligations set out in Articles 12, 13 and 14 GDPR is not sufficient in itself to consider that the data subjects can reasonably expect a given processing." Para. 52: what is "common practice in certain sectors" is not the same as a reasonable expectation. Para. 54 lists relationship proximity, nature of the service, and the person's professional position as factors. - **Direct marketing is not an automatic legitimate interest.** LI Guidelines paras. 110-112 [raw]: Recital 47 "does not mean that direct marketing always constitutes a legitimate interest". Para. 114: unsolicited email/SMS direct marketing needs prior consent under the ePrivacy Directive, so Art. 6(1)(f) cannot be the basis for that sending. Para. 116 describes the soft-opt-in exception for a controller's own customers. Verified [raw]. - **Enrichment for membership administration versus for marketing are different purposes.** Enriching a member record so the association can run its programmes and keep contact data current is a different purpose from using appended data to market to the person. If enriched fields (especially phone, personal email) flow into marketing lists, the marketing rules apply (Sections 3.5, 7). Inferred. - **UK position.** The Data (Use and Access) Act 2025 s.70 (Verified, legislation.gov.uk enacted text) adds "recognised legitimate interests" (new Art. 6(1)(ea)) and inserts examples under Art. 6(1)(f) including "direct marketing" and intra-group transmission. The ICO updated its legitimate-interests guidance on 23 March 2026 to reflect this (Verified, update notice on the ICO page; I could not read the sub-pages). Importantly, the right to object (Art. 21) is extended to recognised legitimate interests. - **Consent.** The EDPB's July 2026 scraping guidelines (para. 44) say consent will "most probably" not be workable for indirect, at-scale collection because there is no direct relationship; but the association DOES have a direct relationship with its members, so consent is operationally feasible for the high-risk tier (personal-email identity data, photos). Inferred. - **Who needs the basis.** The tenant (controller) must have it and document it (a legitimate interests assessment, LIA). Rhythm's DPA should require the tenant to attest it exists before enrichment is switched on (Section 12). Recommendation (Inferred): treat enrichment as two risk tiers, not one feature. Business-context data on a business email address (employer, title, company firmographics) can be run on legitimate interests with notice and an easy objection route. Personal-email-keyed identity data (social profiles, photos, personal phone, home address) should be opt-in or off by default. #### 2.3 Article 14 transparency for data not obtained from the data subject Verified (Art. 14 text, gdpr-info.eu): - 14(1)-(2) list what must be given, including purposes and legal basis, categories of data, **recipients**, transfers, retention period, the legitimate interests pursued, rights (including objection), **source of the data and whether it came from publicly accessible sources**, and meaningful information about automated decision-making/profiling logic. - 14(3) timing: within a reasonable period and at the latest **one month** after obtaining the data; if used to communicate with the person, at the latest at first communication; if disclosed to another recipient, at the latest at first disclosure. - 14(5) exceptions: the person already has the information; impossible/disproportionate effort (with appropriate measures including public availability of the information); law requires the obtaining/disclosure; professional secrecy. How regulators apply it to enrichment and scraped/brokered data: - **EDPB draft Guidelines 03/2026 (adopted 7 July 2026; consultation 8 July to 30 Oct 2026)** [raw]. Para. 27: the Art. 14(5)(b) exception "should not be routinely relied upon" outside archiving/research/statistics. Para. 28: weigh effort against impact on the person, assessed on the dataset as a whole, considering number of data subjects, age of the data and safeguards. Para. 30: if relying on it, publicly available information is always required and must include categories, purposes, legal basis and "a precise indication of the source(s)". Example 3 (para. 34): where "there are means to contact the data subjects" and the data is recent, "it would not involve disproportionate effort" and "the controller should inform the data subjects individually". These guidelines are about scraping for generative-AI training, not enrichment, so apply them by analogy only (Section 9 covers their scope limits). - **EDPB Opinion 28/2024** (adopted 17 Dec 2024) [raw], transparency paragraph: where data is gathered by scraping, "reliance on the exception provided under Article 14(5)(b) GDPR is strictly limited to when the requirements of this provision are fully met". - **ICO guidance for organisations using data brokers** (Verified, ICO page; under review because of the 2025 Act): if you plan to use broker data "to enrich or profile your customers, tell them before you do it"; customers are unlikely to expect it, so it should not be "buried in your privacy information"; privacy information must be given "within one month of obtaining their data"; you remain responsible for your own compliance and must do due diligence on who compiled the data, what privacy information was used, when and how it was collected, and how objections are passed on. - **Experian (UK tribunals)**: the First-tier Tribunal (Feb 2023) found an Art. 14 breach for a residual cohort of about 5.3m people, rejected "disproportionate effort" because "significant business expense alone" does not qualify, but accepted layered notices via a portal as adequate for others; the Upper Tribunal dismissed all five ICO grounds on 23 April 2024 (Reported: Reed Smith 24 Feb 2023; Stephenson Harwood 28 May 2024; the tribunal decisions themselves not read). Takeaway Reported: legitimate interests can support marketing/profiling even where "surprising", but the Art. 14 duty is real and the effort exception is narrow. - **CNIL v. KASPR, 5 Dec 2024** (Reported via EDPB news item; the CNIL page returned 503): breaches of Arts. 6, 5(1)(e), 12, 14 and 15; stop collecting contact details people had limited the visibility of; delete them or, where indistinguishable, inform people within three months with a means to object; stop automatic renewal of storage; answer access requests with all information on data sources. The EDPB item states EUR 200,000; the CNIL headline reported by the search tool states EUR 240,000. The discrepancy is unresolved (Section 16). Design implications (Inferred): (a) Rhythm should generate an Art. 14 notice pack for tenants (what categories, which provider categories, source, retention, objection route); (b) record per member the date enrichment was first run and the date the notice was given; (c) block first-run enrichment for EU/UK-resident members unless the tenant attests that notice was given at or before the one-month deadline or in the membership notice (Art. 14(5)(a) "already has the information"). #### 2.4 Data minimisation, DPIA triggers - Minimisation (Art. 5(1)(c)): the EDPB scraping guidelines paras. 35-38 [raw] list pre-collection criteria, filters for sensitive categories, exclusion of sources that oppose scraping, and syntax filters. By analogy: request only the fields the tenant's purpose needs; do not ask providers for "everything"; drop returned fields not on the tenant's allowlist before storage. Verified for the principle; application Inferred. - DPIA (Art. 35): 35(3)(a) covers "systematic and extensive evaluation of personal aspects ... based on automated processing, including profiling" with legal or similarly significant effects; 35(3)(b) large-scale special categories; the list is "in particular" (Verified, Art. 35 text). Bulk enrichment combines datasets, builds profiles at scale and (for religious/political/trade-union associations) may touch special categories, so a DPIA is the safe default for the feature; regulators' national lists (Art. 35(4)) should be checked per country. The WP29 DPIA criteria (matching/combining datasets, scoring, large scale) are cited in the EDPB scraping guidelines fn. 1 but I did not read WP248 itself. Inferred: DPIA required for the Rhythm-level feature and a tenant-level DPIA template for tenants. #### 2.5 Objection, erasure, rectification - Art. 21(1): objection to legitimate-interest processing, controller must stop unless "compelling legitimate grounds". 21(2)-(3): objection to direct marketing is unconditional once exercised. 21(4): must be brought "explicitly" to the person's attention at the latest at first communication, presented clearly and separately. Verified. - Art. 17(1)(c)-(d): erase on successful objection or unlawful processing. 17(2): reasonable steps to tell other controllers if the data was made public. 17(3): exceptions include legal obligation and establishing/defending legal claims. Verified. - Art. 19: the controller must notify "each recipient to whom the personal data have been disclosed" of rectification, erasure or restriction unless impossible or disproportionate, and tell the person who the recipients are if asked. Verified. Application: the provider that received the member's email is a recipient; the tenant must be able to tell it (suppression request) and Rhythm's connector must carry that request. - Rhythm as processor must help with these (Art. 28(3)(e)) and delete or return data at the end of service (Art. 28(3)(g)). Verified (28(3)(a), (g)); (e) not individually read. #### 2.6 Regulators and courts, 2024-2026: what has been decided (and what is contested) | Date | Body | What | Status/evidence | | --------------------------------------- | ------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 5 Dec 2024 (published 9 Jan 2025) | CNIL v. KASPR | LinkedIn contact scraping; Arts. 6, 5(1)(e), 12, 14, 15; EUR 200k (EDPB) or 240k (CNIL headline per search tool) | Reported; EDPB item fetched, CNIL page unavailable | | May 2025 | CNIL v. Solocal Marketing Services (EUR 900k) and Caloga (EUR 80k) | data brokers; prospecting without valid basis, data passed to partners | Reported via search summary of CNIL tag page; not fetched | | 2024 | CNIL v. Hubside.store (EUR 525k, 4 Apr 2024) and Foriou (EUR 310k) | use of broker-supplied data for prospecting without ensuring valid consent | Reported via search summary; not fetched | | 14 Jul 2026 (announced 27 Jul 2026) | Garante (Italy) v. Lusha Systems Inc. | EUR 2m; enrichment/resale of contact data; legitimate interest not demonstrated; deficient Arts. 12-14 notices; Art. 3(2) applied because continuous profile updates amounted to monitoring; deletion and processing ban for people in Italy | Reported (Captain Compliance; Luxgap; search summary of Garante press release). The Garante's own page for the decision (Provvedimento n. 542 of 14 July 2026, doc-web 10275035) says it "è stato temporaneamente rimosso" after an interim Court of Rome order of 8 Sept 2026; opposition pending. Verified (the notice), Reported (the substance) | | 20 Dec 2024 / Court of Rome 18 Mar 2026 | Garante v. OpenAI (EUR 15m) | training without adequate legal basis; transparency | Verified (Garante page): the decision was temporarily removed after Court of Rome judgment No. 4153/2026, published 18 March 2026, upheld OpenAI's appeal | | 18 Dec 2024 (adopted 17 Dec) | EDPB Opinion 28/2024 | AI models: LI test, scraping, deployer duty to assess upstream lawfulness | Verified [raw] | | 8 Oct 2024 | EDPB Guidelines 1/2024 | legitimate interest | Verified [raw] (consultation version) | | 7 Jul 2026 | EDPB draft Guidelines 03/2026 | scraping for generative-AI training; open to 30 Oct 2026 | Verified [raw] | | Oct 2026 | EDPB news list headlines | Norwegian DPA v. Xandr EUR 8.6m; Italian DPA v. BBVA EUR 5.5m (objection to direct marketing not respected); Dutch DPA v. Uber EUR 824.99m (unlawful automated decision-making, insufficient profiling information) | Reported: headlines only, from the EDPB news listing; underlying decisions not read, figures should be treated as unverified | | 3 Sep 2025 | CJEU General Court, Latombe v. Commission T-553/23 | dismissed the challenge to the EU-US Data Privacy Framework adequacy decision | Verified [raw] (CJEU press release 106/25); appeal status not checked | Pattern across these (Inferred): regulators are not banning enrichment or even B2B contact brokering outright; they fine (a) brokers that cannot show a documented balancing test, (b) deficient Art. 14 notices, (c) retention without limits, (d) failure to honour access/objection, and (e) the buyer's duty to check upstream lawfulness. A Rhythm design that bakes in (a)-(e) tenant-side evidence is aligned with the direction of travel. #### 2.7 Transfers (the "Schrems" angle) and data residency - Verified [raw]: the General Court dismissed the annulment action against the EU-US Data Privacy Framework (DPF) on 3 Sept 2025 and confirmed the US ensured an adequate level of protection on the date of the decision, while noting the Commission must keep monitoring. Schrems I and II invalidated the two earlier frameworks (as recited in the press release). - The DPF only covers US organisations that self-certify (Inferred, general knowledge; not read this session). So the transfer tool for each provider must be checked individually: DPF certification, or SCCs plus a transfer impact assessment. Hunter's DPA (Verified via fetch) uses EU SCCs, the UK Addendum and Swiss modifications, stores in a Google Cloud datacentre in Belgium, and gives 14 days' notice of new sub-processors with a 7-day objection window. - Two transfer legs exist: (1) tenant to Rhythm (Rhythm's AWS region), (2) Rhythm or tenant to provider. Art. 13(1)(f)/14(1)(f) notices must cover (2) (Verified, Art. 13 text and Art. 14 text). - Quebec requires a privacy impact assessment before communicating personal information outside Quebec, including to other provinces (Reported, law-firm commentary; Section 4). - Data residency design (Inferred): the "enrichment run" record should store the region the call was made from, the provider endpoint region where known, and the transfer mechanism relied on, so a tenant can produce the Art. 30 and transfer documentation. Offer EU-only provider routing where providers allow it. #### 2.8 UK GDPR Same structure (Art. 14 UK GDPR; the Art. 14 exceptions were the subject of the Experian litigation). Differences relevant here: (a) the DUAA 2025 changes to Art. 6 (above); (b) PECR distinguishes corporate subscribers (no consent needed for B2B email, but identity and a valid opt-out address required, and the ICO says honour corporate opt-outs) from sole traders and some partnerships (treated as individuals: consent or soft opt-in) (Verified, ICO B2B page, which is under review because of the 2025 Act); (c) ICO: "Being publicly listed doesn't mean someone agrees to marketing"; named business email addresses are personal data; keep objectors on a suppression list rather than deleting (Verified); (d) UK-US transfers need their own mechanism (UK Addendum or the UK-US data bridge; not researched). --- ### 3. Question 2: United States #### 3.1 CCPA/CPRA: who it covers - "Business" requires an entity "organized or operated for the profit or financial benefit of its shareholders or other owners" doing business in California and meeting a threshold (over USD 25m prior-year gross revenue; buys/sells/shares data of 100,000+ consumers or households; or 50%+ of revenue from selling/sharing PI), plus entities that control or are controlled by a business and share common branding, and certain joint ventures (Cal. Civ. Code 1798.140(d), Verified via california.public.law). - Result: most association tenants (501(c)(3)/(c)(6) nonprofits) are not "businesses" under CCPA unless commonly branded with or controlled by a for-profit, or running for-profit activities (commentary: Proskauer FAQ, BCLP, IAPP; Reported). The Attorney General declined to give a general answer for credit unions and other not-for-profits, saying it is fact-specific (Reported). - Rhythm itself is a for-profit. CCPA applies to Rhythm as to any data Rhythm holds for its own purposes if it meets a threshold (Inferred), and Rhythm's service-provider status for tenants' data depends on its contract terms (below). #### 3.2 Sale/sharing and service-provider terms when a member email goes to a provider - Sell = disclosing personal information to a third party "for monetary or other valuable consideration" (1798.140(ad)); share = disclosure for cross-context behavioural advertising (ah); a "third party" is anyone other than the business, its service provider or its contractor (ai) (Verified via california.public.law). - 11 CCR 7051(a) (Verified via Cornell LII): a service-provider contract must prohibit selling/sharing, specify the business purpose "not ... in generic terms", forbid use outside the direct business relationship, forbid combining with data from other sources or the provider's own consumer interactions (with limited exceptions), require CCPA-level protection and help with consumer requests, give the business audit rights, and require notice if the provider can no longer comply; subcontractors need equivalent contracts (7051(b)). - 11 CCR 7050(e) (Verified): a person without a compliant 7051(a) contract is not a service provider/contractor, so disclosure to them may be a sale or sharing, which triggers opt-out rights. - 11 CCR 7050(a) (Verified): a service provider may use data to build or improve its own services only internally and "may not use the data to perform services for another person"; the regulation's example is that an email marketer "cannot use one business's customer list to market for another business". - Application (Inferred): enrichment providers almost always combine your query with their own data and keep what you send, so they will not meet 7051(a); a tenant that is a CCPA "business" and sends member emails to such a provider may be making a "sale" and owes notice at collection and opt-out. If the tenant is a nonprofit outside CCPA, this does not bind the tenant but is an argument for Rhythm to contract in the 7051(a) shape anyway for those few tenants that are in scope, and for Rhythm's own CCPA exposure. #### 3.3 Data broker registration and the Delete Act - **California.** "Data broker" = a business that knowingly collects and sells to third parties the personal information of a consumer with whom it has no direct relationship (Civ. Code 1798.99.80(c), Verified via california.public.law); exclusions include entities covered by FCRA, GLBA, the Insurance Information and Privacy Protection Act, and certain HIPAA-related entities. Registration is annual (1-31 January) with a USD 6,000 annual fee (Verified, CPPA data-broker page). - **DROP.** Consumers could submit a single deletion request from 1 Jan 2026; brokers must begin processing on **1 Aug 2026**, access DROP at least every 45 days, delete matching personal data **including inferences**, report status within 45 days, and keep a suppression list of requests so data stays deleted; independent audits from 1 Jan 2028 and every three years (Verified, CPPA announcement 13 Nov 2025 and CPPA page). A search summary reported first-week processing statistics from a vendor blog; I disregarded them. - **Vermont** (9 V.S.A. 2430, Verified, legislature.vermont.gov): data broker = knowingly collects and sells or licenses brokered personal information of a consumer with whom it has no direct relationship; direct-relationship examples include past or present customer, employee, investor or donor; the current text excludes "developing or maintaining third-party e-commerce or application platforms" and publicly available business/professional information. **A new version takes effect 1 Jan 2027**: brokered personal information becomes any information, "including derived data and unique identifiers", linked or reasonably linkable to a person, excluding publicly available information; "direct relationship" is defined as the consumer having "intentionally interacted with a business" for the purpose of accessing or obtaining its products or services; the application-platform exclusion is not in the text I read. This matters for Rhythm. - **Texas** (Bus. & Com. Code ch. 509): the pre-2025 definition turned on "principal source of revenue" (Verified, texas.public.law page, which appears stale). Reported (WilmerHale, 4 Sep 2025): SB 2121 and SB 1343 took effect 1 Sept 2025; "data broker" now covers an entity that collects, processes or transfers personal data it did not collect directly from the individual; service providers, government entities, nonprofits, consumer reporting agencies and financial institutions are excluded; registration applies if the entity earns over half its revenue from indirectly collected data or earns revenue from indirectly collected data of more than 50,000 individuals; penalty USD 100 per day capped at USD 10,000 per 12 months. - **Oregon** (HB 2052, effective 1 Jan 2024): data broker = business that collects and sells or licenses brokered personal data about Oregon residents; exclusions include direct relationships (current or former customers, employees, agents, investors, donors), federally regulated entities, publicly available business/professional information, and data used to develop or maintain e-commerce services or software; registration with DCBS; penalty up to USD 500 per violation, USD 10,000 per year cap (Reported, ZwillGen note; statute not read). - **Rhythm's exposure (Inferred):** serving a tenant with the tenant's own provider key, stored only in that tenant's records, looks like a service provider/platform role (explicit exclusions in Texas, Oregon, and, until 2026-12-31, Vermont). A cross-tenant cache that licenses or serves enrichment data about people with whom Rhythm has no relationship to other tenants looks like "collects ... and sells or licenses" and moves Rhythm toward registration in all four states and into the DROP deletion regime. This is a strong additional reason not to build the shared cache (Section 11). #### 3.4 Do state comprehensive privacy laws cover nonprofits and associations? Sources: Wiley alert 17 Dec 2024 and Privacy Law Map (dated 30 Mar 2026), both Reported (law-firm and practitioner summaries; statutes not read). Check each statute before relying. | State | Nonprofit treatment (as reported) | | --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | | California (CCPA) | Not a "business" unless for-profit control/common branding/commercial arm | | Colorado, New Jersey, Oregon | No general nonprofit exemption (Wiley); one source marks Oregon "narrow" (insurance-fraud nonprofits only per Privacy Law Map) | | Delaware | Very narrow (insurance-fraud nonprofits); threshold 35,000 residents | | Minnesota | Exempts only nonprofits established to detect/prevent insurance fraud | | Maryland (MODPA; enforcement from 1 Apr 2026) | Limited, insurance-related only | | Virginia, Texas, Tennessee | Broad nonprofit exemption (Texas has no volume threshold but exempts nonprofits) | | Connecticut | Exempts 501(c) organisations; SB 1295 amendments from 1 Jul 2026 "may narrow" | | Rhode Island | 501(c)(3) only | | Kentucky (from 1 Jan 2026) | Religious/charitable/educational, effectively 501(c)(3) | | Indiana | Exempts 501(c)(3), (c)(6) and (c)(19) only | Associations are frequently 501(c)(6) trade bodies, which are exempt in Indiana but not in Rhythm-relevant states like Colorado, Oregon, New Jersey, Minnesota, Maryland, Delaware (Reported). Volume thresholds (often 100,000 residents) may take a tenant out of scope even where no exemption exists. Practical rule (Inferred): the platform cannot know each tenant's status, so ship the strictest baseline (notice, opt-out, sensitive-data opt-in, processor contract) and let tenants disable features rather than build state-by-state logic. Sensitive data: CCPA "sensitive personal information" includes precise geolocation, government IDs, racial/ethnic origin, religious/philosophical beliefs, union membership, private message contents, genetic/neural data, biometric data processed to identify, health data, and sex life/orientation; it excludes lawfully "publicly available" information (1798.140(ae), Verified via california.public.law). Several state laws require opt-in consent for sensitive data (Reported; not read). #### 3.5 CAN-SPAM, TCPA - CAN-SPAM (Verified, FTC compliance guide): no B2B exemption; opt-outs honoured within 10 business days; opted-out addresses may not be sold or transferred; address harvesting and dictionary attacks are prohibited; both the sender and the advertiser are liable and "you can't contract away your legal responsibility"; penalties up to USD 53,088 per email. - TCPA (Reported: Kelley Drye; Foley): the Eleventh Circuit vacated the FCC's "one-to-one" consent rule on 24 Jan 2025; the 2012 prior express written consent requirement remains; the revocation-of-consent rules scheduled for 11 Apr 2025 were under FCC review (current status not verified). Application (Inferred): an appended mobile number is not consent to be texted or auto-dialled. Enriched phone numbers should be classed "contact-info, not marketing-permission" and must not populate SMS audiences automatically. #### 3.6 BIPA and profile photos - Illinois BIPA regulates "biometric identifiers" (retina or iris scan, fingerprint, voiceprint, scan of hand or face geometry) and "biometric information" derived from them. The statutory exclusion of **photographs** is from my general knowledge of 740 ILCS 14/10; I could not fetch the statute (Illinois site access denied) so it is Inferred. What I did read (Reported: Orrick; Eric Goldman; Perkins Coie) is that in Rivera v. Google the claim turned on face templates derived from photos, not on the photographs themselves, and was ultimately dismissed on standing grounds in 2018. - Per-person damages after SB 2979 (signed 2 Aug 2024): a single recovery per person for repeated collection/disclosure in the same way (Reported: Fisher Phillips; Kilpatrick Townsend; retroactivity contested). - Clearview AI: a nationwide BIPA settlement worth about USD 51.75m, mostly as an equity stake, was approved 20 Mar 2025 (Reported); a Seventh Circuit opinion (Weissman v. Clearview, July 2026) raised a procedural problem with subclass benefits; final disposition not verified. - Application (Inferred): storing or displaying a profile photo URL is not itself a BIPA event. Any step that computes a face template, runs face matching to "confirm" that a photo belongs to a member, or searches by face crosses into biometric territory (BIPA; GDPR Art. 9 where used for unique identification; EU AI Act Art. 5(1)(e) ban on creating or expanding facial-recognition databases through untargeted scraping of facial images, Verified, applicable since 2 Feb 2025). The architecture should prohibit face matching and any provider whose method is facial recognition. PDL says it refuses "biometric, facial, or precise location data" (Verified, PDL privacy and security overview PDF [raw text via fetch]). #### 3.7 FTC and data-broker enforcement 2024-2026 - Dec 2024: FTC actions against Gravy Analytics/Venntel and Mobilewalla (proposed consent orders announced 3 Dec 2024; finalised 14 Jan 2025 per Hunton, Reported). WilmerHale (Reported): unfairness theories under FTC Act s.5; the FTC faulted reliance on "vague contractual assurances" from suppliers instead of verifying consent; orders require supplier-assessment programmes, sensitive-location lists, deletion of historic data and derived work product on consent withdrawal (Mobilewalla), and bar misrepresentation. Downstream relevance: a buyer or platform cannot rely solely on a provider's warranty that its data was lawfully sourced; document diligence. - Earlier cases in the series: Kochava, X-Mode, InMarket (Reported; mentioned in the same sources; Kochava's current status not verified). - 2025-2026 FTC data-broker actions and any Protecting Americans' Data from Foreign Adversaries Act enforcement: not verified; my fetch of FTC's own pages returned 404 and search budget ran out. Flag for follow-up. --- ### 4. Question 3: Canada and Australia #### 4.1 Canada - **PIPEDA s.4.01** (Verified, laws-lois.justice.gc.ca): Part 1 "does not apply to an organization in respect of the business contact information of an individual that the organization collects, uses or discloses solely for the purpose of communicating or facilitating communication with the individual in relation to their employment, business or profession." The exclusion is narrow: "solely" for that purpose. Using enriched business-contact data for profiling, segmentation or resale takes it out of the carve-out (Inferred reading). - **"Publicly available" is defined by regulation** (Regulations Specifying Publicly Available Information, SOR/2001-7, Verified): five categories only: telephone directories (where the subscriber can refuse listing), professional or business directories (name, title, address, phone, with use relating directly to the directory's purpose), statutory registries, court records, and publications (magazine, book, newspaper) "where the individual has provided the information". Social network profiles are not on the list. The OPC's Profile Technology investigation found Facebook profiles not "publicly available" under PIPEDA (Reported: BLG, 2018). Consequence (Inferred): "it was public on the internet" is not a consent substitute in Canada; photos and personal social profiles need consent. - **OPC draft guidance on assessing third-party service providers** (released 10 Sep 2026; comments due 4 Dec 2026; Reported, Bennett Jones): extra scrutiny for products relying on publicly available information; check providers' own use of data (including training algorithms), subcontractors, offshore processing, return and destruction at contract end, and ongoing monitoring. - **Quebec Law 25** (Act respecting the protection of personal information in the private sector): Reported only (law-firm and vendor commentary, not the statute): a privacy impact assessment before communicating personal information outside Quebec; consent must be "clear, free and informed" and requested for each purpose; profiling technologies must be disclosed and (commentators disagree) off by default; people must be told of decisions based exclusively on automated processing and may ask about the main factors. I found no source on a Quebec "data broker" registry and did not confirm one exists. Inferred: enrichment that builds profiles of Quebec members is "profiling"; treat as opt-in. - **CASL** (commercial electronic messages): I could not fetch the Spam Reduction site (404). From general knowledge (Inferred, unverified): consent is required for commercial electronic messages; "conspicuous publication" of a business address can give implied consent only if there is no notice refusing unsolicited messages and the message is relevant to the recipient's role; sender must offer an unsubscribe mechanism. A tenant that emails Canadians using an enriched address must be able to show the consent basis. Needs counsel. #### 4.2 Australia - **APP 3.5/3.6** (Verified, OAIC guidelines Ch. 3): collection must be by lawful and fair means; collect from the individual unless "unreasonable or impracticable". Covert collection is usually unfair; publicly available information "can't be collected and used in any way the entity chooses"; inconvenience or cost alone does not make direct collection impracticable; using a data broker for enrichment "doesn't qualify for the 'impracticable' exception simply because direct collection is inconvenient"; the organisation may be liable for a third party's collection on its behalf, "particularly if it is offshore". - **APP 7** (Verified, OAIC guidelines Ch. 7): where information is collected from a third party or the person would not expect marketing, use for direct marketing needs consent (or a showing that obtaining it is impracticable) plus a prominent opt-out in each message; opt-out must be honoured (generally within about 30 days); the entity must disclose the source of the information on request. - **Reforms (Reported):** Privacy and Other Legislation Amendment Act 2024 (most provisions from 10 Dec 2024); a statutory privacy tort from 10 Jun 2025; automated-decision transparency obligations from 10 Dec 2026; the small-business exemption (annual turnover USD 3m or less, s.6D) exists, but the OAIC-relevant carve-out for businesses that trade in personal information is from my general knowledge only (Inferred; the Act page I fetched showed only the table of contents). - Application (Inferred): Australian members whose data was collected by a provider "from the internet" and handed to the association are in APP 3.6 and APP 7 territory; plan for consent-or-impracticability evidence and a source-disclosure answer on request (Rhythm should record the provider and retrieval date for every enriched field to answer it). --- ### 5. Question 5 (placed before the photo and scraping sections because it sets the contractual envelope): provider data-licensing terms Questions asked of each provider: stored indefinitely? refreshed? displayed to the customer's own end users (members in a portal)? used in AI/LLM features? redistributed or exported? OEM versus BYOK / multi-tenant "one key, many customers"? #### 5.1 Comparison table "V" = provider's own page read this session; "R" = reproduced or described by a third party. Version/date is what the page showed. | Provider (version/date; evidence) | Storage/retention | Display to customer's end users | AI/LLM | Redistribution/export | One key, many customers / OEM | | --------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Apollo** ToS, last updated 10 Aug 2026 (V) | Perpetual license to Business Contact Information incorporated in your records (2(a)(ii)), subject to 3(e); may keep after termination (5(c)(3)); non-paying inactive 6 months may be deleted (5(d)) | License is for "internal business purposes" (2(a)(i)); no clause on showing to third parties; 3(d)(ii) bars making Services "available to third parties" | 3(e)(v): may not use Business Contact Information to train AI/ML except own internal models not commercially offered; 3(g)(ii)(c) APIs not to let third parties train | 3(d)(ii) no resell, distribute or sublicense; 3(e)(i)-(ii) no distribute, sell or license Business Contact Information or use it to build competing databases; 3(f)(iv) no bots/crawlers | 3(d)(iii): accessing the Platform "on behalf of another person or entity" prohibited; 3(g)(ii): sublicensing API access prohibited; seats cannot be shared (4(d)(i)); multiple accounts to dodge limits prohibited (4(b)(ii)). Implication (Inferred): a Rhythm-held key serving many tenants is outside the terms; per-tenant account is the only clean path; or negotiate a partner agreement | | **ZoomInfo** License Terms, "Version: May 1, 2025" (V); 2021 version also read (V) | 2025 3.3.1(a): licensee may keep using Licensed Materials obtained during the Term; 2021 3.4.1 required destruction of all copies and derived information on termination. Same company, different position in four years; pin the version | 2.3: use limited to licensee's B2B sales, marketing, recruiting; 2.3: no use "for the benefit of or on behalf of any person or entity except Licensee" | No AI-specific clause in 2025 LTC; 5.5/5.6 let ZoomInfo use submitted/related information to improve its products | 2.3: no distribute, sublicense, transfer, sell; no incorporating into "Licensee's own products or services"; 2.4(d) no "service bureau"; no audience segmentation except via ZoomInfo | 2.3: may not "incorporate any ZoomInfo API credentials into any Third Party Applications"; 2021 7.3 liquidated damages USD 2.00 per record accessed by third party via credentials. No public OEM terms found (Reported: only a generic third-party OEM guide). Opt-out: 5.4 licensee must review ZoomInfo's removal list at least monthly and delete unless an independent lawful basis exists | | **FullContact** Services Agreement, effective 29 Jul 2026 (V) | 2(b)/7(c): data licensed for the Subscription Term; on end, renew or "irrecoverably delete" within 30 days (pre-existing data and qualifying Derived Insights excluded); 7(e) keep records 3 years, SQL Log Attestation | Exhibit A: a Customer Application must "provide significantly additional data, content and functionality than that provided by the FullContact Data alone"; access limited to Third Party End Users using it for a Permitted Use; end users must delete on termination | 2(d)(1)(vi): no use to "develop, train, fine-tune, refine, or improve" AI systems; exception: AI "to analyze, process, or manipulate" for internal business purposes | 2(b) non-transferable, non-sublicensable; 4(a) FullContact will not resell your Submitted Information; Exhibit B: Fourth Party End Users must not resell or redistribute | Only provider examined with an express end-user pass-through: Exhibit A (Third Party End Users) and Exhibit B (Fourth Party) under written flow-down terms. Prohibited uses: employability/credit/FCRA 603(d) characteristics (2(d)(1)(i)); automated decisions on financial, housing, insurance, education, employment, health, essential goods (2(d)(1)(ii)); customer must maintain a privacy policy and hold "any third party consents required" (4(b)); may not submit health/payment/SSN/ID data (4(b)) | | **People Data Labs** | PDL's own license pages could not be fetched (privacy.peopledatalabs.com renders without text; peopledatalabs.com/terms-of-service 404). Docs (V): licenses are annual; use "in any way as long as it is compliant" with the Acceptable Data Use Policy. Reported (Primer reproduction of PDL terms): use limited to "internal business purposes"; after termination retained data may be used under the same terms; customer "solely responsible" for ongoing privacy workflows and opt-outs | not established | not established | Reported (Primer, PDL section 2.2): no resell or sharing; no giving third parties access to "PDL Services or data derived from them"; "may not act as a reseller"; no derivative works of the services; no PDL trademarks. 2.5: no law-enforcement/defence/government eligibility uses; 2.6: no FCRA purposes, no consumer reports, no adverse action | Reported: reselling and third-party access barred, so a many-tenant model needs a separate agreement with PDL. PDL privacy overview (V): sources are public records, "manifestly public" data and proprietary suppliers that "warrant" compliance, plus PDL-generated inferences; it does not sell sensitive PII and refuses biometric, facial or precise location data; accepts opt-out, portability, correction and deletion requests | | **HubSpot (Clearbit/Breeze Intelligence)** Product Specific Terms, "Last Modified" 16 Sep 2026, section 6 (V) | PST does not state a retention rule; help article: continuous enrichment updates previously enriched records monthly | n/a (data sits inside HubSpot) | PST 6.5: HubSpot may use Enrichment Data you transmit to develop its commercial dataset; AI is separate (section 7) | no express restriction on Enrichment Outputs; outputs are "HubSpot Content" (6.3) | Controller-to-controller (6.5(iii)); customer warrants notices, opt-out mechanisms and consents (6.7); contact-level "enrichment opt out" property when someone asks HubSpot to delete (help article, V). Clearbit's free Logo API shut down 8 Dec 2025 (Reported: HubSpot developer changelog via search) | | **Hunter** ToS, last updated 23 May 2024 (V); DPA (V) | ToS silent on retention; DPA: deletion at customer's choice at end of service; customer can self-serve delete | ToS silent | ToS silent | ToS 8: no "similar or competitive service"; no manual copying of the website | ToS 1: one account only, "must not share your account with others"; ToS 11/15 and DPA: customer is an independent controller for its use; Hunter independent controller for Profile Data collected from public web pages; notifies customer of data-subject requests, customer must pass erasure/objection back (DPA 7.1) | | **Clay** ToS, "last updated" May 23, 2024 per fetch (V; the same date as Hunter's, so re-check) | 11: kept while account active, deleted on account deletion | uploaded data not visible outside workspace | n/a | 4: "You also agree not to re-sell any data you obtain from Clay"; credits non-transferable | Aggregator of "more than 50 data providers"; each provider's own terms apply (7); account/credits cannot be transferred (4). A reseller layer over providers; adds flow-down, not relief | | **Cognism** | Only the website terms were retrievable (V: they bar sublicensing the Site; no data terms). Customer agreement, DNC screening, retention, AI: not obtained | not established | not established | not established | not established; ask for the customer agreement | | **Brandfetch** (logos/brand data) GTC, no date shown (V) | 5.1: Cached Content for a maximum of 30 days from retrieval; new API call at least every 30 days to renew; delete expired cache; failing is a material breach (4); caching only under a specific written agreement (2.1) | 5.1 licence does not extend to the original logos; third-party IP stays with its owners; 4(ii)/4(v): may not display/transmit/distribute the Services or make them available to third parties | 2.6: no training generative or ML AI on data from the Services without written agreement | 2.6: Services for Customer's own business purposes only; no use for third parties or sublicensing without written consent; no scraping at scale | Customer-only; Authorised Users named (2.5); a multi-tenant use needs written consent | | **Gravatar** (Automattic) ToS last updated 6 Oct 2026, s.8.b; Developer guidelines (V) | Docs: the API "allows you to save user data in your database for any other use or functionality in your app" (V, docs.gravatar.com); Guidelines: apps "may cache or store" content but should "refresh any data that you store at reasonable intervals" and tell users how you copy, cache and store it | Profile information is public by default (ToS 8.b); contact_info only returned if the user made it public and only to authenticated requests (V) | not addressed | Guidelines: do not share or misuse your API token; no unsolicited communications; no governmental surveillance supply; a privacy policy is required | Free API; limits 100/hour unauthenticated, 1,000/hour authenticated, raisable (V). A 2013 clause (Reported) said third parties "must not copy, store or modify User Submissions"; today's docs say the opposite. Treat the current docs/guidelines as controlling but ask Automattic for written confirmation if photos will be copied | | **Crunchbase** data-access license v3.1 page (agreement dated 29 May 2017; page metadata 2025; may be superseded by v4 terms) (V) | Enterprise/Research/Venture: expunge data within 10 days of termination; Basic: no expungement duty | Attribution with hyperlink required for Enterprise/Research/Venture when content is visible to third parties | not mentioned | Enterprise: "internal research and analysis" for a single department/use case; no license, sublicense, sale or distribution to third parties | Enterprise is single-use-case; per-customer Order Form controls | | **Dun & Bradstreet** generic Terms and Conditions page, no date (V); other D&B agreements (R: SEC exhibits, DACH terms) | 10.4: delete originals and copies after license term ends or data superseded; certify on request; only reasonable internal copies (4.1) | 3.2: may not provide Information to others "directly or indirectly, including through a database, marketing list, or report" | not mentioned | 3.2 as above; no statistics or recommendations to third parties built from the Information | 6.1: a third party that processes or hosts Information must first sign a D&B Processor's Agreement, so Rhythm hosting a tenant's D&B data would need that agreement. Personal-data: no use for consumer credit/insurance/employment eligibility (3.5) | #### 5.2 What the table says about the multi-tenant key question - **BYOK is the only generally safe default** (Inferred from the clauses above): each tenant holds its own contract and key; Rhythm is the tenant's agent/processor invoking the tenant's account. Even then, "accessing on behalf of another" (Apollo 3(d)(iii)) and "incorporate ... API credentials into any Third Party Applications" (ZoomInfo 2.3) language is a risk that the provider treats Rhythm as an unauthorised intermediary; written confirmation from each provider is prudent. - **A Rhythm-held key serving many tenants needs an OEM/reseller/embedded agreement** whose terms must cover: end-user (tenant and member) display rights, caching/retention beyond the subscription, deletion and opt-out feeds, AI use of results, per-tenant data segregation, controller/processor allocation, and who gives the Art. 14 information. FullContact's Exhibit A/B structure is the template closest to what Rhythm needs. For ZoomInfo, PDL, Apollo, Brandfetch and D&B no public OEM terms exist in what I could read (ZoomInfo: Reported only generically). - **Rhythm should not credit-pool or share one subscription across tenants** (Apollo 4(b)(ii), 4(d)(i); ZoomInfo 2.1 credential sharing and 2021 7.3 liquidated damages; Hunter ToS 1). - **Display to members in a portal** is generally unaddressed or restricted. ZoomInfo restricts use to the licensee's own B2B sales/marketing/recruiting; D&B bars providing Information to others "including through a database ... or report". A member-facing directory or profile page populated from licensed enrichment data is outside what most licenses grant (Inferred); Gravatar public avatars and tenant-supplied data are the safest to show. - **Eligibility/FCRA exclusions are common** (PDL 2.6 Reported; FullContact 2(d)(1)(i)-(ii) Verified; ZoomInfo 2.3 Verified; D&B 3.5 Verified). Any tenant use of enriched data to decide membership eligibility, credentialing, board vetting, awards or hiring is outside the licenses (Inferred application; association use cases are not obviously covered, so state the prohibition in the tenant terms). #### 5.3 Provider-sourcing statements relevant to risk - Apollo (Privacy Policy, V): data from public websites, professional directories, public regulatory sources, vetted third-party providers, and customer-contributed data (contact details, call recordings, engagement logs, email and calendar data when authorised). Legal basis in EEA/UK/CH: legitimate interests mainly. Apollo keeps a minimal suppression list; a Global Privacy Control signal stops sale/sharing but does not remove the person. - PDL (privacy overview, V): public sources, proprietary suppliers, PDL-generated inferences; compliance review before adding sources. - Hunter (DPA, V): profile data collected "from public web pages using the Company robot". - ZoomInfo: opt-out list and Do Not Sell link; "Contributor" network and community data not read. #### 5.4 Terms drift (important for contract design) ZoomInfo changed from "destroy all copies of Licensed Materials ... and information derived" (2021 LTC 3.4.1) to "may keep using Licensed Materials obtained during the Term" (2025 LTC 3.3.1(a)). Brandfetch caches are 30 days. Crunchbase's v3.1 page says 10-day expunging. Pin version and hash of the terms accepted per provider per tenant; store `license_class` per field with the effective date (Section 14). --- ### 6. Question 4: Profile photos and logos #### 6.1 Legal character - A profile photo is personal data (GDPR Art. 4(1)). It is not special-category data by itself: Recital 51 (Verified): photographs "should not systematically be considered" special-category data and fall under biometric data "only when processed through a specific technical means allowing the unique identification or authentication of a natural person". So no face matching (also see AI Act Art. 5(1)(e), BIPA, Section 3.6). - Copyright: the photographer (usually the member for a selfie, otherwise a third party) owns copyright. The member's licence to the platform (Gravatar ToS 8.b: "worldwide, royalty-free, transferable, sub-licensable, and non-exclusive" licence to Automattic for providing/improving its services and "promoting your profile ... including via the API"; LinkedIn User Agreement 3.1: members own original content and grant LinkedIn a licence) does not itself grant Rhythm or the association a licence to copy or redistribute the image; the permission to use comes from the API terms and from the uploader's intent (Inferred). - Personality/publicity rights: New York Civil Rights Law s.50 (Verified): use of a living person's name, portrait, picture, likeness or voice "for advertising purposes, or for the purposes of trade" without written consent is a misdemeanour (with a civil claim under s.51, not read). Other US states have analogous statutes (California Civil Code 3344, Illinois Right of Publicity Act; not read). Whether an association member directory is "advertising or trade" is fact-dependent (Inferred). GDPR adds the image-as-personal-data analysis and, in some EU states, image rights (not researched). #### 6.2 Source-specific rules - **Gravatar**: Verified. Public by default; API docs permit saving user data in your database; guidelines for responsible use require refresh "at reasonable intervals", disclosure of how you cache/store, a privacy policy; token must not be shared or misused; restrictions on unsolicited communications. Free API rate limits: 100/hour unauthenticated, 1,000/hour authenticated, raisable at no cost. - Note (Inferred): requesting an avatar or profile by email hash reveals to Gravatar that the hash is of interest and, if the browser requests it, the viewer's IP address. A hash of an email is still personal data. Requests from Rhythm's backend (not the member's or staff browser) avoid leaking staff/member IPs. - **LinkedIn**: Verified (User Agreement, effective 3 Nov 2025, section 8.2): prohibits using tools to scrape or copy profiles and other data (item 2), copying, using, displaying or distributing information obtained from the Services "whether directly or through third parties such as search tools or data brokers" without the content owner's consent (item 4), bypassing access controls (item 3), and monetising the Services or related data (item 11). LinkedIn photos obtained via scraping-based providers are therefore unlicensed copies in LinkedIn's eyes; the enforcement record is in Section 7. - **Clearbit/FullContact/PDL-style photos**: PDL's overview says it refuses facial data (does not say it excludes photo URLs; not established). FullContact terms contain no photo clause I could find (Inferred: photos fall within generic "FullContact Data" and its deletion-on-termination rule). - **Brandfetch/Logo.dev (logos)**: Verified for Brandfetch (above): 30-day cache, no license to the original logos, not for third-party display. Trademark/copyright of logos belongs to companies. Clearbit's Logo API shut down on 8 Dec 2025 (Reported), a vendor-continuity lesson: store logo URL plus fetched-at, not a hard dependency. #### 6.3 What CRMs commonly do (Inferred: no source for market practice found) 1. Store a provider URL plus `fetched_at` and render through an image proxy with a short TTL, instead of hotlinking from the member's browser (avoids IP/Referer leakage and breakage when the source changes). 2. Keep a durable copy only for images the member or the tenant uploaded themselves. 3. Never include enrichment photos in exports, public directories or cross-tenant stores by default; treat photo as "display inside the tenant's staff UI only" until a license says otherwise. 4. Honour provider refresh/expiry (Gravatar "reasonable intervals"; Brandfetch 30 days) and purge on provider deletion/opt-out signals. 5. Offer a "member can upload their own photo" path so the enriched photo is a fallback, not the system of record. --- ### 7. Question 6: Scraping, LinkedIn and downstream risk #### 7.1 hiQ v. LinkedIn Verified/Reported (Proskauer note, Dec 2022; Reported by Privacy World and NatLawReview): - 2017 preliminary injunction for hiQ; Ninth Circuit affirmed 2019; Supreme Court vacated in light of Van Buren (2021); Ninth Circuit again affirmed in April 2022 reasoning that scraping publicly available data likely falls outside the CFAA; the district court dissolved the injunction in August 2022; on **4 Nov 2022** the court held hiQ breached LinkedIn's User Agreement by scraping and creating fake accounts (subject to equitable defences), while hiQ prevailed on the CFAA "unauthorised access" issue for public data. - **Consent judgment (entered 8 Dec 2022):** USD 500,000 judgment; permanent injunction against scraping in violation of the User Agreement, using fake accounts, and building a commercial service on LinkedIn without permission, whether or not logged in; hiQ must **delete all LinkedIn member profile data it holds, including data stored with third parties**, and delete code designed to access LinkedIn, including code held by third parties. hiQ's CFAA stipulations carry no precedential value. - Takeaway (Inferred): public availability defeated the CFAA theory but not the contract theory; and the remedy reached copies in other hands. #### 7.2 Later cases - **Proxycurl / Nubela**: LinkedIn sued on 24 Jan 2025 (N.D. Cal.; breach of contract, fraud, CFAA, among six claims; allegation of hundreds of thousands of fake accounts). Proxycurl shut down in July 2025; a permanent injunction requires it to erase LinkedIn data obtained by unauthorised means and stop accessing LinkedIn unlawfully, and requires it to send the injunction to its customers (Reported: Social Media Today 28 Jul 2025; other sources are vendor blogs with an interest). - **ProAPIs (iScraper)**: LinkedIn sued in early October 2025 (N.D. Cal.), alleging a network of millions of fake accounts and sale of data including members' posts, reactions and comments, for up to USD 15,000 per month (Verified: The Record article dated 3 Oct 2025). A consent judgment/agreement banning ProAPIs from access via fake accounts or bots and from selling the data was reported in 2026 (Reported; exact dates inconsistent in sources; confirm against the docket). - **Apollo and Seamless.AI** lost LinkedIn company pages in March 2025 without warning (Reported); there is no lawsuit against them in the sources I saw. Apollo's ToS 3(g)(ii) prohibits using its APIs "with LinkedIn automations". - **Mantheos** and other cases named in your brief: not researched (search budget exhausted); flag for follow-up. - **European enforcement of LinkedIn-derived data**: CNIL v. KASPR (Dec 2024) and Garante v. Lusha (Jul 2026, now suspended) both involved contact data derived from social networks (Section 2.6). #### 7.3 Which providers source from scraping, and the downstream risk - Not established from first-hand terms for most providers. What the sources say: KASPR scraped LinkedIn (CNIL); Lusha "scraping from social networks" plus purchased data (Reported press); Proxycurl and ProAPIs scraped (court filings as reported); Apollo's own policy lists public websites, third-party providers and customer-contributed data; PDL's overview lists public sources and proprietary suppliers; Hunter crawls public web pages. None of PDL, Apollo, ZoomInfo, FullContact or Cognism state "we scrape LinkedIn" in what I read, and I make no claim that they do. - **Downstream user risk (Inferred, with supporting evidence):** 1. Contract: LinkedIn's terms bind members and users, not a tenant who never signed up; but LinkedIn's remedies have reached third-party-held copies (hiQ) and customers have been served with injunction notices (Proxycurl). A tenant could suddenly be told to delete a provider's data. 2. GDPR/UK GDPR: the ICO states the buyer must check who compiled the data and how; EDPB Opinion 28/2024 scenario 2 says a deployer is expected to assess whether upstream processing was unlawful, scaled to risk; the Experian line says consent-sourced data cannot be silently repurposed under legitimate interests. CNIL's KASPR order shows deletion obligations for data collected despite members limiting visibility. 3. FTC: reliance on supplier assurances alone has been called unfair (WilmerHale, Reported). 4. Practical: contract for deletion-notice flow-down, indemnity, and a provenance field per value. --- ### 8. Question 7: Email verification/append and consumer data appends I could not read AtData/TowerData, Versium, Melissa or Experian terms (AtData page 404; others not attempted before the search budget ran out). What follows is law applied to the pattern, not vendor-specific findings. - **What an append is**: adding an email, phone or postal address to a person record from a third-party database, typically a consumer-identity graph. The legal posture is that of a broker supplying data for direct marketing. - **EU/UK** (Verified sources above): the buyer needs its own lawful basis before obtaining the data, must give Art. 14 information within a month or at first communication, and for electronic marketing to individuals needs ePrivacy consent (EDPB LI Guidelines para. 114; ICO). The ICO page lists "profiling and data enrichment (adding data to existing profiles)" among broker services and says customers are unlikely to expect them. A consumer/personal email append for an association member is the weakest case under reasonable-expectations analysis. - **US**: no federal statute bans appends; CAN-SPAM's address-harvesting ban targets automated collection from websites and random-generation attacks, but its opt-out and no-transfer rules apply (Verified FTC guide). TCPA consent attaches to the person and number, so an appended mobile number gives no consent (Section 3.5). State laws on data brokers and "sale" apply to the provider more than to the association, except where the association is a covered business and receiving data from a registered broker (Inferred). Sensitive-data inferences in an append (health, ethnicity, religion proxies) raise state sensitive-data and FTC unfairness issues (Section 3.7). - **Canada/Australia**: Section 4 (PIPEDA "publicly available" narrowness; APP 3.6 and APP 7 consent-or-impracticability and source disclosure). - **Opt-out obligations to model**: GDPR Art. 21(2)-(3) objection to direct marketing is unconditional once made; keep a suppression record (ICO); CAN-SPAM 10 business days and no transfer of opted-out addresses; APP 7 simple opt-out and honouring it; Delete Act DROP for registered brokers (the broker, not the tenant). - **Membership-organisation angle**: an association already has a direct relationship with its members and so can lawfully hold their contact details; the legal risk is in appending data the member did not give the association. A defensible narrow case is verification (checking that an existing email is deliverable, without adding new personal data) and correction of data the member gave; the weak case is acquiring new channels (personal phone, home address) for marketing (Inferred). --- ### 9. Question 8: AI-driven research enrichment (LLM agent browsing the web) #### 9.1 What regulators have actually said, and its scope - **EDPB draft Guidelines 03/2026** (adopted 7 Jul 2026; consultation to 30 Oct 2026; Verified [raw]): scope is web scraping "in the context of training generative AI" by private entities. It expressly does "not address processing of an organisation's own personal data" (para. 5) and does not address "data brokers who scrape data and just hold data sets" (para. 4). So an inference-time research agent that browses to profile a known member is **not squarely covered** (Inferred); the principles are the nearest authority. - **Principles the draft sets that carry over (Verified [raw]):** - Legitimate interests is the main avenue; consent is generally unworkable at scale (para. 44); three cumulative conditions (para. 47); the absence of robots.txt "does not amount to consent" (para. 45). - Reasonable expectations turn on source type, whether data was made public by the person, and restrictions on the source (robots.txt/ai.txt, CAPTCHA, login) (para. 64); mitigating measures include excluding login-gated sources and sources people cannot be aware are accessible (para. 66(c)); an opt-out list (para. 66(f)). - Para. 54 flags a "chilling effect" where "the AI model is connected to a web search tool" and may "identify data subjects who intended to express themselves anonymously" or be used for profiling. - Accuracy (paras. 39-42): scrape from reliable sources, **timestamp** the data, **validate** formats and spot-check factual accuracy; avoid "secondary or outdated aggregators". - Special categories (paras. 67-77): Art. 9 derogation needed; incidental, residual collection may be tolerated only with filters before, deletion after and output filters during/after; delete on a plausible request. - Example 7 (para. 66): a text system using only freely available sources manifestly made public, excluding copyrighted content, with memorisation safeguards and a public privacy policy listing sources, "may generally be considered" to pass the balancing test. - **EDPB Opinion 28/2024** (17 Dec 2024; Verified [raw]): a deployer must assess whether the model was developed unlawfully, scaled to risk; Art. 14(5)(b) strictly limited. - **EDPB ChatGPT Taskforce report** (23 May 2024; Verified [raw], paras. 29-31): accuracy principle applies to input and output; outputs "are likely to be taken as factually accurate by end users, including information relating to individuals"; transparency about unreliability is "not sufficient to comply with the data accuracy principle". Directly relevant: an LLM-written statement about a member (employer, title, affiliation) is personal data and must be accurate. - **ICO**: the ICO consulted on generative AI and published a response page (Verified existence; the content pages on lawful basis for web scraping, accuracy and individual rights were not readable). Not relied on here. - **Italian Garante**: fined OpenAI EUR 15m (20 Dec 2024) over lack of legal basis and transparency, then the decision was annulled by the Court of Rome (judgment 4153/2026, published 18 Mar 2026) (Verified, Garante page). Regulator positions on AI scraping are being tested in court. #### 9.2 Specific issues for an agent-driven enrichment feature | Issue | Position | Evidence/label | | ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Lawful basis and necessity | Same three-part test; narrow collection, do not crawl widely; prefer structured sources | EDPB 03/2026 paras. 47-52 (Verified [raw]); application Inferred | | Reasonable expectations | A member does not expect an AI agent to compile a dossier from social and web sources; the EDPB says awareness of AI scraping does not extend to "all situations, for all purposes" | Para. 64 (Verified [raw]) | | Article 22 | Applies only to decisions "based solely on automated processing" with legal or similarly significant effects; enrichment itself is profiling input, not usually a decision. Becomes live if enriched data drives automated eligibility, pricing, discipline or access decisions without human involvement. Exceptions require contract necessity, law or explicit consent, with human intervention rights | Art. 22 text (Verified). CJEU SCHUFA (C-634/21) treating a third party's scoring as a decision when it is strongly relied upon: from my memory, not verified, Inferred | | Art. 14(2)(g) | If profiling with significant effects, give meaningful information on logic | Art. 14 text (Verified) | | Accuracy/hallucination | Controller remains responsible for accuracy of what it stores; disclaimers do not cure it; need validation, source evidence, correction path (Art. 16) | ChatGPT TF paras. 30-31 (Verified [raw]); Art. 5(1)(d) | | Special categories by inference | Agent may surface religion, health, politics, union membership from web pages; for religious/political/union associations the membership link is itself Art. 9 data | Art. 9(1), 9(2)(d) (Verified); EDPB 03/2026 paras. 67-77 (Verified [raw]); application Inferred | | EU AI Act | Art. 5(1)(e) (applies since 2 Feb 2025; Verified): no building or expanding facial-recognition databases through untargeted scraping of facial images. Other prohibitions (5(1)(c) social scoring; 5(1)(g) biometric categorisation inferring race, political opinions, union membership, religion, sex life; 5(1)(f) emotion inference at work/education) bound the agent's tasks. High-risk (Annex III) uses such as employment or access to essential services were not read; avoid them. Fetched page also shows amendments to 5(1)(ba)/(bb) with effect 2 Dec 2026 (not analysed) | Verified (Art. 5 text); Annex III Inferred | | Provenance | Every agent-written field needs: source URL(s), retrieval timestamp, model/version, prompt/tool-policy version, evidence snippet hash, confidence, and an "observed versus inferred" flag; no auto-write to canonical member fields without human confirmation | Inferred design; supported by EDPB accuracy recommendations (timestamp, validate) | | Respect for technical signals | Honour robots.txt, ai.txt, CAPTCHA, login walls; never use authenticated or fake-account browsing (cf. hiQ/Proxycurl) | EDPB 03/2026 paras. 37, 64, 66 (Verified [raw]); hiQ (Section 7) | | Transparency | Publish the crawler's characteristics and source types; list sources "to the greatest extent possible" | EDPB 03/2026 para. 30-31 (Verified [raw]) | #### 9.3 Practical rule (Inferred) Treat agent research as the highest-risk enrichment source: opt-in per tenant, scoped to a named list of sources (company website, official registries, the member's own business page), no social networks or login-gated sources, no photo/face tasks, and results written to a "proposed enrichment" queue for human approval with provenance attached. --- ### 10. Question 9: Retention, deletion, erasure propagation, residency #### 10.1 What the providers require (summary of Section 5) - FullContact: delete within 30 days of term end; 3-year records and SQL-log attestation. Brandfetch: 30-day cache. Crunchbase (Enterprise): expunge in 10 days. ZoomInfo: monthly review of opt-out list and removal (5.4); post-termination retention changed between 2021 and 2025. HubSpot: monthly refresh; contact-level "enrichment opt out". Apollo: perpetual licence to incorporated contact data; suppression list at Apollo. Gravatar: refresh at reasonable intervals. PDL (Reported secondary): API deletions/opt-outs applied immediately to its dataset while licensed-file customers receive them in later deliveries; customer is responsible for downstream opt-out workflows. D&B: delete superseded data and certify. - Consequence: the framework must offer per-provider TTL, per-provider "deletion feed ingestion" (where a provider publishes opt-out lists) and a per-field expiry date. #### 10.2 Right to be forgotten propagation - Controller (tenant) obligations: erase (Art. 17), tell recipients (Art. 19), tell other controllers if made public (17(2)). Rhythm must wire (a) tenant-initiated erasure to remove the person's enrichment payloads from every store (live tables, backups on rotation, caches, search indexes, analytics extracts, AI feature stores, exported snapshots held by Rhythm) and (b) an outbound suppression notice to each provider that received the identifier, where the provider has an API or form for it (Hunter DPA 7.1: customer must pass erasure/objection requests back; Apollo: Privacy Center/Removal page; HubSpot: opt-out property). Verified for the cited provider mechanics. - Upstream deletion (a provider deletes a person after a request or DROP) does not delete copies already in tenants' records. Providers push this to the customer (PDL, Reported; ZoomInfo 5.4 Verified). So Rhythm needs an ingestion path: periodic checks of provider opt-out lists; on a match, either delete the provider-sourced fields or require the tenant's recorded independent lawful basis (ZoomInfo 5.4 says delete "unless it has an independent lawful basis"). - DROP: deletion includes inferences (Verified); Rhythm-held derived fields (scores, segments) built from provider data are in the same risk class as inferences (Inferred). #### 10.3 Audit logs and enrichment history versus erasure - Art. 17(3)(b) (legal obligation) and 17(3)(e) (legal claims) can justify retaining a minimal record, but they do not justify keeping the enriched values after an erasure request merely for convenience (Inferred reading of the exceptions as Verified text). - Suppression is a recognised exception in practice: ICO B2B guidance says keep objecting contacts on a suppression list; CPPA's DROP rules require brokers to keep a list of requests so data stays deleted; Apollo keeps a minimal suppression list; EDPB 03/2026 para. 66(f) mentions opt-out lists (all Verified). - Recommended design (Inferred): split enrichment storage into (1) payload (values, deletable), (2) event log (who/when/provider/field names/outcome, no values, retained for the audit retention period), (3) suppression ledger (salted-hash of identifier, scope, date, reason; retained as long as the objection stands). If the audit log must keep values for a legal hold, encrypt per-person and shred the key on erasure. - Retention of provider data after the tenant terminates is governed by the provider licence (above), not only by tenant instruction; Rhythm must be able to delete a tenant's provider-sourced data on termination within the shortest provider deadline (FullContact 30 days; Crunchbase 10 days). #### 10.4 Residency - Provider processing locations are provider-specific (Hunter: Belgium with US sub-processors; PDL: not established). Quebec: assess before communicating outside Quebec (Reported). PIPEDA: accountability for offshore processing in the OPC draft guidance (Reported). Australia APP 8 (cross-border) not researched. - Design: record region per call and per storage location; allow per-tenant region pinning and provider allow-lists by region. --- ### 11. Question 10: Is sharing one provider response across tenants (an email-keyed cache) legally sound? Short answer: no. Evidence by layer. 1. **Controller/processor.** Each tenant is a controller of its own member data. Rhythm may process only on the tenant's documented instructions (Art. 28(3)(a), Verified). A response fetched at tenant A's instruction and reused to serve tenant B serves B's purposes with A's instruction; Rhythm decides to reuse it, i.e. determines purposes and means, and is "considered to be a controller in respect of that processing" (Art. 28(10), EDPB 07/2020 para. 81, both Verified). Rhythm would then need its own lawful basis, Art. 14 notice to every person in the cache, retention rules, and rights handling, the same position that brought Lusha and KASPR enforcement. 2. **Purpose limitation and expectations.** The data was requested for A's relationship with the person. Reuse by B, an unrelated association the person may never have joined, is a different purpose and a different relationship; EDPB LI guidelines para. 54 treat the existence and proximity of a relationship as central to reasonable expectations (Verified [raw]). Inferred application. 3. **Confidentiality of the member list.** A cache hit/miss and latency reveal that "someone already looked this email up", leaking membership overlap between competing associations; for religious/political/union tenants that is potentially Art. 9 data (Art. 9(2)(d): not to be disclosed outside the body without consent; Verified text; Inferred application). 4. **US service-provider rules.** 11 CCR 7050(a): a service provider "may not use the data to perform services for another person" and the example forbids using one business's customer list for another (Verified). 7051(a): no combining outside the direct relationship (Verified). 5. **Provider licences.** Redistribution/third-party-benefit bans: ZoomInfo 2.3, 2.4(d); Apollo 3(d)(ii)-(iii), 3(e)(i)-(ii), 4(b)(ii); FullContact 2(b) (non-sublicensable), 4(a) (FullContact does not resell Submitted Information, a symmetrical principle), Exhibit B; Brandfetch 4(v), 2.6; D&B 3.2; Crunchbase Enterprise; PDL 2.2(i) (Reported). Cached storage beyond the subscription term or across customers is generally not licensed (Brandfetch 30-day cap; FullContact deletion on termination). Verified (except PDL). 6. **Data-broker status.** A shared people-keyed store that Rhythm "licenses" to multiple customers matches the CA/VT/OR/TX definitions where Rhythm has no direct relationship with the people (Section 3.3, Inferred), pulling Rhythm into registration and DROP. 7. **Deletion and opt-out semantics.** A shared cache means a deletion requested through tenant A either (a) leaves B's copy (breaking the person's request if made against the provider-wide record), or (b) deletes B's copy without B's instruction. Per-tenant stores avoid this. 8. **Where sharing might be defensible (Inferred):** (a) non-personal reference data keyed by company domain/ticker (but sole-trader domains are personal data) where the provider licence expressly allows sharing; (b) provider-neutral metadata such as "this email domain is a free-mail domain"; (c) a global _suppression_ set (do-not-enrich identifiers) applied across tenants, which reduces rather than expands processing (but a global suppression based on one tenant's objection should be limited to provider-level opt-outs; a tenant-level objection should not suppress other tenants without a lawful reason). Recommendation: key every cache entry by (tenant_id, provider, identifier hash, field); never read across tenant_id; run a tenant-scope check in the connector, not only in application code (Section 14). --- ### 12. Question 11: Contractual patterns and design (all Inferred unless cited) #### 12.1 What Rhythm's customer terms/DPA must say to enable enrichment 1. **Feature is optional and tenant-activated.** Off by default; activation requires the tenant (controller) to attest to a lawful basis, completed LIA (or consent mechanism), Art. 14/13 notice given, and DPIA where required. Mirrors HubSpot 6.7 (customer warrants notices, opt-out and consents), ZoomInfo 5.7, FullContact 4(b) (all Verified). 2. **Instructions.** The DPA names enrichment as a processing purpose, lists the permitted provider connectors, the identifier types sent (email, name, employer, domain), the fields received and stored, and the retention rule. Rhythm acts only on those instructions (Art. 28(3)(a)). 3. **Recipient/sub-processor list.** Two categories: (a) sub-processors acting only on instructions (e.g. a provider used strictly as a processor); (b) independent-controller recipients (most enrichment providers). Both must be listed (Art. 28(2) prior authorisation for (a); Art. 13/14 recipients for (b)). Give tenants notice of changes with an objection right (Hunter's 14-day/7-day pattern is a market precedent; Verified). 4. **No Rhythm secondary use.** Express prohibition: no use of tenant lookups, results or identifiers for Rhythm's own purposes, model training, benchmarking or any other tenant; aggregated, de-identified service metrics only (compare 11 CCR 7050(a) Verified). 5. **Provider flow-downs.** Tenant agrees to the provider-specific "enrichment terms" (restricted uses: no eligibility/FCRA, no sensitive data, no redistribution, no AI training where the provider forbids; FullContact Exhibit A is a model of end-user flow-down). Rhythm's contract with the tenant must be at least as restrictive as each provider licence it passes through. 6. **BYOK versus Rhythm-held key.** BYOK: tenant's own account, tenant accepts provider terms, Rhythm stores the credential encrypted and uses it only for that tenant (confirm with each provider; Apollo/ZoomInfo language above). Rhythm-held: only under a signed OEM/reseller addendum per provider; the tenant-facing terms disclose which provider contracts apply. 7. **Deletion and suppression covenants.** Time-bound deletion on termination, on member erasure/objection, and on provider deletion notices (state the SLA, e.g. within 30 days to match FullContact and shorter for Crunchbase-style 10 days; align to the strictest provider licence). 8. **Cooperation.** Rhythm helps with access/erasure (Art. 28(3)(e)); supplies enrichment history to the tenant for an access request (source, provider, date). 9. **Liability and audit.** Right for the tenant to audit enrichment logs; Rhythm's indemnity limited to its own breach; tenant indemnifies for unlawful enrichment instructions or misuse (eligibility decisions). 10. **Transfers/residency.** State the regions used, provider locations where known, and the transfer mechanism (SCCs/DPF) per provider. 11. **Region/tenant-type restrictions.** Disable agent research and personal-email identity tiers for tenants who attest they are religious, political or union bodies (Art. 9) unless they provide a documented Art. 9 basis; allow EU/UK tenants to restrict enrichment to business-context fields. #### 12.2 Member-facing privacy notice text (what the tenant must be able to say) (Art. 13/14 content; drafting Inferred) A short block the tenant can drop into its privacy notice: - Purpose: "We supplement our membership records with professional and organisational information (for example your employer, role, and organisation details) so that we can keep our records accurate and serve you." - Source and recipients: "We obtain this information from third-party data providers [named or categorised], which may be provided your email address or name to match your record. Those providers may keep and use that information under their own privacy policies [links]." - Categories: "Typically job title, employer, industry, business location, public professional profile links. We do not collect [health, political, religious, trade-union, biometric or precise location data]." - Legal basis: "Our legitimate interest in keeping our membership records accurate [or: your consent, where we use personal-email-based sources]." - Retention: "Provider-supplied information is refreshed or deleted every [x] months and removed when you leave or ask." - Rights: access, rectification, erasure, objection (unconditional for marketing), complaint; how to object ("Do not enrich my record" link); transfers (country, safeguards). - Automated decision statement where relevant (Art. 14(2)(g)). #### 12.3 Opt-in/opt-out design - **Tier model** (Inferred, using the risk gradient above): - Tier 0: tenant-supplied or member-supplied data (no enrichment). - Tier 1: company firmographics/logos from company-level providers, no personal data (domain-keyed). - Tier 2: professional business-context data keyed by a business email (LI, notice, easy objection). - Tier 3: personal-email-keyed identity data (photo, social profile, personal phone/address) (opt-in or consent). - Tier X (not offered): face matching, biometric, precise location, special-category inference, scraping behind logins, eligibility use. - **Record-level suppression**: per-person "Do not enrich" flag honoured before any provider call and before any cache refresh; stored as a salted hash in the suppression ledger so it survives erasure of the record; settable by member self-service (portal), tenant staff, or an inbound provider opt-out feed. - **Member self-service**: portal control to see which enrichment fields exist, their source/date, to correct/delete, and to opt out; this operationalises Art. 14(2), 15, 16, 17, 21. - **Region-aware defaults**: EU/UK/Canada/Australia members: Tier 3 off, Tier 2 requires recorded notice date; US: Tier 2 on by tenant choice with opt-out, subject to state-law and tenant-type attestations. #### 12.4 Enrichment data classification (so exports and AI respect licences) Every stored enrichment value carries metadata (Inferred): `provider_id`, `provider_terms_version`, `license_class`, `sensitivity_class`, `purpose_tag`, `collected_at`, `refresh_after`, `expires_at`, `legal_basis_ref`, `tenant_id`, `suppress_on_optout`, `provenance` (URL/endpoint/model), and `confidence`. Suggested license classes derived from Section 5: `INTERNAL_ONLY_STAFF_UI`, `DISPLAY_MEMBER_PORTAL_ALLOWED`, `EXPORT_ALLOWED`, `AI_ANALYSIS_ALLOWED` (e.g. FullContact internal-use analysis), `AI_TRAINING_FORBIDDEN` (Apollo, FullContact, Brandfetch default), `CACHE_MAX_TTL` (Brandfetch 30 days), `DELETE_ON_TERMINATION_DAYS`. Export, reporting, mail-merge/audience builders, webhooks, portal rendering and AI/LLM prompt construction must call a single policy check that denies by default when a field's class does not permit the action. --- ### 13. Summary table: requirement -> generic framework handling versus provider-specific handling | Requirement | Generic framework handling (provider-neutral) | Provider-specific handling | | --------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Lawful basis and LIA record | Tenant activation gate: attestations, LIA/consent reference, DPIA reference stored per tenant and per tier | Some providers add duties (FullContact customer privacy policy and third-party consents, 4(b); HubSpot notices/consents, 6.7; ZoomInfo lawful-basis requirement for UK/EU, 5.4) | | Art. 13/14 notice and recipient listing | Notice pack generator; per-member notice timestamp; recipient/sub-processor registry | Provider names/policies/links; Apollo and Hunter controller status; transfer mechanism per provider | | Data minimisation | Field allowlist per tenant/tier; drop non-allowlisted fields on receipt | Provider field catalogues differ; request only needed fields (some providers charge or log per field) | | Opt-out / objection / suppression | Suppression ledger (salted hash); pre-call check; member self-service; cross-provider propagation job | Provider opt-out mechanisms: HubSpot "enrichment opt out" property; ZoomInfo monthly list; Apollo Privacy Center; Hunter DPA 7.1; PDL web forms; DROP for registered brokers | | Erasure propagation | Delete payloads in all stores incl. caches; event log without values; crypto-shred where needed | Each provider's deletion API/form and SLA; provider-side deletion notices into the tenant (ZoomInfo 5.4; PDL Reported) | | Retention and refresh | Per-field `refresh_after`/`expires_at`; purge jobs; deletion on tenant termination | FullContact 30 days; Brandfetch 30-day cache; Crunchbase 10 days; ZoomInfo version-dependent; Gravatar "reasonable intervals"; HubSpot monthly | | Display to members/portal | Policy engine: deny by default unless license class allows | Most licences silent or restrictive (ZoomInfo, D&B, Brandfetch); FullContact Exhibit A permits Third Party End Users under conditions; Gravatar public avatars | | Export/redistribution | Export filter by license class; watermark provenance | No redistribution: Apollo, ZoomInfo, PDL (R), FullContact, Brandfetch, D&B | | AI/LLM features | `AI_TRAINING_FORBIDDEN` default; prompt builder reads license class; no fine-tuning on enrichment | Apollo 3(e)(v); FullContact 2(d)(1)(vi) with internal-analysis exception; Brandfetch 2.6 | | Multi-tenant key / OEM | Tenant-scoped credentials and caches; per-tenant metering; no pooled credits | BYOK default; OEM addenda where available (FullContact Exhibit A/B model); Apollo/ZoomInfo/Hunter bar pooled or on-behalf use | | Cross-tenant sharing | Not permitted: cache key includes tenant_id | n/a (it breaches all licences above) | | Sensitive/prohibited uses | Static deny list: eligibility/FCRA, special-category inference, biometrics, precise location, scraping behind logins | Provider-specific lists: PDL 2.5-2.6 (R); FullContact 2(d)(1); ZoomInfo 2.3; D&B 3.5 | | Provenance and accuracy | Mandatory provenance fields; confidence; human-approval queue for agent-written data | Provider-supplied confidence/last-verified dates where offered | | Photo/logo handling | Proxy with TTL; link not copy; no face processing; no export | Brandfetch 30-day cache and written agreement; Gravatar guidelines; LinkedIn prohibits copying | | Transfers/residency | Region recorded per call/store; tenant region pinning; provider allow-list per region | Provider locations: Hunter EU (Belgium) with SCCs; others to be confirmed per provider; DPF certification check | | Audit and evidence | Immutable enrichment event log (no values) with tenant, member, provider, terms version | Provider audit rights: FullContact SQL logs and 3-year records; ZoomInfo 2-year usage records; Apollo CCPA-style audit | | US state-law gating | Strictest-baseline posture; tenant-type attestation (nonprofit status, special-category body) | Data-broker status of the provider (registered or not) as a vendor-risk attribute | --- ### 14. Architectural requirements for Rhythm (15 items) 1. **Tenant-scoped everything.** No cross-tenant read of provider responses, caches, suppression lists (other than provider-level opt-outs) or credentials; enforce `tenant_id` in the data-access layer, not only in handlers (Sections 11). 2. **Default-deny license policy engine.** Every enrichment value carries a `license_class`; export, portal display, mail-merge, webhooks and AI/LLM prompts call one policy function (Section 12.4). 3. **Tiered activation with attestations.** Tier 1-3 switches per tenant, with recorded lawful-basis, notice and DPIA attestations and tenant-type gating for religious/political/union bodies (Sections 2.2, 12.3). 4. **BYOK as the default credential model;** a Rhythm-held key only per provider after a signed OEM/reseller addendum covering end-user display, caching, AI use, deletion feeds and controller/processor allocation (Sections 5.2, 12.1). 5. **Suppression ledger separate from payloads,** keyed by salted hash, checked before every provider call and cache refresh, survives erasure, fed by member self-service, tenant staff and provider opt-out feeds (Sections 10.3, 12.3). 6. **Erasure/objection orchestration:** one request fans out to payload stores, caches, derived/AI feature stores, and outbound provider suppression calls, with completion evidence and SLA tracking (Section 10.2). 7. **Per-field provenance and expiry:** provider, endpoint, terms version, retrieved_at, refresh_after, expires_at, legal_basis_ref, confidence; purge on expiry and on provider-driven deletion (Sections 5.4, 9.2). 8. **Provider registry with pinned terms:** store provider terms version/hash, restricted uses, retention/deletion rules, controller/processor status, data-broker registration status, transfer mechanism, and sub-processor lists; re-review on change (Section 5.4). 9. **Minimisation at the connector:** field allowlist, return-field dropping, no "fetch everything" calls, and no sensitive/special-category field types stored (Sections 2.4, 3.4). 10. **Notice evidence:** Art. 14 notice pack and per-member notice date; block first-run Tier 2/3 enrichment for EU/UK/Canada/Australia members until the tenant attests notice or the notice date is recorded (Section 2.3). 11. **Photos and logos via a TTL proxy;** no durable copies except member/tenant-uploaded images; no face detection/matching code paths at all (Sections 3.6, 6.3). 12. **AI research agent as a separate, opt-in, human-approved pipeline:** allow-listed sources, robots/CAPTCHA/login respect, no social/login-gated sources, provenance on every claim, proposals queue not auto-write, output filters for special categories (Section 9). 13. **Transfer and region controls:** per-call region record, tenant region pinning, provider allow-lists by region, transfer mechanism documented per provider (Section 2.7). 14. **Prohibited-use and eligibility guardrails** in code and contract: block enrichment-driven automated eligibility, credentialing, hiring, credit or insurance decisions (Section 5.2). 15. **Immutable enrichment audit log without values:** who requested, which provider and fields, legal basis reference, outcome, and later erasure events; retained for the legal period, separate from payloads (Section 10.3). --- ### 15. Source list (URL; what it is; access date; evidence note) All accessed 2026-10-22. `[raw]` = I downloaded the document and read the full text with `pdftotext`; others were read through a summarising fetch tool, or are search-result summaries (marked "search summary"). #### 15.1 EU/UK/Canada/Australia law and regulators - GDPR Art. 9: https://gdpr-info.eu/art-9-gdpr/ - GDPR Art. 13: https://gdpr-info.eu/art-13-gdpr/ - GDPR Art. 14: https://gdpr-info.eu/art-14-gdpr/ - GDPR Art. 17: https://gdpr-info.eu/art-17-gdpr/ - GDPR Art. 19: https://gdpr-info.eu/art-19-gdpr/ - GDPR Art. 21: https://gdpr-info.eu/art-21-gdpr/ - GDPR Art. 22: https://gdpr-info.eu/art-22-gdpr/ - GDPR Art. 28: https://gdpr-info.eu/art-28-gdpr/ - GDPR Art. 35: https://gdpr-info.eu/art-35-gdpr/ - GDPR Recital 51: https://gdpr-info.eu/recitals/no-51/ - EDPB Guidelines 03/2026 on web scraping in the context of generative AI (adopted 7 Jul 2026) [raw]: https://www.edpb.europa.eu/system/files/2026-07/edpb_guidelines_2020603_webscraping_v1_en_0.pdf ; consultation page https://www.edpb.europa.eu/public-consultations/guidelines-032026-on-web-scraping-in-the-context-of-generative-ai_en (search summary) - EDPB Guidelines 1/2024 on legitimate interest (consultation version) [raw]: https://www.edpb.europa.eu/system/files/2024-10/edpb_guidelines_202401_legitimateinterest_en.pdf ; page https://www.edpb.europa.eu/our-work-tools/documents/public-consultations/2024/guidelines-12024-processing-personal-data-based_en - EDPB Opinion 28/2024 [raw]: https://www.edpb.europa.eu/system/files/2024-12/edpb_opinion_202428_ai-models_en.pdf ; page https://www.edpb.europa.eu/our-work-tools/our-documents/opinion-board-art-64/opinion-282024-certain-data-protection-aspects_en - EDPB ChatGPT Taskforce report (23 May 2024) [raw]: https://www.edpb.europa.eu/system/files/2024-05/edpb_20240523_report_chatgpt_taskforce_en.pdf - EDPB Guidelines 07/2020 controller/processor v2.1 [raw]: https://www.edpb.europa.eu/system/files/2023-10/edpb_guidelines_202007_controllerprocessor_final_en.pdf - EDPB news on KASPR: https://www.edpb.europa.eu/news/news/2025/data-scraping-french-sa-fined-kaspr-eu200-000_en - EDPB news listing (headline fines, Oct 2026): https://www.edpb.europa.eu/news/news/2025/data-scraping-french-sa-fined-kaspr-eu240-000_en (returned the news listing page) - CJEU press release 106/25, Latombe v Commission T-553/23 [raw]: https://curia.europa.eu/jcms/upload/docs/application/pdf/2025-09/cp250106en.pdf - EU AI Act Art. 5 (text mirror): https://artificialintelligenceact.eu/article/5/ - Garante, Provvedimento 542 of 14 Jul 2026 (page with removal notice): https://www.garanteprivacy.it/home/docweb/-/docweb-display/docweb/10275035 - Garante, OpenAI press release 20 Dec 2024 (with Court of Rome note): https://www.garanteprivacy.it/home/docweb/-/docweb-display/docweb/10085432 - Lusha coverage: https://www.luxgap.com/articles/garante-vs-lusha-amende-2m-data-enrichment-rgpd/?lang=en ; https://captaincompliance.com/education/italian-garante-hits-us-data-broker-lusha-with-e2-million-fine-and-processing-ban/ (secondary) - ICO, organisations using marketing services of data brokers: https://ico.org.uk/for-organisations/direct-marketing-and-privacy-and-electronic-communications/organisations-using-marketing-services-of-data-brokers/ - ICO, B2B marketing: https://ico.org.uk/for-organisations/direct-marketing-and-privacy-and-electronic-communications/business-to-business-marketing/ - ICO legitimate interests guidance (update notice 23 Mar 2026): https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/lawful-basis/legitimate-interests/ - ICO generative AI consultation response page: https://ico.org.uk/about-the-ico/what-we-do/our-work-on-artificial-intelligence/response-to-the-consultation-series-on-generative-ai/ - Data (Use and Access) Act 2025 s.70 (enacted text): https://www.legislation.gov.uk/ukpga/2025/18/section/70/enacted - Experian tribunal commentary: https://www.reedsmith.com/our-insights/blogs/viewpoints/102i8l8/a-sigh-of-relief-for-data-brokers-tribunal-overturns-most-of-icos-experian-en/ ; https://www.stephensonharwood.com/insights/experians-win-against-the-ico-in-upper-tribunal/ - PIPEDA s.4.01: https://laws-lois.justice.gc.ca/eng/acts/P-8.6/section-4.01.html - Regulations Specifying Publicly Available Information SOR/2001-7: https://laws-lois.justice.gc.ca/eng/regulations/SOR-2001-7/FullText.html - OPC draft third-party service provider guidance (Bennett Jones note): https://www.bennettjones.com/Insights/Blogs/2026/09/OPC-Issues-New-PIPEDA-Guidance-on-Assessing-Third-Party-Service-Providers - BLG note on public Facebook profiles (OPC): https://www.blg.com/en/insights/2018/07/public-facebook-profiles-not-equivalent-to-public-information-under-pipeda (search summary) - Quebec Law 25 commentary (search summaries): https://www.mccarthy.ca/en/insights/blogs/techlex/key-operational-considerations-ahead-major-law-25-entry-force-phase ; https://www.clym.io/blog/quebec-law-25 - OAIC APP guidelines Ch. 3: https://www.oaic.gov.au/privacy/australian-privacy-principles/australian-privacy-principles-guidelines/chapter-3-app-3-collection-of-solicited-personal-information - OAIC APP guidelines Ch. 7: https://www.oaic.gov.au/privacy/australian-privacy-principles/australian-privacy-principles-guidelines/chapter-7-app-7-direct-marketing - Australian 2024 reforms (search summaries): https://www.privacyworld.blog/2024/12/first-tranche-of-reforms-to-australian-privacy-law-passed-with-amendments/ #### 15.2 United States - Cal. Civ. Code 1798.99.80 (mirror): https://california.public.law/codes/ca_civ_code_section_1798.99.80 - Cal. Civ. Code 1798.140 (mirror): https://california.public.law/codes/ca_civ_code_section_1798.140 - 11 CCR 7050: https://www.law.cornell.edu/regulations/california/11-CCR-7050 - 11 CCR 7051: https://www.law.cornell.edu/regulations/california/11-CCR-7051 - CPPA data broker/DROP page: https://www.cppa.ca.gov/data_brokers/ ; registry page https://cppa.ca.gov/data_broker_registry/ ; announcement https://cppa.ca.gov/announcements/2025/20251113.html - Vermont 9 V.S.A. 2430: https://legislature.vermont.gov/statutes/section/09/062/02430 ; Vermont SOS page (404 on one URL; search summary of https://sos.vermont.gov/corporations/other-services/data-brokers/) - Texas Bus. & Com. Code ch. 509 (mirror, possibly stale): https://texas.public.law/statutes/tex._bus._&_com._code_section_509.001 ; WilmerHale on SB 2121/SB 1343: https://www.wilmerhale.com/en/insights/blogs/wilmerhale-privacy-and-cybersecurity-law/20250904-texas-expands-and-modifies-data-broker-registration-law - Oregon HB 2052 (ZwillGen note): https://www.zwillgen.com/privacy/oregon-data-broker-registration/ - State nonprofit exemptions: https://wiley.law/alert-New-State-Privacy-Laws-May-Apply-to-Nonprofit-Organizations ; https://privacylawmap.com/blog/state-privacy-laws-nonprofit-organizations-guide - FTC CAN-SPAM guide: https://www.ftc.gov/business-guidance/resources/can-spam-act-compliance-guide-business - TCPA 11th Cir.: https://www.kelleydrye.com/viewpoints/blogs/ad-law-access/eleventh-circuit-vacates-tcpa-11-consent-rule - FTC data broker actions: https://www.wilmerhale.com/en/insights/blogs/wilmerhale-privacy-and-cybersecurity-law/20241216-ftc-continues-to-bring-enforcement-actions-against-data-brokers ; Hunton (search summary) https://hunton.com/privacy-and-information-security-law/ftc-finalizes-orders-against-data-brokers-over-sensitive-location-data - BIPA commentary (search summaries): https://ktslaw.com/insights/alert/2024/5/illinois-legislature-passes-bill-amending-bipa-violation-accrual-standards ; https://blog.ericgoldman.org/archives/2018/12/google-photos-defeats-privacy-lawsuit-over-face-scans-rivera-v-google.htm ; https://www.orrick.com/en/Insights/2019/01/emRivera-v-Googleem-Bolsters-Article-III-Challenges-to-Privacy-Suits--But-Risks-Remain - Clearview BIPA settlement (search summaries): https://www.isba.org/node/98222 ; https://storage.courtlistener.com/pdf/2026/07/20/robert_weissman_v._clearview_ai_inc..pdf - NY Civil Rights Law s.50: https://www.nysenate.gov/legislation/laws/CVR/50 - hiQ/LinkedIn: https://www.proskauer.com/blog/hiq-and-linkedin-reach-proposed-settlement-in-landmark-scraping-case - Proxycurl: https://www.socialmediatoday.com/news/linkedin-wins-legal-case-data-scrapers-proxycurl/756101/ ; ProAPIs: https://therecord.media/linkedin-sues-data-scraping-company ; search summaries on ProAPIs consent judgment and Apollo/Seamless page restrictions (https://www.gblock.app/articles/linkedin-proapis-scraping-injunction-2026, https://news.bloomberglaw.com/artificial-intelligence/linkedin-strikes-deal-to-settle-scraping-battle-with-startup) #### 15.3 Providers - Apollo ToS (last updated 10 Aug 2026): https://www.apollo.io/terms ; Privacy Policy: https://www.apollo.io/privacy-policy - ZoomInfo License Terms (Version May 1, 2025): https://www.zoominfo.com/legal/ltc ; 2021 version: https://www.zoominfo.com/legal/ltc_12_20_2021 ; legal index: https://www.zoominfo.com/legal - FullContact Services Agreement (effective 29 Jul 2026): https://www.fullcontact.com/privacy/services-agreement/ (prior-version page found via search: https://www.fullcontact.com/privacy/services-agreement-previous-version/) - People Data Labs: privacy and security overview https://www.peopledatalabs.com/pdf/privacy-security-overview.pdf ; data-license docs https://docs.peopledatalabs.com/docs/data-license ; Primer third-party terms (reproduces PDL terms) https://www.sayprimer.com/third-party-terms (Reported) - HubSpot Product Specific Terms (Last Modified 16 Sep 2026): https://legal.hubspot.com/product-specific-terms ; enrichment help: https://knowledge.hubspot.com/records/get-started-with-data-enrichment ; Clearbit Logo API sunset (search summary): https://developers.hubspot.com/changelog/upcoming-sunset-of-clearbits-free-logo-api - Hunter ToS (23 May 2024): https://www.hunter.io/terms ; DPA: https://www.hunter.io/data-processing-agreement - Clay ToS: https://www.clay.com/terms-of-service - Brandfetch GTC: https://brandfetch.com/terms - Gravatar/Automattic: https://wordpress.com/tos/ (last updated 6 Oct 2026) ; https://automattic.com/privacy/ ; https://docs.gravatar.com/api/profiles/ ; https://docs.gravatar.com/api/profiles/rest-api/ ; https://developer.wordpress.com/guidelines/ ; 2013 clause discussion https://lists.openstreetmap.org/pipermail/rails-dev/2013-November/004032.html (search summary) - Crunchbase license agreement v3.1: https://data.crunchbase.com/v3.1/docs/license-agreement - D&B Terms and Conditions: https://solutions.dnb.com/html/EN_LnL_TermsConditions.html - LinkedIn User Agreement (effective 3 Nov 2025): https://www.linkedin.com/legal/user-agreement --- ### 16. Pages and topics I could not fetch or research (and what that leaves open) Fetch failures: - CNIL pages (KASPR sanction, data-broker tag pages): HTTP 503 on three attempts. KASPR amount (EUR 200k vs 240k) unresolved; Solocal/Caloga/Hubside/Foriou details rest on a search summary. - Illinois General Assembly (BIPA text): "Access Denied" (also certificate error). The photographs exclusion is therefore Inferred, not Verified. - California leginfo and Justia (403): statutes read from california.public.law and Cornell LII mirrors instead. - FTC press release and data-broker guidance pages: 404; FTC enforcement taken from WilmerHale (Reported). No verified 2025-2026 FTC data-broker actions. - People Data Labs' own Terms/ADUP/License pages (peopledatalabs.com/terms-of-service 404; privacy.peopledatalabs.com renders empty; peopledatalabs.com/legal, /opt-out 404). PDL's license restrictions rest on Primer's reproduction (Reported). - Cognism customer terms (cognism.com/terms, /legal 404; only website terms read). - Logo.dev terms (404); Brandfetch Logo Fair Use guide not read. - ZoomInfo Controller-to-Controller DPA and C2P DPA PDFs (links found but not read). - Gravatar Responsible Use page (docs.gravatar.com/rest-api/responsible-use 404; read https://developer.wordpress.com/guidelines/ instead, which I believe is the same guideline). - CASL: fightspam.gc.ca (404); OPC PIPEDA principles pages (404); EDPB DPF information note (404); EDPB KASPR "240" URL returned a news listing. - Australia: Privacy Act s.6D text (page showed only a table of contents). - Quebec statute and CAI guidance (only law-firm and vendor commentary). - ICO subpages on legitimate interests/direct marketing and on generative AI web-scraping outcomes (index pages only). Not researched (search budget exhausted or not reached): - Mantheos and other 2025-2026 scraping cases; Meta v. Bright Data; X Corp v. Bright Data. - AtData/TowerData, Versium, Melissa, Experian append terms and posture. - Which specific providers source from LinkedIn (no first-hand statement found; see Section 7.3). - Dutch AP/other EU data-broker decisions beyond the headlines above; EDPB's final (non-draft) LI guidelines status; status of any appeal in Latombe. - CCPA ADMT/risk-assessment regulations (effective 2026) and their application to profiling features. - UK-US data bridge status; EU member-state image/personality rights; Illinois Right of Publicity Act; California Civil Code 3344. - CJEU SCHUFA (C-634/21) text (cited from memory only). Quality notes on the evidence: - Several "Verified" provider terms came through a summarising fetch tool. Quotations and section numbers should be checked against the live pages before they are copied into any customer contract or provider negotiation. - Dates for provider terms are those the pages showed; I did not archive copies. Take screenshots/PDF copies of each provider's terms when the proposal moves to contracts. - This is research, not legal advice. Counsel should review at least: Art. 9 handling for religious/political/union tenants, the cross-tenant-cache conclusion, data-broker registration exposure in CA/VT/TX/OR (especially Vermont's 2027 text), the CASL position, and any OEM agreement with a provider. ## 27. W5. Supplementary provider checks (read directly, 2026-10-22, 11:00 to 11:30 ET) ## W5. Supplementary provider checks (read directly, 2026-10-22, 11:00 to 11:30 ET) The four market reports (w1 to w4) ran short of web-search budget before they could read several provider pages. This note records what the architect then read directly, so every claim in the chapters that rests on it has a source. Labels: **Verified** (read on the vendor's page at the URL, on 2026-10-22), **Reported** (a third party's account), **Unavailable** (the page could not be read: 404, script-rendered or login). ### 1. People Data Labs (PDL) | Item | Finding | Label | Source | | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Person Data pricing | Free: "Up to 100 credits/mo", contact data not included (e-mails, phones, addresses obfuscated). Pro: "Starting at $98/mo", "as few as 350 credits/mo", monthly or annual. Enterprise: custom, annual. Annual: "Save 20%". | Verified | https://www.peopledatalabs.com/pricing | | Per-record cost | Not stated on the page; $98 ÷ 350 = $0.28 per credit is arithmetic (a credit is one matched enrichment; a 404 "no match" is not charged per w3 §2.1). | Derived | same | | Company pricing | On a separate page (`/pricing/company`), not read. | Unavailable | | | Job-change detection | No dedicated product page found. What exists is the **Person Changelog API**: "updates, additions, deletions, merges, and opt-outs for individuals between specified versions", filterable by ids, request type and `fields_updated` (the examples page filters on `job_title`, `work_email`, `mobile_phone`); versions must be consecutive monthly or quarterly releases; from v25.0 the flat-file changelog "will only include records with high-signal changes". PDL **webhooks** for person updates are "only available for our subscribing enterprise API and Data License customers". | Verified (docs pages, via search summaries of docs.peopledatalabs.com) | https://docs.peopledatalabs.com/docs/person-changelog-api ; https://docs.peopledatalabs.com/docs/examples-person-changelog-api ; https://docs.peopledatalabs.com/docs/webhooks | | Terms of service / Services Subscription Agreement | The site's footer link "Terms of Use" points to `https://privacy.peopledatalabs.com/policies?name=services-subscription-agreement`; that page renders through a script and returned an empty body to both the fetch tool and the in-app browser. `/terms`, `/legal/terms`, `/legal/terms-of-service`, `/terms-of-service` and `/legal` all answer 404. PDL's own resale, retention and display terms therefore remain **unread**; the only text available is a reseller's reproduction (w4 §5). | Unavailable | https://privacy.peopledatalabs.com/policies?name=services-subscription-agreement | ### 2. FullContact (Ziff Davis) | Item | Finding | Label | Source | | -------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | | Person enrich inputs | `email` (clear text, MD5 or SHA-256), `emails`, `phone`, `phones`, `profiles` (social URLs, ids, usernames), `name.full` or `name.given` + `name.family` **only with a postal address**, `location.*` **only with a name**, `maids`, `recordId`, `personId`, `li_nonid`. Minimum: an e-mail, a phone, a profile, a MAID, or name plus one of (addressLine1 + city + region/regionCode) or (addressLine1 + postalCode). "There is a maximum of 25 identifiers by type", "Only one name/address combination can be included". With a hashed e-mail "the following PII fields will no longer be returned" (name, clear e-mail, phone, social handles). No confidence score documented. | Verified | https://docs.fullcontact.com/docs/multi-field-request | | Person enrich response (getting started) | Sample response top level: `fullName`, `ageRange`, `gender`, `location`, `title`, `organization`, `twitter`, `bio`, `avatar`, `website`, plus `details`. Endpoint `POST https://api.fullcontact.com/v3/person.enrich`. Batch enrichment "by contacting sales". | Verified | https://docs.fullcontact.com/docs/getting-started-with-enrich-1 | | Insights Bundles | Bundles appear "as a key inside of the `details` property of a Person Summary object"; a match "starts with the notion of an identity" resolved to a persistent Person ID. Identity graph "240M U.S. profiles", "rebuilt biweekly", consumer data "updated quarterly". Sources listed include public records, voter registrations, property, surveys, purchases, card transactions, social affinities, professional and employment data. | Verified (marketing-style docs page) | https://docs.fullcontact.com/docs/enrich-overview ; https://docs.fullcontact.com/docs/person-insights-bundles | | Services Agreement (previous version; the current one, `?p=21102`, answered 404) | §2(b): licence is "limited, non-exclusive, non-transferable, non-sublicensable"; at term end "renew the license ... or delete". §1(d), §1(g), §3 and Exhibit A: data may reach "Third Party End Users" (the client's customers) **only through a "Customer Application" named in an Order Form**, which "must provide significantly additional data, content and functionality than that provided by the FullContact Data alone", with a written end-user agreement and flow-down restrictions; Exhibit B binds "Fourth Party End Users" and bars resale or redistribution. §7(c) and Exhibit C: delete all FullContact Data within 30 days of termination and sign a deletion acknowledgment; cause end users to delete. §7(e): keep records three years; provide SQL logs on request; pay pro-rata fees for data kept past deletion. §2(d)(1)(vi): no use "in connection with any software or services that incorporates a large language model, machine learning", including training, fine-tuning or embedding. §2(d)(1)(i)-(ii): no eligibility decisions (employment, credit, insurance, housing). §4(a): FullContact may use Submitted Information to improve its algorithms. §4(b): client keeps a compliant privacy policy and holds the necessary consents. §2(d)(2): FullContact may impose Additional Restrictions at any time. No photo-specific clause, no reseller clause, no CCPA service-provider clause in the extract. | Verified (previous version; current version unread) | https://www.fullcontact.com/privacy/services-agreement-previous-version/ | ### 3. Clearbit and HubSpot Breeze Intelligence | Item | Finding | Label | Source | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Standalone Clearbit API | Reported discontinued for new customers: Clearbit's help centre "no longer offering new standalone Clearbit subscriptions"; API keys only for accounts created in 2023 or earlier. A HubSpot Community reply (closest to first party) says "there are not APIs available in HubSpot to replace the Clearbit Company and Logo API"; Breeze Intelligence is a HubSpot feature, not an API. HubSpot recommends Logo.dev as the Logo API replacement. Free Logo API ended 1 December 2025 (one source says 8 December). | Reported | https://www.cleanlist.ai/blog/2026-10-16-clearbit-api-guide ; https://community.hubspot.com/t/the-future-of-clearbit-s-free-tools/125894 ; https://bitscale.ai/blogs/breeze-intelligence-review | | Breeze pricing | About $45 per month for 100 credits on annual billing, on top of a HubSpot subscription. | Reported | https://www.zeliq.com/blog/hubspot-data-enrichment | ### 4. Logo.dev | Item | Finding | Label | Source | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- | ---------------------------- | | Plans | Community: free, 500,000 Logo CDN requests/month, $5 of included API credits, attribution required "only on the free plan, and only for commercial projects". Startup: $33/month, 1,000,000 CDN requests, $15 API credits. Pro: $150/month, 5,000,000 CDN requests, $150 API credits. Enterprise: custom, "Custom legal terms". | Verified | https://www.logo.dev/pricing | | Storage | "On every plan, you can store anything the Logo.dev API returns, logo files included"; "cache API results and host the returned assets yourself". | Verified | same | | Overage | Free plan stops at its CDN limit; paid plans "keep serving, and we get in touch if you stay over"; "We never bill you for overage automatically". | Verified | same | | Multi-tenant use | No restriction on the pricing page; the Terms of Service (`/legal/terms`) were not read. | Unavailable | | ### 5. Gravatar (Automattic) | Item | Finding | Label | Source | | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------------------------------------------------------------------------------------ | | Avatar request | `https://gravatar.com/avatar/HASH`, HASH = SHA-256 of the e-mail; `d=404` returns HTTP 404 when no image exists (so "has this person a Gravatar" costs one HEAD request and stores nothing); `s=` 1 to 2048 px; `r=` g/pg/r/x, G only by default; `f=y` forces the default. No API key stated for avatar requests. No caching statement on the page. | Verified | https://docs.gravatar.com/api/avatars/images/ | | Terms | "Use of our free APIs is governed by the Guidelines for Responsible Use"; the Terms of Service link resolves to WordPress.com's ToS. The Guidelines name the Gravatar API in scope; rules: "refresh any data that you store at reasonable intervals"; do not "place excessive API calls"; "Use of our API is free"; the app "should include a privacy policy that discloses, in clear terms" what it does; no use "to governmental entities for intelligence gathering or surveillance purposes". No numeric rate limit, no attribution rule, no explicit ban on CRM display. | Verified | https://docs.gravatar.com/api/avatars/ ; https://developer.wordpress.com/guidelines/ | ### 6. What this changes in the chapters - PDL remains the premium person candidate on documented capability and price, but **its own licence text is unread**; chapter 12 treats the OEM question as open and chapter 15 makes obtaining written terms (PDL, FullContact, Logo.dev Enterprise) a prerequisite of any Rhythm-managed tier (ENR-D4). - FullContact's agreement is the one text that spells out how an application may pass data to its own customers (Customer Application, Third Party End Users, Exhibit A); it also bans any LLM or machine-learning use of its data outright, which rules it out as a source for AI features (chapter 09 §5). - Clearbit is not a candidate; Breeze is HubSpot-internal (chapter 03). - Logo.dev's "store anything returned, on every plan" makes employer logos the cheapest safe baseline (chapter 12 §2), pending its full terms. - Gravatar's `d=404` probe lets the baseline tier learn "has a public avatar" without storing a provider payload (chapter 06 §2, chapter 09 §4).