--- id: CAP-PFM-INT-013 title: Retain and verify incoming provider notifications feature: FEAT-PFM-INT-04 product_area: platform module: integrations revision: 1 status: draft product_intent: proposed implementation_evidence: observed supersedes: - CAP-INT-EVT-001 - CAP-INT-EVT-002 - CAP-INT-EVT-003 - CAP-INT-EVT-004 - CAP-INT-EVT-005 - CAP-INT-EVT-006 - CAP-INT-EVT-017 - CAP-INT-LIB-015 - CAP-INT-SEC-009 - CAP-INT-SEC-010 - CAP-INT-ADP-012 source_sha: services-platform: ad3f6a6582843008ed0f5a17b131878d15829af1 react-console: baf6dae382d6f41dce976c7f848667ccc37e076f intent_sources: - 'services-platform:docs/milestones/INTG-M3-webhooks-events-deliveries.md#Objective' purpose: 'Rhythm preserves provider notifications, verifies their origin and supports controlled reprocessing.' outcomes: - Durable storage precedes notification acknowledgement. - Duplicate handling avoids repeated processing except during an explicit replay. - Operators can reprocess retained messages following a correction. concepts: - CON-PFM-INT-004 actors: - ACT-PFM-006 - ACT-PFM-021 - ACT-PFM-017 - ACT-PFM-018 surfaces: - api - job - console preconditions: [] postconditions: [] permissions: - key: platform.integrations.read where_checked: api scope: tenant - key: platform.integrations.manage where_checked: api scope: tenant - key: platform.integration-executions.operate where_checked: api scope: tenant rules: - id: BR-CAP-PFM-INT-013-01 proved_by: [ATS-PFM-INT-013-01, TST-PFM-INT-567] statement: >- Ingress accepts messages only for enabled provisioned endpoints, returns bodyless not found otherwise, refuses bodies over one megabyte and throttles each endpoint above fifty requests per second. kind: validation enforced_at: - api - service must_prove: true provenance: test source: >- services-platform:modules/integrations/tests/integration/services/webhooks/ingress.integration.spec.js#answers 404 for a disabled endpoint, and records nothing origin: none - id: BR-CAP-PFM-INT-013-02 proved_by: [ATS-PFM-INT-013-02, TST-PFM-INT-567, TST-PFM-INT-584] statement: >- Ingress stores body and headers before recording and queueing the notification or acknowledging receipt, omitting authorization and cookies unless verification requires them. kind: invariant enforced_at: - service must_prove: true provenance: test source: >- services-platform:modules/integrations/tests/integration/services/webhooks/ingress.integration.spec.js#stores the body and headers, then records the event pending and queued with its digest and a queue row origin: none - id: BR-CAP-PFM-INT-013-03 proved_by: [] statement: Registration-challenge endpoints verify and return the adapter’s challenge response inline. kind: policy enforced_at: - service must_prove: true provenance: test source: >- services-platform:modules/integrations/tests/integration/services/webhooks/ingress.integration.spec.js#answers a registration challenge inline with the adapter’s response, after verifying it origin: none - id: BR-CAP-PFM-INT-013-04 proved_by: [ATS-PFM-INT-013-03, ATS-PFM-INT-013-10, ATS-PFM-INT-013-11, ATS-PFM-INT-015-10, TST-PFM-INT-480, TST-PFM-INT-568] statement: >- The worker verifies stored notifications with endpoint secrets, rejects invalid signatures without events and queues events from verified messages; retryable failures follow six asynchronous attempts and permanent failures stop immediately. kind: lifecycle enforced_at: - service must_prove: true provenance: test source: >- services-platform:modules/integrations/tests/integration/services/webhooks/webhook-worker.integration.spec.js#rejects a request whose signature does not verify, writing no event origin: none - id: BR-CAP-PFM-INT-013-05 proved_by: [ATS-PFM-INT-013-04, ATS-PFM-INT-013-10, TST-PFM-INT-480, TST-PFM-INT-568] statement: >- Processed provider event identifiers remain duplicates permanently, body duplicates on the same endpoint remain duplicates for twenty-four hours when no event identifier exists, and explicit replay bypasses both checks. kind: invariant enforced_at: - service - database must_prove: true provenance: test source: >- services-platform:modules/integrations/tests/integration/services/webhooks/webhook-worker.integration.spec.js#ignores a second request with the provider event id of one already processed, pointing at the first origin: none - id: BR-CAP-PFM-INT-013-06 proved_by: [ATS-PFM-INT-004-15, ATS-PFM-INT-005-17, ATS-PFM-INT-013-05, ATS-PFM-INT-013-09, ATS-PFM-INT-013-12, ATS-PFM-INT-013-13, TST-PFM-INT-480, TST-PFM-INT-491] statement: >- Verification provisions declared endpoints with minted secrets, and manually registered endpoints expose their address and secret once to an authorized administrator. kind: policy enforced_at: - service - api must_prove: true provenance: test source: >- services-platform:modules/integrations/tests/e2e-adapters/events-dummy.adapters.spec.js#never answers the secret again origin: none - id: BR-CAP-PFM-INT-013-07 proved_by: [ATS-PFM-INT-013-06, ATS-PFM-INT-013-09, TST-PFM-INT-568] statement: >- Secret rotation accepts both old and new signing secrets for twenty-four hours and then accepts only the replacement. kind: lifecycle enforced_at: - service must_prove: true provenance: test source: >- services-platform:modules/integrations/tests/integration/services/webhooks/webhook-worker.integration.spec.js#verifies with the previous secret too while the endpoint rotates origin: none - id: BR-CAP-PFM-INT-013-08 proved_by: [ATS-PFM-INT-013-07] statement: >- Operators may replay an individual stored message or an endpoint window no longer than seven days, with each replay entering history. kind: policy enforced_at: - api - service must_prove: true provenance: documented source: 'services-platform:docs/integrations/README.md#Webhooks (WS8, 2026-10-18)' origin: none - id: BR-CAP-PFM-INT-013-09 proved_by: [ATS-PFM-INT-013-08, TST-PFM-INT-491] statement: Integration readers inspect endpoint and notification records without signing-secret values. kind: permission enforced_at: - api must_prove: true provenance: test source: >- services-platform:modules/integrations/tests/e2e/reader-access.e2e.spec.js#reads no webhook signing secret in any answer origin: 'services-platform:docs/milestones/INTG-M1-find-install-connect.md#D4' scenarios: - id: SC-CAP-PFM-INT-013-01 given: The test provider posts a payment-state notification. when: Rhythm verifies and processes the retained message. then: A payment update is routed to Accounting and acknowledged. provenance: test source: >- services-platform:modules/integrations/tests/e2e-adapters/events-dummy.adapters.spec.js#reaches Accounting’s consumer, which acknowledges it edge_cases: [] constraints: [] depends_on: - CAP-PFM-INT-004 related: - CAP-PFM-INT-015 journeys: [] open_questions: - OQ-PFM-INT-002 evidence: - >- services-platform source inventory: modules/integrations/services/webhooks/* (ingress.ts, webhook-worker.ts, webhook-registration.ts), aws/api/webhooks.routes.ts, events.routes.ts, aws/logic/integrations-events.lambda.ts; rule-level automated test references; docs/integrations/README.md (Webhooks, Deduplication); docs/milestones/INTG-M3-webhooks-events-deliveries.md (Proposed). - 'react-console source inventory: components/integrations/webhooks-screen.tsx.' - >- Operation coverage: receiveIntegrationWebhook, listIntegrationWebhookEndpoints, rotateIntegrationWebhookEndpoint, replayIntegrationWebhookEndpoint, listIntegrationWebhookEvents, getIntegrationWebhookEvent, replayIntegrationWebhookEvent. - >- Persistent record coverage: integration_webhook_endpoints, integration_webhook_events, integration_webhook_event_keys, integration_work_dispatches. - 'End-to-end definition inventory: events-dummy.adapters.spec.js, reader-access.e2e.spec.js.' controls: [TC-AVL-001, TC-AVL-004, TC-AZ-001, TC-AZ-002, TC-AZ-003, TC-AZ-004, TC-CRY-001, TC-CRY-002, TC-CRY-003, TC-CRY-004, TC-DAT-001, TC-DAT-002, TC-DAT-003, TC-IAM-001, TC-IAM-006, TC-LOG-002, TC-LOG-003, TC-LOG-004, TC-LOG-005, TC-MON-001, TC-MON-002, TC-PI-001, TC-PI-002, TC-PI-003, TC-PI-004, TC-PI-005, TC-SEC-001, TC-SEC-002, TC-SEC-003, TC-SEC-004, TC-TEN-001, TC-TEN-002, TC-TEN-003, TC-VEN-001, TC-VEN-002, TC-VEN-003] exercised_by: [FAS-PFM-INT-04-01, FAS-PFM-INT-04-02, FAS-PFM-INT-04-04] suites: [TST-PFM-INT-480, TST-PFM-INT-491, TST-PFM-INT-567, TST-PFM-INT-568, TST-PFM-INT-584] --- # Retain and verify incoming provider notifications ## Purpose Rhythm preserves provider notifications, verifies their origin and supports controlled reprocessing. ## How It Works A provider sends account notifications to an endpoint Rhythm provisions for the connection. Rhythm retains each message before acknowledging it. A worker asks the integration to verify and interpret the retained message. Accepted interpretations become events for application routing. Repeated provider sends are handled as duplicates. An operator can replay stored material after correcting a processing problem. Signing-secret setup and rotation support the endpoint’s authentication lifecycle. This machinery is implemented without an accepted webhook milestone. The product should approve its customer-facing recovery and secret-handling behaviour before design treats it as settled. ## Relationships - Requires maintain a verified provider account connection to supply the prerequisite configuration or operation. - Connects with route integration events to their application owners through shared integration records.