--- id: WF-PFM-INT-015 title: Receive and process a provider notification trigger: The initiating provider starts this flow. product_area: platform module: integrations revision: 1 status: draft design_intent: proposed implementation_evidence: observed inference: [] source_sha: services-platform: ad3f6a6582843008ed0f5a17b131878d15829af1 implements: - CAP-PFM-INT-013 - CAP-PFM-INT-015 moves: - STATE-PFM-INT-009 - STATE-PFM-INT-010 crosses: [] idempotency: >- Worker processing recognizes provider identifiers and one-day body digests; explicit replay bypasses those keys. concurrency: >- Processing claims numbered message attempts and settles generated events with message status in a transaction. hops: - 'n': 1 actor_or_service: Provider action: Accept a provider request at its endpoint token address. anchor: 'services-platform:modules/integrations/aws/api/webhooks.routes.ts' seam: sync on_failure: The provider controls retries after request failure. left_inconsistent: No Rhythm record is guaranteed before ingress succeeds. observer: Provider dashboard provenance: observed source: 'services-platform:modules/integrations/aws/api/webhooks.routes.ts' - 'n': 2 actor_or_service: Webhook ingress action: 'Locate an enabled, provisioned endpoint, enforce body size and attempt the per-endpoint rate permit.' anchor: 'services-platform:modules/integrations/services/webhooks/ingress.ts:receiveWebhook' seam: sync on_failure: Reject missing endpoints or oversized input; permit-store failure admits the request. left_inconsistent: Provider retries can generate another ingress request. observer: Webhook metrics provenance: observed source: 'services-platform:modules/integrations/services/webhooks/ingress.ts:receiveWebhook' - 'n': 3 actor_or_service: Webhook ingress action: >- Save body and selected headers, commit the message and dispatch work, then answer receipt or a declared inline challenge. anchor: 'services-platform:modules/integrations/services/webhooks/ingress.ts:receiveWebhook' seam: sync on_failure: Return a storage or record error before receipt acknowledgement. left_inconsistent: Successfully stored objects can remain orphaned until expiry. observer: Webhook metrics provenance: observed source: 'services-platform:modules/integrations/services/webhooks/ingress.ts:receiveWebhook' - 'n': 4 actor_or_service: Message work dispatcher action: Publish pending processing work to the webhook queue. anchor: 'services-platform:modules/integrations/services/work/work-dispatcher.ts:dispatchWork' seam: queue on_failure: Retry publication from the retained dispatch row. left_inconsistent: A crash between publication and deletion can duplicate queue delivery. observer: Dispatcher silence alarm provenance: observed source: 'services-platform:modules/integrations/services/work/work-dispatcher.ts:dispatchWork' - 'n': 5 actor_or_service: Webhook worker action: Lease signing secrets and ask the adapter to verify and interpret the stored request. anchor: 'services-platform:modules/integrations/services/webhooks/webhook-worker.ts:processWebhookEvent' seam: queue on_failure: Retry classified transient failures up to six attempts and stop permanent failures immediately. left_inconsistent: The received message remains inspectable with its failure outcome. observer: Webhooks screen and dead-webhook alarm provenance: observed source: 'services-platform:modules/integrations/services/webhooks/webhook-worker.ts:processWebhookEvent' - 'n': 6 actor_or_service: Webhook worker action: >- Deduplicate interpreted input, resolve supported subject context and atomically save events with routing work and processed status. anchor: 'services-platform:modules/integrations/services/events/complete-event.ts:resolveEventSubject' seam: sync on_failure: Rollback failed event recording and leave processing to recovery. left_inconsistent: No subset of the interpreted events commits alone. observer: Webhooks and events screens provenance: observed source: 'services-platform:modules/integrations/services/events/complete-event.ts:resolveEventSubject' - 'n': 7 actor_or_service: Executions operator action: >- Replay a retained message or endpoint window of no more than seven days into new replay records that bypass duplicate checks. anchor: 'services-platform:modules/integrations/services/events/event-operations.ts:replayWebhookEndpoint' seam: sync on_failure: A replay whose stored body expired ends dead. left_inconsistent: Replayed events can reach consumers that handled earlier processing. observer: Integration history provenance: documented source: 'services-platform:docs/integrations/README.md#Webhooks (WS8, 2026-10-18)' evidence: - >- services-platform at ad3f6a65: modules/integrations/aws/api/webhooks.routes.ts; services/webhooks/ingress.ts, webhook-worker.ts; services/work/work-dispatcher.ts; services/events/complete-event.ts, event-operations.ts; docs/integrations/README.md (Webhooks; Deduplication) - >- services-platform:modules/integrations/tests/integration/services/webhooks/ingress.integration.spec.js#answers 404 for a disabled endpoint, and records nothing - >- services-platform:modules/integrations/tests/integration/services/webhooks/ingress.integration.spec.js#answers 404 for a disabled endpoint, and records nothing - >- services-platform:modules/integrations/tests/integration/services/webhooks/ingress.integration.spec.js#stores the body and headers, then records the event pending and queued with its digest and a queue row - >- services-platform:modules/integrations/tests/integration/services/webhooks/webhook-worker.integration.spec.js#rejects a request whose signature does not verify, writing no event - >- services-platform:modules/integrations/tests/unit/services/webhook-rules.spec.ts#names the Rhythm record of the execution that created the provider object controls: [TC-AVL-001, TC-AVL-004, TC-AVL-005, TC-AZ-001, TC-AZ-002, TC-AZ-003, TC-AZ-004, TC-AZ-005, TC-CHG-001, TC-CHG-002, TC-CHG-003, TC-CHG-004, TC-CHG-005, TC-CHG-006, TC-CRY-001, TC-CRY-002, TC-CRY-003, TC-CRY-004, TC-DAT-001, TC-DAT-002, TC-DAT-003, TC-DAT-004, TC-IAM-001, TC-IAM-002, TC-IAM-003, TC-IAM-004, TC-IAM-005, TC-IAM-006, TC-LOG-001, TC-LOG-002, TC-LOG-003, TC-LOG-004, TC-LOG-005, TC-MON-001, TC-MON-002, TC-PI-001, TC-PI-002, TC-PI-003, TC-PI-004, TC-PI-005, TC-PRV-001, TC-PRV-002, TC-SEC-001, TC-SEC-002, TC-SEC-003, TC-SEC-004, TC-TEN-001, TC-TEN-002, TC-TEN-003, TC-TEN-004, TC-VEN-001, TC-VEN-002, TC-VEN-003] --- # Receive and process a provider notification ## What It Does Ingress retains provider material before acknowledgement, then queues signature verification and interpretation. Accepted messages produce typed events for applications; duplicate or rejected messages remain inspectable. Replay creates separate message records from retained bodies. ## The Flow ```mermaid sequenceDiagram participant P0 as Provider participant P1 as Webhook ingress participant P2 as Message work dispatcher participant P3 as Webhook worker participant P4 as Executions operator P0->>P0: 1. Accept a provider request at its endpoint token address. P1->>P1: 2. Locate an enabled P1->>P1: 3. Save body and selected headers P2->>P2: 4. Publish pending processing work to the webhook queue. P3->>P3: 5. Lease signing secrets and ask the adapter to verify and interpret the stored request. P3->>P3: 6. Deduplicate interpreted input P4->>P4: 7. Replay a retained message or endpoint window of no more than seven days into new replay records that bypass du ``` ## When Something Fails The provider controls retries after request failure. No Rhythm record is guaranteed before ingress succeeds. Reject missing endpoints or oversized input; permit-store failure admits the request. Provider retries can generate another ingress request. Return a storage or record error before receipt acknowledgement. Successfully stored objects can remain orphaned until expiry. Retry publication from the retained dispatch row. A crash between publication and deletion can duplicate queue delivery. Retry classified transient failures up to six attempts and stop permanent failures immediately. The received message remains inspectable with its failure outcome. Rollback failed event recording and leave processing to recovery. No subset of the interpreted events commits alone. A replay whose stored body expired ends dead. Replayed events can reach consumers that handled earlier processing. ## Open Design Questions Product still needs an accepted message milestone and an explicit replay-consumer contract. Stored-body expiry limits recovery.