--- id: WF-PFM-INT-012 title: Run an asynchronous call through the work queue trigger: The initiating gateway starts this flow. product_area: platform module: integrations revision: 1 status: draft design_intent: confirmed implementation_evidence: observed origin: 'docs/milestones/INTG-M2-calls-through-the-gateway.md#Decisions M2-D3, M2-D6 (Andrew)' inference: [] source_sha: services-platform: ad3f6a6582843008ed0f5a17b131878d15829af1 implements: - CAP-PFM-INT-009 - CAP-PFM-INT-010 moves: - STATE-PFM-INT-007 crosses: [] idempotency: >- Obsolete or duplicated queue work cannot start an already claimed attempt; completion-event creation follows the winning settlement. concurrency: >- Attempt startup atomically compares status and attempt number, preventing a duplicate message from starting the same attempt. hops: - 'n': 1 actor_or_service: Gateway action: 'Commit accepted execution and dispatch work, answer acceptance and request a dispatcher nudge.' anchor: 'services-platform:modules/integrations/services/executions/invoke-capability.ts:invokeCapability' seam: sync on_failure: Database failure rejects acceptance; a lost nudge is recovered by the schedule. left_inconsistent: Committed dispatch work can wait until the next scheduled pass. observer: The caller provenance: observed source: 'services-platform:modules/integrations/services/executions/invoke-capability.ts:invokeCapability' - 'n': 2 actor_or_service: Execution dispatcher action: >- Publish due dispatch records to the execution queue, applying bounded queue delays and removing accepted dispatch rows. anchor: 'services-platform:modules/integrations/services/executions/dispatcher.ts:dispatchExecutions' seam: queue on_failure: Leave failed publications for retry after their claim expires. left_inconsistent: Publication followed by failed deletion can send the same work again. observer: Dispatcher silence alarm provenance: observed source: 'services-platform:modules/integrations/services/executions/dispatcher.ts:dispatchExecutions' - 'n': 3 actor_or_service: Execution consumer action: 'Continue admitted work on the asynchronous retry schedule, creating another dispatch record when needed.' anchor: 'services-platform:modules/integrations/services/executions/dispatcher.ts:consumeExecutionMessage' seam: queue on_failure: Skip obsolete messages; retries beyond the attempt ceiling settle as failure. left_inconsistent: Later continuation can remain queued during an outage. observer: Activity provenance: observed source: 'services-platform:modules/integrations/services/executions/dispatcher.ts:consumeExecutionMessage' - 'n': 4 actor_or_service: Adapter action: >- Let the adapter renew leases, report progress and complete long-running work through its callback contract. anchor: 'services-platform:modules/integrations/services/adapter-callbacks/adapter-callbacks.ts:completeExecution' seam: sync on_failure: 'Refuse invalid, foreign or no-longer-applicable callbacks.' left_inconsistent: Unsettled work can remain in flight until timeout investigation. observer: Activity provenance: observed source: 'services-platform:modules/integrations/services/adapter-callbacks/adapter-callbacks.ts:completeExecution' - 'n': 5 actor_or_service: Gateway action: Commit final outcome and its completion event only when the settlement comparison succeeds. anchor: 'services-platform:modules/integrations/services/executions/settle-execution.ts:settleExecution' seam: sync on_failure: A losing writer changes no outcome and creates no completion. left_inconsistent: A failed transaction retains the previous execution status. observer: The invoking application provenance: observed source: 'services-platform:modules/integrations/services/executions/settle-execution.ts:settleExecution' - 'n': 6 actor_or_service: Integration event router action: Route completion to the initiating application’s registered consumer. anchor: 'services-platform:modules/integrations/services/events/event-router.ts:routeIntegrationEvents' seam: queue on_failure: 'Retry failed routing with queued backoff, ending dead after six attempts.' left_inconsistent: The application can remain unaware of a settled provider operation. observer: Rhythm HQ event list provenance: observed source: 'services-platform:modules/integrations/services/events/event-router.ts:routeIntegrationEvents' waits: - 'Queued retry delays of one, five and thirty minutes, two and six hours' - The asynchronous budget of a day unless the release declares less evidence: - >- services-platform at ad3f6a65: modules/integrations/services/executions/dispatcher.ts, attempt-loop.ts, settle-execution.ts, timeouts.ts; services/adapter-callbacks/adapter-callbacks.ts; services/events/execution-completed.ts; docs/queued-work.md - >- services-platform:modules/integrations/tests/integration/services/executions/invoke-capability.integration.spec.js#answers 202 accepted and queues the first attempt with the execution, calling no adapter - >- services-platform:modules/integrations/tests/integration/services/executions/invoke-capability.integration.spec.js#answers 202 accepted and queues the first attempt with the execution, calling no adapter - >- services-platform:modules/integrations/tests/integration/services/executions/execution-settlement.integration.spec.js#settles an execution whose next attempt is the eleventh failed, with the last class and retry_exhausted.attempt_limit - >- services-platform:modules/integrations/tests/integration/services/adapter-callbacks/adapter-callbacks.integration.spec.js#refuses a sync execution, which only the gateway settles - >- services-platform:modules/integrations/tests/integration/services/executions/execution-settlement.integration.spec.js#is written once for two settlements that both read the execution running, the loser moving nothing - >- services-platform:modules/integrations/tests/e2e-adapters/events-dummy.adapters.spec.js#reaches the consumer of the application that invoked it, carrying the response controls: [TC-AVL-001, TC-AVL-004, TC-AVL-005, TC-AZ-001, TC-AZ-002, TC-AZ-003, TC-AZ-004, TC-CRY-001, TC-CRY-002, TC-CRY-003, TC-CRY-004, TC-DAT-001, TC-DAT-002, TC-DAT-003, TC-DAT-004, TC-IAM-001, TC-IAM-002, TC-IAM-004, TC-IAM-005, TC-IAM-006, TC-LOG-001, TC-LOG-002, TC-LOG-003, TC-LOG-004, TC-LOG-005, TC-MON-001, TC-MON-002, TC-PI-001, TC-PI-002, TC-PI-003, TC-PI-004, TC-PI-005, TC-PRV-001, TC-PRV-002, TC-SEC-001, TC-SEC-002, TC-SEC-003, TC-SEC-004, TC-TEN-001, TC-TEN-002, TC-TEN-003, TC-VEN-001, TC-VEN-002, TC-VEN-003] --- # Run an asynchronous call through the work queue ## What It Does Asynchronous execution commits work before reporting acceptance. Dispatchers transfer retained work to queues and consumers resume the attempt loop. Later completion writes the application’s notification in the same transaction as settlement, with routing handled separately. ## The Flow ```mermaid sequenceDiagram participant P0 as Gateway participant P1 as Execution dispatcher participant P2 as Execution consumer participant P3 as Adapter participant P4 as Integration event router P0->>P0: 1. Commit accepted execution and dispatch work P1->>P1: 2. Publish due dispatch records to the execution queue P2->>P2: 3. Continue admitted work on the asynchronous retry schedule P3->>P3: 4. Let the adapter renew leases P0->>P0: 5. Commit final outcome and its completion event only when the settlement comparison succeeds. P4->>P4: 6. Route completion to the initiating application’s registered consumer. ``` ## When Something Fails Database failure rejects acceptance; a lost nudge is recovered by the schedule. Committed dispatch work can wait until the next scheduled pass. Leave failed publications for retry after their claim expires. Publication followed by failed deletion can send the same work again. Skip obsolete messages; retries beyond the attempt ceiling settle as failure. Later continuation can remain queued during an outage. Refuse invalid, foreign or no-longer-applicable callbacks. Unsettled work can remain in flight until timeout investigation. A losing writer changes no outcome and creates no completion. A failed transaction retains the previous execution status. Retry failed routing with queued backoff, ending dead after six attempts. The application can remain unaware of a settled provider operation. ## Open Design Questions Completed provider work can precede successful event delivery. Receiver deduplication remains required even though completion creation is atomic.