--- id: STATE-PFM-INT-007 title: Execution lifecycle object: DOM-PFM-INT-003 product_area: platform module: integrations revision: 1 status: draft design_intent: confirmed implementation_evidence: observed origin: >- docs/milestones/INTG-M2-calls-through-the-gateway.md#Decisions M2-D2, M2-D3, M2-D4, M2-D6, M2-D8 (Andrew); docs/integrations/README.md#The pipeline (13 §5) (Andrew 2026-10-19 decisions 1 to 3) source_sha: services-platform: ad3f6a6582843008ed0f5a17b131878d15829af1 macroservice-libraries: 54536a441935c6e09fdb7f96f0bec71c52ba4e03 states: - name: accepted meaning: The gateway retained the request before beginning an attempt. - name: running meaning: An admitted attempt is executing or the adapter is completing asynchronous work. - name: retrying meaning: Safe continuation awaits another attempt. - name: unknown meaning: Provider work may have happened and requires investigation. - name: needs_attention meaning: Automatic investigation stopped and an operator must record the result. - name: succeeded meaning: The saved final answer reports successful provider work. - name: failed meaning: The saved final answer reports classified failure. - name: cancelled meaning: The execution stopped without further work and its reusable key was released where applicable. transitions: - from: accepted to: running trigger: Begin provider delivery guard: 'Circuit, rate and credential admission succeeded' side_effects: - Create an attempt record before delivery. who: Attempt loop or queue consumer provenance: observed source: 'services-platform:modules/integrations/services/executions/attempt-loop.ts' - from: accepted to: failed trigger: Refuse provider admission guard: 'Circuit, rate or authentication prevents delivery' side_effects: - Release the unused request key and return retry or authorization guidance. who: Attempt loop provenance: test source: >- services-platform:modules/integrations/tests/integration/services/executions/invoke-capability.integration.spec.js#refuses with 503 provider_unavailable while the circuit is open, and releases the key - from: running to: succeeded trigger: Receive a valid successful answer guard: The adapter result passes response validation side_effects: - Retain the replayable answer and eligible connector notices. who: 'Attempt loop, consumer or completion callback' provenance: test source: >- services-platform:modules/integrations/tests/integration/services/executions/invoke-capability.integration.spec.js#records the execution succeeded, with its digest, summaries, reference and one attempt - from: running to: failed trigger: End a failed attempt guard: The failure is terminal or further attempts are exhausted side_effects: - Retain the failure answer and release the key when appropriate for pre-send failure. who: Attempt loop or consumer provenance: observed source: 'services-platform:modules/integrations/services/executions/attempt-loop.ts' - from: running to: retrying trigger: Prepare a safe retry guard: The request could not reach the provider and retry rules permit continuation side_effects: - Schedule another attempt within budget or on the asynchronous schedule. who: Attempt loop or consumer provenance: test source: >- services-platform:modules/integrations/tests/integration/services/executions/invoke-capability.integration.spec.js#retries a timeout before send after the sync schedule’s first delay, with the same execution id, and succeeds - from: retrying to: running trigger: Begin a scheduled retry guard: Admission succeeds for the next attempt side_effects: - Create the next attempt record. who: Attempt loop or consumer provenance: observed source: 'services-platform:modules/integrations/services/executions/attempt-loop.ts' - from: retrying to: failed trigger: Stop retry continuation guard: Sensitive work exceeds its synchronous budget or the attempt ceiling prevents another try side_effects: - Release the request key and return failure without more queued work. who: Attempt loop or consumer provenance: test source: >- services-platform:modules/integrations/tests/integration/services/executions/invoke-capability.integration.spec.js#queues no continuation and nudges no dispatcher, so nothing runs in the background - from: running to: unknown trigger: Lose certainty after sending guard: The failure can occur after provider receipt side_effects: - Schedule investigation after thirty seconds and do not resend. who: Attempt loop or consumer provenance: test source: >- services-platform:modules/integrations/tests/integration/services/executions/reconciler.integration.spec.js#looks the execution up on the same connection with the contracts’ lookup shape, never repeating the authorize - from: retrying to: unknown trigger: Detect orphaned continuation guard: Retrying work has no dispatch and its last change is more than eight hours old side_effects: - Make provider investigation due immediately. who: Reconciler minute pass provenance: observed source: 'services-platform:modules/integrations/services/executions/reconciler.ts' - from: accepted to: unknown trigger: Detect an overdue accepted call guard: Its asynchronous completion budget is spent side_effects: - Make provider investigation due immediately. who: Reconciler minute pass provenance: observed source: 'services-platform:modules/integrations/services/executions/reconciler.ts' - from: running to: unknown trigger: Detect a lost running attempt guard: The last unfinished attempt exceeds its bound plus one minute side_effects: - Make provider investigation due immediately. who: Reconciler minute pass provenance: observed source: 'services-platform:modules/integrations/services/executions/reconciler.ts' - from: unknown to: succeeded trigger: Find a successful provider result guard: Lookup confirms success or an applicable late adapter completion wins settlement side_effects: - Save the answer and create its applicable completion event. who: Reconciler or completion callback provenance: observed source: 'services-platform:modules/integrations/services/executions/reconciler.ts' - from: unknown to: failed trigger: Find a failed provider result guard: >- Lookup confirms failure, final non-sensitive investigation finds nothing, or a valid late asynchronous timeout completion reports failure side_effects: - Save failure and create its applicable completion event. who: Reconciler provenance: observed source: 'services-platform:modules/integrations/services/executions/reconciler.ts' - from: unknown to: needs_attention trigger: Exhaust inconclusive investigation guard: 'Five lookups remain inconclusive, final sensitive lookup is not found, or no usable lookup exists' side_effects: - 'Save attention state, request administrator mail and record attention telemetry.' who: Reconciler provenance: observed source: 'services-platform:modules/integrations/services/executions/reconciler.ts' - from: needs_attention to: succeeded trigger: Record a person’s successful finding guard: The execution needs attention and a note accompanies the result side_effects: - Create completion and audit without calling the provider. who: Executions operator provenance: test source: >- services-platform:modules/integrations/tests/integration/services/executions/reconciler.integration.spec.js#lets an operator resolve a needs_attention execution by hand, audited, with no call to the provider - from: needs_attention to: failed trigger: Record a person’s failed finding guard: The execution needs attention and a note accompanies the result side_effects: - Create completion and audit without calling the provider. who: Executions operator provenance: test source: >- services-platform:modules/integrations/tests/integration/services/executions/reconciler.integration.spec.js#lets an operator resolve a needs_attention execution by hand, audited, with no call to the provider - from: accepted to: cancelled trigger: Cancel before the first attempt guard: Accepted work has no started attempt side_effects: - Remove queued work and release its key - Create applicable asynchronous cancellation completion. who: Executions operator provenance: test source: >- services-platform:modules/integrations/tests/integration/services/executions/invoke-capability.integration.spec.js#cancels an accepted execution before its first attempt, releasing its key and its queued dispatch - from: running to: cancelled trigger: Accept adapter cancellation guard: A running asynchronous call has cancellation requested and receives the cancellation result side_effects: - Create the applicable cancellation completion. who: Completion callback provenance: observed source: 'services-platform:modules/integrations/services/adapter-callbacks/adapter-callbacks.ts:completeExecution' - from: failed to: retrying trigger: Request an operator retry guard: 'Failure classification permits retry, the attempt limit remains and original input matches' side_effects: - Queue the next attempt under the same execution and record retry history. who: Executions operator provenance: test source: >- services-platform:modules/integrations/tests/integration/services/executions/execution-operations.integration.spec.js#refuses a request handed back that is not the one recorded - from: accepted to: succeeded trigger: Receive asynchronous success before a running marker guard: A valid asynchronous completion wins settlement side_effects: - Save the response and applicable completion event. who: Adapter completion callback provenance: observed source: 'services-platform:modules/integrations/services/adapter-callbacks/adapter-callbacks.ts:completeExecution' - from: unknown to: cancelled trigger: Receive late acknowledged cancellation guard: The asynchronous timeout is still unresolved and the adapter reports the requested cancellation side_effects: - Settle cancellation through the shared writer. who: Adapter completion callback provenance: observed source: 'services-platform:modules/integrations/services/adapter-callbacks/adapter-callbacks.ts:completeExecution' invalid: - from: succeeded to: failed why: Final successful outcome cannot be overwritten by a later failed answer. - from: failed to: succeeded why: A failed result cannot become success without an eligible new attempt. - from: unknown to: retrying why: Possible provider receipt forbids another attempt of the operation. - from: needs_attention to: retrying why: An exhausted investigation must be resolved rather than resent. - from: cancelled to: running why: Cancellation is final for that execution; reuse of a released key creates a new execution. concurrency: >- Attempt startup compares status and increasing attempt number, while settlement compares allowed source status before recording result and completion. Investigation runs can overlap and select the account’s current pin. Lookup waits begin at thirty seconds and then use two minutes, ten minutes, one hour and six hours. invariants: - INV-DOM-PFM-INT-003-03 - INV-DOM-PFM-INT-003-04 - INV-DOM-PFM-INT-003-05 - INV-DOM-PFM-INT-003-06 - INV-DOM-PFM-INT-003-07 - INV-DOM-PFM-INT-003-10 evidence: - >- services-platform at ad3f6a65: modules/integrations/db/integration-executions.table.ts; services/executions/attempt-loop.ts, settle-execution.ts, reconciler.ts, timeouts.ts, execution-operations.ts; services/adapter-callbacks/adapter-callbacks.ts; docs/integrations/README.md (The runtime) --- # Execution lifecycle ## The Lifecycle ```mermaid stateDiagram-v2 state "accepted" as s0 state "running" as s1 state "retrying" as s2 state "unknown" as s3 state "needs_attention" as s4 state "succeeded" as s5 state "failed" as s6 state "cancelled" as s7 s0 --> s1: Begin provider delivery s0 --> s6: Refuse provider admission s1 --> s5: Receive a valid successful answer s1 --> s6: End a failed attempt s1 --> s2: Prepare a safe retry s2 --> s1: Begin a scheduled retry s2 --> s6: Stop retry continuation s1 --> s3: Lose certainty after sending s2 --> s3: Detect orphaned continuation s0 --> s3: Detect an overdue accepted call s1 --> s3: Detect a lost running attempt s3 --> s5: Find a successful provider result s3 --> s6: Find a failed provider result s3 --> s4: Exhaust inconclusive investigation s4 --> s5: Record a person’s successful finding s4 --> s6: Record a person’s failed finding s0 --> s7: Cancel before the first attempt s1 --> s7: Accept adapter cancellation s6 --> s2: Request an operator retry s0 --> s5: Receive asynchronous success before a running marker s3 --> s7: Receive late acknowledged cancellation ``` ## Transitions Explained A retained request can be attempted, safely retried, investigated or settled. Post-send uncertainty goes to lookup rather than resend. Exhausted inconclusive investigation needs a person, while final non-sensitive not-found results can fail automatically. Cancellation and manual retry have separate source guards. ## What Cannot Happen Final successful outcome cannot be overwritten by a later failed answer. A failed result cannot become success without an eligible new attempt. Possible provider receipt forbids another attempt of the operation. An exhausted investigation must be resolved rather than resent. Cancellation is final for that execution; reuse of a released key creates a new execution. ## Concurrency and Timing Attempt startup compares status and increasing attempt number, while settlement compares allowed source status before recording result and completion. Investigation runs can overlap and select the account’s current pin. Lookup waits begin at thirty seconds and then use two minutes, ten minutes, one hour and six hours.