---
id: WF-PFM-LLM-004
title: Recover interrupted requests and tasks hourly
trigger: Recover stale requests and refresh usage bookkeeping hourly.
product_area: platform
module: llm
revision: 1
status: draft
design_intent: proposed
implementation_evidence: observed
inference: []
source_sha:
  services-platform: ad3f6a6582843008ed0f5a17b131878d15829af1
implements:
  - CAP-PFM-LLM-009
  - CAP-PFM-LLM-003
moves:
  - STATE-PFM-LLM-001
  - STATE-PFM-LLM-002
crosses:
  - SVC-PFM-LLM-llm
idempotency: >-
  Settlement and alert writes compare stored status, while abandonment requires running. Repetition skips
  terminal requests even if their period movement failed.
concurrency: >-
  The deployed function reserves one execution slot. It visits organizations sequentially, catches their sweep
  failures and independently attempts notice and reporting steps.
hops:
  - 'n': 1
    actor_or_service: EventBridge schedule
    action: Invoke hourly reconciliation through the declared EventBridge rule and delivery dead-letter queue.
    anchor: 'services-platform:modules/llm/aws/logic/llm-reconciliation.lambda.ts:LLM_RECONCILIATION_CRON'
    seam: schedule
    on_failure: >-
      Failed target delivery is retained for replay; function execution follows its configured invocation
      behavior.
    left_inconsistent: Stale requests remain until a successful later pass.
    observer: 'Reconciliation logs, persisted recovery state and configured operational metrics.'
    provenance: observed
    source: 'services-platform:modules/llm/aws/logic/llm-reconciliation.lambda.ts:createLlmReconciliationSchedule'
  - 'n': 2
    actor_or_service: Reconciliation pass
    action: Visit each operational organization and run its settlement sweep.
    anchor: 'services-platform:modules/llm/services/reconciliation/reconciliation-pass.ts:runLlmReconciliation'
    seam: sync
    on_failure: Log an organization failure and continue the remaining visits.
    left_inconsistent: That organization’s unfinished accounting remains.
    observer: 'Reconciliation logs, persisted recovery state and configured operational metrics.'
    provenance: observed
    source: 'services-platform:modules/llm/services/reconciliation/reconciliation-pass.ts:runLlmReconciliation'
  - 'n': 3
    actor_or_service: Settlement sweep (in the organization)
    action: >-
      Select up to two hundred unsettled requests older than a day and charge each estimate before separately
      moving its period.
    anchor: 'services-platform:modules/llm/services/reconciliation/llm-settlement-sweep.ts:sweepUnsettledInvocations'
    seam: sync
    on_failure: Stop that organization’s sweep if a write throws.
    left_inconsistent: >-
      Settled requests with failed period movements are excluded from later sweeps; missing recorded holds
      cannot be released.
    observer: 'Reconciliation logs, persisted recovery state and configured operational metrics.'
    provenance: observed
    source: 'services-platform:modules/llm/services/persistence/invocations.ts:listUnsettledInvocations'
  - 'n': 4
    actor_or_service: Settlement sweep (in the organization)
    action: Stamp the first past-window alert on each processed request.
    anchor: 'services-platform:modules/llm/services/persistence/invocations.ts:markPastWindowAlerted'
    seam: sync
    on_failure: A stamping error ends this organization’s sweep.
    left_inconsistent: Accounting already committed can remain without its recovery-alert metric.
    observer: 'Reconciliation logs, persisted recovery state and configured operational metrics.'
    provenance: observed
    source: >-
      services-platform:modules/llm/services/reconciliation/llm-settlement-sweep.ts:LLM_RECONCILIATION_PAST_WINDOW_METRIC
  - 'n': 5
    actor_or_service: Settlement sweep (in the organization)
    action: Abandon stale running tasks and compare recent closed-task totals with settled requests.
    anchor: 'services-platform:modules/llm/services/persistence/executions.ts:markExecutionAbandoned'
    seam: sync
    on_failure: A write or read error ends this organization’s sweep.
    left_inconsistent: Drift is reported without repairing totals.
    observer: 'Reconciliation logs, persisted recovery state and configured operational metrics.'
    provenance: observed
    source: 'services-platform:modules/llm/services/reconciliation/llm-settlement-sweep.ts:sweepUnsettledInvocations'
  - 'n': 6
    actor_or_service: Reconciliation pass
    action: Retry owed notices and rebuild reporting as independent follow-up steps.
    anchor: 'services-platform:modules/llm/services/notices/retry-threshold-notices.ts:retryThresholdNotices'
    seam: sync
    on_failure: Log failed steps while allowing the next step to run.
    left_inconsistent: Unsent notices and stale daily totals await later passes.
    observer: 'Reconciliation logs, persisted recovery state and configured operational metrics.'
    provenance: observed
    source: 'services-platform:modules/llm/services/reconciliation/reconciliation-pass.ts:runLlmReconciliation'
branches:
  - The structured hops retain the applicable alternative path and its failure boundary.
waits:
  - The sweep selects requests and running tasks older than twenty-four hours.
evidence:
  - >-
    services-platform at ad3f6a6: modules/llm/aws/lambda/RECONCILIATION.handler.ts;
    aws/logic/llm-reconciliation.lambda.ts; services/reconciliation/reconciliation-pass.ts,
    llm-settlement-sweep.ts; services/persistence/invocations.ts, executions.ts;
    services/metering/llm-credit-ledger.ts; tests/integration/services/reconciliation/*.
controls: [TC-AI-006, TC-AI-007, TC-AVL-001, TC-AVL-004, TC-AVL-005, TC-AZ-001, TC-AZ-002, TC-AZ-003, TC-AZ-004, TC-CHG-001, TC-CHG-002, TC-CHG-003, TC-CHG-004, TC-CHG-005, TC-CHG-006, TC-DAT-001, TC-DAT-002, TC-DAT-003, TC-IAM-001, TC-IAM-002, TC-IAM-004, TC-IAM-005, TC-IAM-006, TC-LOG-002, TC-LOG-003, TC-LOG-004, TC-LOG-005, TC-MON-001, TC-MON-002, TC-PI-001, TC-PI-002, TC-PI-003, TC-PI-004, TC-PI-005, TC-PRV-001, TC-PRV-002, TC-TEN-001, TC-TEN-002, TC-TEN-003, TC-VEN-001, TC-VEN-002, TC-VEN-003]
---

# Recover interrupted requests and tasks hourly

## What It Does

The scheduled function visits operational organizations, closes work older than a day and records accounting signals. It then retries notices and rebuilds daily history.

## The Flow

```mermaid
flowchart TD
  H1["Invoke hourly reconciliation through the declared EventBridge rule and delivery dead-letter queue."]
  H2["Visit each operational organization and run its settlement sweep."]
  H1 --> H2
  H3["Select up to two hundred unsettled requests older than a day and charge each estimate before separately moving its period."]
  H2 --> H3
  H4["Stamp the first past-window alert on each processed request."]
  H3 --> H4
  H5["Abandon stale running tasks and compare recent closed-task totals with settled requests."]
  H4 --> H5
  H6["Retry owed notices and rebuild reporting as independent follow-up steps."]
  H5 --> H6
```

Request settlement, period movement, alert stamping, task abandonment and seven-day drift checks occur in sequence per organization. Follow-up steps are independently attempted.

## When Something Fails

One organization’s failure is logged without stopping other visits. A request settled before its failed period movement will not be selected again, and alert stamping may never occur for it.

## Open Design Questions

Product must choose recovery charges; implementation preparation needs interruption proof around each separate write.