---
id: WF-COM-FUL-007
title: Detect stuck and inconsistent order work every minute
trigger: The minute reconciler reaches a check’s due cadence.
product_area: commerce
module: fulfillment
revision: 1
status: draft
design_intent: proposed
implementation_evidence: observed
inference: []
source_sha:
  services-commerce: c8b66a98e2490617aec38ba5c8165dd9bd0a8f9b
implements:
  - CAP-COM-FUL-009
  - CAP-COM-FUL-007
moves:
  - STATE-COM-FUL-004
  - STATE-COM-FUL-003
crosses:
  - SVC-COM-FUL-operations
  - SVC-COM-FUL-fulfillment
idempotency: >-
  Standing findings refresh their recurrence record. Event findings deduplicate
  by subject identity, and money resumes reuse the worker’s retained keys.
concurrency: >-
  Checks use their due schedule and cross-workspace read policy. Candidate
  findings are written within each affected workspace, with no claimed complete
  scan per run.
hops:
  - 'n': 1
    actor_or_service: Reconciler
    action: >-
      Select checks due by the current minute, fifth minute, hour or daily
      cadence.
    anchor: >-
      services-commerce:modules/operations/services/reconciler/reconciler.ts:runReconciler
    seam: schedule
    on_failure: Log a thrown check and continue other checks.
    left_inconsistent: That check’s detection remains incomplete.
    observer: Check metrics and logs.
    provenance: documented
    source: 'services-commerce:docs/migrations/operations/README.md#D-P5-07'
  - 'n': 2
    actor_or_service: Reconciler
    action: >-
      Read cross-workspace candidates for billing, tenders, holds, missing
      links, migrated blocks, reversals and drift.
    anchor: >-
      services-commerce:modules/operations/services/persistence/reconciler-candidates.ts
    seam: schedule
    on_failure: Abort the affected check when its read is refused.
    left_inconsistent: No diagnosis is retained for unread candidates.
    observer: Candidate metrics.
    provenance: observed
    source: >-
      services-commerce:modules/operations/services/persistence/reconciler-candidates.ts
  - 'n': 3
    actor_or_service: Reconciler
    action: >-
      Enter each candidate’s workspace and create or refresh its standing
      finding.
    anchor: >-
      services-commerce:modules/fulfillment/services/persistence/operational-exceptions.ts
    seam: sync
    on_failure: Log failure of the affected check.
    left_inconsistent: Earlier findings in the scan can remain stored.
    observer: Operations finding list.
    provenance: test
    source: >-
      services-commerce:modules/operations/tests/integration/services/reconciler/reconciler.integration.spec.js
  - 'n': 4
    actor_or_service: Reconciler
    action: >-
      Delegate stale adjustment execution to the keyed worker, with the longer
      wait for processor-only refunds.
    anchor: >-
      services-commerce:modules/fulfillment/services/adjustments/adjustment-worker.ts:executeAdjustment
    seam: sync
    on_failure: Use the adjustment worker’s result and finding behavior.
    left_inconsistent: Financial work remains at its retained step.
    observer: Adjustment results and findings.
    provenance: documented
    source: 'services-commerce:docs/migrations/operations/README.md#D-P5-16'
  - 'n': 5
    actor_or_service: Reconciler (hourly)
    action: >-
      Measure obligations dead-letter depth without creating an operational
      exception.
    anchor: >-
      services-commerce:modules/operations/services/reconciler/queue-depth-port.ts
    seam: schedule
    on_failure: Log the metric failure.
    left_inconsistent: No business record changes in this hop.
    observer: Dead-letter depth metric.
    provenance: observed
    source: 'services-commerce:modules/operations/services/reconciler/reconciler.ts'
branches:
  - >-
    The register's stale and acknowledgement passes, the billing, tender and
    void passes and the money mirror check belong to Orders' checkout sweeper;
    the reconciler records them as owned there and runs nothing twice.
  - >-
    Scheduled amendments, overdue returns and inventory balances are skipped as
    later work.
waits: []
evidence:
  - >-
    services-commerce at c8b66a98: modules/operations/services/reconciler/*.ts;
    services/persistence/reconciler-candidates.ts;
    aws/logic/reconciler.lambda.ts; docs/migrations/operations/README.md
    (D-P5-07, D-P5-08, D-P5-09, D-P5-16, D-P5-17).
controls: [TC-IAM-006, TC-AZ-001, TC-AZ-002, TC-AZ-003, TC-AZ-004, TC-AZ-005, TC-AZ-006, TC-TEN-001, TC-TEN-002, TC-TEN-003, TC-CRY-001, TC-DAT-001, TC-DAT-002, TC-DAT-003, TC-LOG-001, TC-LOG-002, TC-LOG-003, TC-LOG-004, TC-LOG-005, TC-CHG-001, TC-CHG-002, TC-CHG-003, TC-CHG-004, TC-CHG-007, TC-AVL-004, TC-MON-001, TC-MON-002, TC-MON-004, TC-PI-001, TC-PI-002, TC-PI-003, TC-PI-004, TC-PI-005, TC-VEN-001, TC-VEN-002, TC-VEN-003, TC-PRV-001, TC-PRV-002]
---

# Detect Stuck and Inconsistent Order Work Every Minute

## What It Does

The reconciler selects checks due at the current minute and investigates candidate records across workspaces. It writes findings in each record’s workspace and delegates unfinished adjustments to their owning worker. Queue depth is measured separately from finding creation.

## The Flow

```mermaid
sequenceDiagram
  participant C as Schedule
  participant R as Reconciler
  participant X as Cross-workspace reads
  participant E as Exceptions
  participant W as Adjustment worker
  C->>R: every minute
  R->>X: candidates (bounded page)
  R->>E: raise in each workspace
  R->>W: resume stale adjustments
  R->>R: metrics per check
```

## When Something Fails

A thrown check is logged without preventing the remaining checks. Several declared checks are placeholders. Findings and delegated resumes can be partial, and the schedule alone does not guarantee that a failed page completes.