--- id: STATE-CRM-DQ-003 title: Duplicate re-check run and its batches object: DOM-CRM-DQ-001 product_area: crm module: data-quality revision: 1 status: draft design_intent: proposed implementation_evidence: observed source_sha: services-crm: a8d1364f1dd44b723bb5f9acba12f983e9ec8d69 states: - name: queued meaning: The run and its windows are committed with no completed batch yet. - name: running meaning: Completed batches exist and some work remains. - name: succeeded meaning: All stored batches have completed. - name: failed meaning: A batch has recorded final failure; prior detection writes remain. - name: batch queued meaning: A stored window awaits dispatch. - name: batch sent meaning: Dispatch sent a message while retaining its recoverable batch row. - name: batch processed meaning: Conditional completion counted the window after detection. - name: batch failed meaning: Final caught delivery failure stored a batch reason. transitions: - from: queued to: succeeded trigger: Creation finds an empty record population. guard: There are no windows to process. side_effects: - finish the run immediately who: Authorized administrator with duplicate-configuration permission provenance: test source: >- services-crm:modules/duplicates/tests/integration/services/persistence/duplicate-rescans.integration.spec.js#finishes such a run at once rather than leaving it queued for ever - from: queued to: running trigger: The first completed window leaves further batches. guard: The batch remains queued or sent. side_effects: - 'count that window’s records, keys and pairs' - record run start time who: re-check consumer provenance: test source: >- services-crm:modules/duplicates/tests/integration/services/queue/rescan-queue.integration.spec.js#counts the batch once, so a redelivery never moves the progress past the end - from: running to: running trigger: A later window completes while work remains. guard: Its completion does not reach the total batch count. side_effects: - add the window’s progress once who: re-check consumer provenance: test source: >- services-crm:modules/duplicates/tests/integration/services/queue/rescan-queue.integration.spec.js#counts the batch once, so a redelivery never moves the progress past the end - from: running to: succeeded trigger: Completion reaches the final batch. guard: Processed batch count reaches the initial total. side_effects: - record run completion time who: re-check consumer provenance: test source: >- services-crm:modules/duplicates/tests/integration/services/queue/rescan-queue.integration.spec.js#counts the batch once, so a redelivery never moves the progress past the end - from: queued to: succeeded trigger: The sole stored batch completes. guard: Conditional completion reaches the batch total. side_effects: - record its progress and start and finish times who: Re-check consumer provenance: observed source: >- services-crm:modules/duplicates/services/persistence/duplicate-rescans.ts:completeDuplicateRescanBatch - from: queued to: failed trigger: A caught last delivery fails before any window completes. guard: The message receive count is at least three. side_effects: - record failure on the window and run who: re-check consumer provenance: test source: >- services-crm:modules/duplicates/tests/integration/services/persistence/duplicate-rescans.integration.spec.js#fails the run, so a person polling the progress route is told rather than left watching - from: running to: failed trigger: A caught last delivery fails after earlier work. guard: The message receive count is at least three. side_effects: - fail the window and run - retain earlier committed detection who: re-check consumer provenance: test source: >- services-crm:modules/duplicates/tests/integration/services/persistence/duplicate-rescans.integration.spec.js#fails the run, so a person polling the progress route is told rather than left watching - from: failed to: failed trigger: Another window completes after run failure. guard: Dispatch and processing do not exclude failed runs. side_effects: - increment progress - retain failed run status who: re-check dispatcher and consumer provenance: observed source: >- services-crm:modules/duplicates/services/persistence/duplicate-rescans.ts:completeDuplicateRescanBatch - from: batch queued to: batch sent trigger: Dispatch successfully sends an eligible window. guard: The batch is eligible for the conditional sent update. side_effects: - retain the row in sent standing who: Re-check dispatcher or consumer provenance: observed source: >- services-crm:modules/duplicates/services/persistence/duplicate-rescans.ts:markDuplicateRescanBatchesSent - from: batch sent to: batch sent trigger: The dispatcher sends a window again after claim expiry. guard: The stored sent row has an expired claim. side_effects: - renew the claim and offer another message who: Re-check dispatcher or consumer provenance: observed source: >- services-crm:modules/duplicates/services/persistence/duplicate-rescans.ts:claimDuplicateRescanBatches - from: batch queued to: batch processed trigger: Delivery completes a window still recorded as queued. guard: The completion update accepts queued or sent standing. side_effects: - increment run progress once who: Re-check dispatcher or consumer provenance: observed source: >- services-crm:modules/duplicates/services/persistence/duplicate-rescans.ts:completeDuplicateRescanBatch - from: batch sent to: batch processed trigger: Delivery completes a sent window. guard: The completion update accepts queued or sent standing. side_effects: - increment run progress once who: Re-check dispatcher or consumer provenance: observed source: >- services-crm:modules/duplicates/services/persistence/duplicate-rescans.ts:completeDuplicateRescanBatch - from: batch queued to: batch failed trigger: A queued window records caught final failure. guard: The failure update accepts queued or sent standing. side_effects: - store the reason and fail its run who: Re-check dispatcher or consumer provenance: observed source: >- services-crm:modules/duplicates/services/persistence/duplicate-rescans.ts:failDuplicateRescanBatch - from: batch sent to: batch failed trigger: A sent window records caught final failure. guard: The failure update accepts queued or sent standing. side_effects: - store the reason and fail its run who: Re-check dispatcher or consumer provenance: observed source: >- services-crm:modules/duplicates/services/persistence/duplicate-rescans.ts:failDuplicateRescanBatch invalid: - from: succeeded to: running why: A successful run stays terminal; another request creates another run. - from: failed to: running why: >- Failure status remains terminal even while leftover batches move its counters. - from: queued to: queued why: Start refuses another run of the same type while one is queued or running. concurrency: >- Run creation serializes per workspace and type through a transaction advisory lock. Dispatch uses skip-locked claims lasting ten minutes; consumer concurrency is capped at four. Conditional completion prevents duplicate counting, but neither claims nor completion prevent overlapping detection after a resend. invariants: - INV-DOM-CRM-DQ-001-12 - INV-DOM-CRM-DQ-001-13 evidence: - >- services-crm at a8d1364f: modules/duplicates/db/duplicate-rescans.table.ts, duplicate-rescan-batches.table.ts; services/persistence/duplicate-rescans.ts; services/queue/rescan-dispatch.ts, rescan-worker.ts; aws/crm.duplicate-rescans.names.ts; modules/duplicates/aws/logic/duplicate-rescans.lambda.ts. --- # Duplicate Re-Check Run And Its Batches ## The Lifecycle ```mermaid stateDiagram-v2 state "queued" as queued state "succeeded" as succeeded state "running" as running state "failed" as failed state "batch queued" as batch_queued state "batch sent" as batch_sent state "batch processed" as batch_processed state "batch failed" as batch_failed queued --> succeeded: Creation finds an empty record population queued --> running: The first completed window leaves further batches running --> running: A later window completes while work remains running --> succeeded: Completion reaches the final batch queued --> succeeded: The sole stored batch completes queued --> failed: A caught last delivery fails before any window completes running --> failed: A caught last delivery fails after earlier work failed --> failed: Another window completes after run failure batch_queued --> batch_sent: Dispatch successfully sends an eligible window batch_sent --> batch_sent: The dispatcher sends a window again after claim expiry batch_queued --> batch_processed: Delivery completes a window still recorded as queued batch_sent --> batch_processed: Delivery completes a sent window batch_queued --> batch_failed: A queued window records caught final failure batch_sent --> batch_failed: A sent window records caught final failure ``` ## Transitions Explained Run states describe aggregate progress, while batch states describe queued delivery and completion. The empty run and a completed single-window run both finish directly. A caught exhausted delivery fails the batch and run; unprocessed windows of that failed run can still execute. ## What Cannot Happen Terminal runs do not return to running. A second run is rejected only while another is queued or running; failed leftover work may therefore overlap a new run. ## Concurrency and Timing Totals are fixed at start, but windows are reread against current records. New identities inside an old window can enter processing, and deletions remove identities. Ten-minute resend claims are shorter than queue visibility; a consumer crash can leave an unresolved window repeatedly offered.