--- id: WF-CRM-ENT-005 title: Reprocess compliance across the workspace trigger: >- An organization updater requests a workspace compliance reprocess, often after editing type policy. product_area: crm module: entities revision: 1 status: draft design_intent: proposed implementation_evidence: observed inference: [] source_sha: services-crm: a8d1364f1dd44b723bb5f9acba12f983e9ec8d69 services-platform: ad3f6a6582843008ed0f5a17b131878d15829af1 react-console: baf6dae382d6f41dce976c7f848667ccc37e076f implements: - CAP-CRM-ENT-012 moves: - STATE-CRM-ENT-004 - STATE-CRM-ENT-003 crosses: - SVC-CRM-ENT-entities idempotency: >- A request returns the existing running run without creating another. Batch cursor movement commits with organization recalculation, so redelivery of a running run skips committed work. Failed and completed runs are terminal and cannot resume on redrive. concurrency: >- The database permits one running run per workspace. Batch workers lock the run and organization rows, committing progress together; policy edits can change the rules used by later batches. hops: - 'n': 1 actor_or_service: Entities route action: >- Create the run with its scan count or return the existing running run, then request event dispatch. anchor: >- services-crm:modules/entities/services/organizations/compliance/compliance-reprocess.ts:startComplianceReprocess seam: sync on_failure: Refuse persistence failure without recording a partial run. left_inconsistent: No partial business write remains. observer: The caller receives the result. provenance: test source: >- services-crm:modules/entities/tests/integration/services/persistence/compliance-reprocesses.integration.spec.js#refuses a second running run for the tenant - 'n': 2 actor_or_service: CRM change-events dispatcher action: Publish the run insertion event to the reprocess handler. anchor: 'services-crm:aws/crm.change-events.ts:crmChangeEventStreams' seam: event on_failure: Retain the event until queue acceptance succeeds. left_inconsistent: The run can remain recorded but not yet started. observer: Dispatcher logs identify delayed starts. provenance: documented source: 'services-crm:docs/change-events.md#CRM cutover from the shared outbox' - 'n': 3 actor_or_service: Compliance reprocess runner action: >- Process up to 25 organizations per transaction while a 40-second worker budget remains, recalculating each and committing cursor and counts with the batch. anchor: >- services-crm:modules/entities/services/organizations/compliance/compliance-reprocess.ts:runReprocessToBudget seam: queue on_failure: >- Roll back a failed batch, record a redacted reason and throw for ordinary SQS retry. left_inconsistent: >- Earlier batches remain committed while later organizations await successful continuation. observer: 'Run counts, reason and consumer logs show partial progress.' provenance: observed source: >- services-crm:modules/entities/services/organizations/compliance/compliance-reprocess.ts:runComplianceReprocess - 'n': 4 actor_or_service: Compliance reprocess runner action: >- Increment the continuation counter when more organizations remain after the budget, producing another worker event. anchor: >- services-crm:modules/entities/services/persistence/compliance-reprocesses.ts:recordReprocessContinuation seam: event on_failure: >- Fail the delivery if continuation cannot be recorded so the queue can retry. left_inconsistent: The still-running run may await another successful delivery. observer: Run progress and continuation records show the wait. provenance: observed source: >- services-crm:modules/entities/services/persistence/compliance-reprocesses.ts:recordReprocessContinuation - 'n': 5 actor_or_service: Compliance reprocess runner action: >- Mark the exhausted scan completed, or mark the run failed on the final permitted failing receive before throwing. anchor: >- services-crm:modules/entities/services/organizations/compliance/compliance-reprocess.ts:recordFailure seam: queue on_failure: >- The final failure is dead-lettered; requesting a new run is the recovery path. Redrive skips the terminal run. left_inconsistent: A failed run leaves a partially reprocessed workspace. observer: >- Run status, redacted reason and the dead-letter queue identify terminal failure. provenance: observed source: >- services-crm:modules/entities/services/organizations/compliance/compliance-reprocess.ts:recordFailure branches: - >- Nested recalculations submit notice batches normally but rely on scheduled communication dispatch after the batch commits. waits: [] evidence: - >- services-crm at 7ea48a41: modules/entities/services/organizations/compliance/compliance-reprocess.ts; services/persistence/compliance-reprocesses.ts; db/compliance-reprocesses.table.ts; aws/crm.change-events.ts; docs/change-events.md. - >- services-platform at ad3f6a65: docs/queued-work.md §3 (failure recorded where an operator reads it). controls: - TC-AVL-004 - TC-AZ-001 - TC-AZ-002 - TC-AZ-003 - TC-AZ-004 - TC-AZ-005 - TC-AZ-006 - TC-DAT-001 - TC-DAT-002 - TC-DAT-003 - TC-IAM-001 - TC-IAM-005 - TC-IAM-006 - TC-LOG-004 - TC-LOG-005 - TC-MON-001 - TC-MON-002 - TC-PI-001 - TC-PI-002 - TC-PI-003 - TC-PI-004 - TC-PI-005 - TC-PRV-001 - TC-PRV-002 - TC-TEN-001 - TC-TEN-002 - TC-TEN-003 narrative: status: verified review: NARRATIVE-REVIEW-2026-10-23 narrative_sources: - section: What It Does provenance: observed source: >- services-crm:modules/entities/services/organizations/compliance/compliance-reprocess.ts:startComplianceReprocess - section: What It Does provenance: documented source: 'services-crm:aws/crm.change-events.ts:crmChangeEventStreams' - section: What It Does provenance: observed source: >- services-crm:modules/entities/services/organizations/compliance/compliance-reprocess.ts:runReprocessToBudget - section: What It Does provenance: observed source: >- services-crm:modules/entities/services/persistence/compliance-reprocesses.ts:recordReprocessContinuation - section: What It Does provenance: observed source: >- services-crm:modules/entities/services/organizations/compliance/compliance-reprocess.ts:recordFailure - section: What Starts It provenance: observed source: >- services-crm:modules/entities/services/organizations/compliance/compliance-reprocess.ts:startComplianceReprocess - section: What Starts It provenance: documented source: 'services-crm:aws/crm.change-events.ts:crmChangeEventStreams' - section: What Starts It provenance: observed source: >- services-crm:modules/entities/services/organizations/compliance/compliance-reprocess.ts:runReprocessToBudget - section: What Starts It provenance: observed source: >- services-crm:modules/entities/services/persistence/compliance-reprocesses.ts:recordReprocessContinuation - section: What Starts It provenance: observed source: >- services-crm:modules/entities/services/organizations/compliance/compliance-reprocess.ts:recordFailure - section: The Flow provenance: observed source: >- services-crm:modules/entities/services/organizations/compliance/compliance-reprocess.ts:startComplianceReprocess - section: The Flow provenance: documented source: 'services-crm:aws/crm.change-events.ts:crmChangeEventStreams' - section: The Flow provenance: observed source: >- services-crm:modules/entities/services/organizations/compliance/compliance-reprocess.ts:runReprocessToBudget - section: The Flow provenance: observed source: >- services-crm:modules/entities/services/persistence/compliance-reprocesses.ts:recordReprocessContinuation - section: The Flow provenance: observed source: >- services-crm:modules/entities/services/organizations/compliance/compliance-reprocess.ts:recordFailure - section: Hop by Hop provenance: observed source: >- services-crm:modules/entities/services/organizations/compliance/compliance-reprocess.ts:startComplianceReprocess - section: Hop by Hop provenance: documented source: 'services-crm:aws/crm.change-events.ts:crmChangeEventStreams' - section: Hop by Hop provenance: observed source: >- services-crm:modules/entities/services/organizations/compliance/compliance-reprocess.ts:runReprocessToBudget - section: Hop by Hop provenance: observed source: >- services-crm:modules/entities/services/persistence/compliance-reprocesses.ts:recordReprocessContinuation - section: Hop by Hop provenance: observed source: >- services-crm:modules/entities/services/organizations/compliance/compliance-reprocess.ts:recordFailure - section: When Something Fails provenance: observed source: >- services-crm:modules/entities/services/organizations/compliance/compliance-reprocess.ts:startComplianceReprocess - section: When Something Fails provenance: documented source: 'services-crm:aws/crm.change-events.ts:crmChangeEventStreams' - section: When Something Fails provenance: observed source: >- services-crm:modules/entities/services/organizations/compliance/compliance-reprocess.ts:runReprocessToBudget - section: When Something Fails provenance: observed source: >- services-crm:modules/entities/services/persistence/compliance-reprocesses.ts:recordReprocessContinuation - section: When Something Fails provenance: observed source: >- services-crm:modules/entities/services/organizations/compliance/compliance-reprocess.ts:recordFailure - section: Repeats and Races provenance: observed source: >- services-crm:modules/entities/services/organizations/compliance/compliance-reprocess.ts:startComplianceReprocess - section: Repeats and Races provenance: documented source: 'services-crm:aws/crm.change-events.ts:crmChangeEventStreams' - section: Repeats and Races provenance: observed source: >- services-crm:modules/entities/services/organizations/compliance/compliance-reprocess.ts:runReprocessToBudget - section: Repeats and Races provenance: observed source: >- services-crm:modules/entities/services/persistence/compliance-reprocesses.ts:recordReprocessContinuation - section: Repeats and Races provenance: observed source: >- services-crm:modules/entities/services/organizations/compliance/compliance-reprocess.ts:recordFailure - section: What Is Observable provenance: observed source: >- services-crm:modules/entities/services/organizations/compliance/compliance-reprocess.ts:startComplianceReprocess - section: What Is Observable provenance: documented source: 'services-crm:aws/crm.change-events.ts:crmChangeEventStreams' - section: What Is Observable provenance: observed source: >- services-crm:modules/entities/services/organizations/compliance/compliance-reprocess.ts:runReprocessToBudget - section: What Is Observable provenance: observed source: >- services-crm:modules/entities/services/persistence/compliance-reprocesses.ts:recordReprocessContinuation - section: What Is Observable provenance: observed source: >- services-crm:modules/entities/services/organizations/compliance/compliance-reprocess.ts:recordFailure - section: What It Does provenance: inferred source: >- services-crm:modules/entities/services/organizations/compliance/compliance-reprocess.ts:startComplianceReprocess; services-crm:aws/crm.change-events.ts:crmChangeEventStreams; services-crm:modules/entities/services/organizations/compliance/compliance-reprocess.ts:runReprocessToBudget; services-crm:modules/entities/services/persistence/compliance-reprocesses.ts:recordReprocessContinuation; services-crm:modules/entities/services/organizations/compliance/compliance-reprocess.ts:recordFailure note: >- The explicitly labeled rationale interprets the existing transaction, transport and recovery boundaries; it is not a product decision. - section: Open Design Questions provenance: documented source: 'rhythm-v2-docs:specs/crm/entities/FINDINGS.md#Findings' --- # Reprocess compliance across the workspace ## What It Does A reprocess reevaluates existing organizations after administration changes their role expectations. An association with many chapters needs the scan to survive bounded workers and partial failure. The route records or reuses a durable run and returns promptly, while later workers retain progress and finish or fail explicitly. Our reading is that a durable cursor is useful only when progress and business changes share a commit. Otherwise a retry could skip an organization that was never recalculated or keep repeating one that already changed. Here each batch owns that boundary; the whole workspace is not one rollback unit. ## What Starts It A staff member with organization update authority requests another compliance scan. The service records a durable run or returns the workspace’s existing running run. That response begins or identifies queued work; it does not report that every organization has already been assessed. Operators inspect worker progress and the terminal result to determine how much work completed. ## The Flow The diagram shows the sequence and service seams. Each synchronous stage waits for its result, while captured-event, queue and scheduled stages can run after the originating request has returned. The hop descriptions below identify the write and failure boundary, so the last arrow must not be mistaken for one atomic transaction across every participant. ```mermaid sequenceDiagram participant U as Requester participant D as CRM database participant Q as Change queue participant W as Scan worker U->>D: create scan or return running scan D->>Q: insertion event Q->>W: deliver scan work W->>D: commit organization batch and cursor alt more work beyond budget W->>D: increment continuation counter D->>Q: continuation event else scan exhausted W->>D: mark completed end Note over W,D: Final failure marks terminal failed; redrive skips it. ``` ## Hop by Hop ### 1. Entities route The route requires organization update authority and starts a run or returns the workspace’s running one. It records the scan request and asks for dispatch. The immediate response’s organization count is request information, not a report that all those organizations have already been refreshed. ### 2. CRM change-events dispatcher The dispatcher publishes the insertion event to the reprocess handler. Queue acceptance is the point at which work is handed off; until then the saved run can remain waiting. Failed publication retains the event so later dispatch can make another attempt. ### 3. Compliance reprocess runner The worker locks the run and takes the next ordered organization batch. Up to twenty-five subjects are recalculated per transaction; cursor and counts commit with their assessments. A failed batch rolls back together, while earlier successful batches remain a valid partial result. ### 4. Compliance reprocess runner When the worker reaches its time budget before finishing, it increments the run’s continuation counter. That captured change requests another worker delivery. A failure to record continuation fails the current delivery, leaving the still-running run available for normal queue recovery. ### 5. Compliance reprocess runner The exhausted scan records completion and finish time. A batch failure on the final permitted receive records terminal failure and a redacted reason, then throws into queue dead-letter handling. A later delivery of either terminal run skips it rather than opening it for more work. ## When Something Fails ### Hop 1 — Entities route The failure boundary is: Refuse persistence failure without recording a partial run. The remaining work or data is: No partial business write remains. The caller receives the result. ### Hop 2 — CRM change-events dispatcher The failure boundary is: Retain the event until queue acceptance succeeds. The remaining work or data is: The run can remain recorded but not yet started. Dispatcher logs identify delayed starts. A later attempt can be required; the scheduling or publication boundary does not establish an immediate user-visible completion. ### Hop 3 — Compliance reprocess runner The failure boundary is: Roll back a failed batch, record a redacted reason and throw for ordinary SQS retry. The remaining work or data is: Earlier batches remain committed while later organizations await successful continuation. Run counts, reason and consumer logs show partial progress. This failure is downstream of committed input; operators investigate its own delivery rather than treating the caller’s original save as failed. ### Hop 4 — Compliance reprocess runner The failure boundary is: Fail the delivery if continuation cannot be recorded so the queue can retry. The remaining work or data is: The still-running run may await another successful delivery. Run progress and continuation records show the wait. A later attempt can be required; the scheduling or publication boundary does not establish an immediate user-visible completion. ### Hop 5 — Compliance reprocess runner The failure boundary is: The final failure is dead-lettered; requesting a new run is the recovery path. Redrive skips the terminal run. The remaining work or data is: A failed run leaves a partially reprocessed workspace. Run status, redacted reason and the dead-letter queue identify terminal failure. This failure is downstream of committed input; operators investigate its own delivery rather than treating the caller’s original save as failed. ## Repeats and Races Starting while a run is already running returns that run. Database enforcement resolves simultaneous starts the same way. Worker redelivery uses the committed cursor and locks the run, so it resumes unfinished batches rather than repeating committed progress or letting two workers independently scan the same position. A completed or failed run is terminal. Redrive does not reopen it; recovery requests another run. Type policy can change between batches because there is no pinned policy snapshot. Nested evaluations can produce ordinary notice submissions, whose later dispatch and delivery remain distinct from successful scan completion. ## What Is Observable The caller receives the result. Dispatcher logs identify delayed starts. Run counts, reason and consumer logs show partial progress. Run progress and continuation records show the wait. Run status, redacted reason and the dead-letter queue identify terminal failure. The requester sees the accepted scan message, while operators use stored run status, cursor, counts and redacted reason to distinguish waiting, progress, completion and failure. The route does not provide a live progress page, and notice delivery is established separately through Platform outcomes. ## Open Design Questions A failed run can leave a partially refreshed workspace, and a completed run can have used different policy revisions across its batches. The existing findings preserve those boundaries and the limited operator surface for inspecting progress. The immediate request message must not be read as a synchronous completion report.