--- id: CAP-PFM-LLM-009 title: Recover interrupted requests and identify accounting drift feature: FEAT-PFM-LLM-03 product_area: platform module: llm revision: 1 status: draft product_intent: proposed implementation_evidence: observed supersedes: [] source_sha: services-platform: ad3f6a6582843008ed0f5a17b131878d15829af1 intent_sources: [] purpose: Release stranded reservations and make abandoned tasks visible to Rhythm operators. outcomes: - Recovery settles eligible stranded requests and releases reservations. - Stalled tasks receive an abandoned outcome. - Task accounting drift is reported for investigation. concepts: - CON-PFM-LLM-001 - CON-PFM-LLM-003 actors: - ACT-PFM-012 surfaces: - job preconditions: [] postconditions: - Eligible stranded work receives recovery outcomes. permissions: [] rules: - id: BR-CAP-PFM-LLM-009-01 proved_by: [ATS-PFM-LLM-007-12, ATS-PFM-LLM-008-10, ATS-PFM-LLM-009-01, ATS-PFM-LLM-009-06, TST-PFM-LLM-028, TST-PFM-LLM-046] statement: >- Hourly reconciliation visits operational organizations and settles requests older than twenty-four hours at their stored estimate without a money figure, releasing their recorded holds. kind: lifecycle enforced_at: - service must_prove: true provenance: observed source: 'services-platform:modules/llm/aws/lambda/RECONCILIATION.handler.ts' origin: none - id: BR-CAP-PFM-LLM-009-02 proved_by: [ATS-PFM-LLM-009-02, TST-PFM-LLM-046] statement: Requests inside the recovery window keep their state and reservations. kind: policy enforced_at: - service must_prove: true provenance: test source: >- services-platform:modules/llm/tests/integration/services/reconciliation/llm-settlement-sweep.integration.spec.js#leaves the hold in place, so a live call keeps the credits it reserved origin: none - id: BR-CAP-PFM-LLM-009-03 proved_by: [ATS-PFM-LLM-009-03, TST-PFM-LLM-046] statement: >- Repeating successful recovery does not add another charge or another past-window alert for the same request. kind: invariant enforced_at: - service - database must_prove: true provenance: observed source: 'services-platform:modules/llm/services/reconciliation/llm-settlement-sweep.ts' origin: none - id: BR-CAP-PFM-LLM-009-04 proved_by: [] statement: Each organization’s recovery pass handles a bounded batch and defers remaining requests to another run. kind: policy enforced_at: - service must_prove: false provenance: test source: >- services-platform:modules/llm/tests/integration/services/reconciliation/llm-settlement-sweep.integration.spec.js#settles only as many as the batch allows, so one tenant cannot make a run unbounded origin: none - id: BR-CAP-PFM-LLM-009-05 proved_by: [ATS-PFM-LLM-003-08, ATS-PFM-LLM-009-04, TST-PFM-LLM-045] statement: >- Stale running executions become abandoned with the detection time, no duration and their existing settlement totals. kind: lifecycle enforced_at: - service - database must_prove: true provenance: observed source: 'services-platform:modules/llm/services/reconciliation/llm-settlement-sweep.ts' origin: none - id: BR-CAP-PFM-LLM-009-06 proved_by: [ATS-PFM-LLM-009-05, TST-PFM-LLM-045] statement: >- The drift check compares recently closed executions from the last seven days with settled requests, logs both figures and identities on disagreement, and changes no totals. kind: policy enforced_at: - service must_prove: true provenance: observed source: 'services-platform:modules/llm/services/reconciliation/llm-settlement-sweep.ts' origin: none - id: BR-CAP-PFM-LLM-009-07 proved_by: [] statement: >- A request beyond the recovery window contributes a once-only alert measure, and missing reconciliation runs have a separate alarm. kind: policy enforced_at: - service must_prove: false provenance: observed source: 'services-platform:modules/llm/services/reconciliation/llm-settlement-sweep.ts' origin: none scenarios: - id: SC-CAP-PFM-LLM-009-01 given: A worker stops after provider dispatch and leaves a reservation. when: Recovery processes the eligible request. then: Fallback settlement releases its hold and records the recovery charge. provenance: test source: >- services-platform:modules/llm/tests/integration/services/reconciliation/llm-settlement-sweep.integration.spec.js#releases the hold, which is the whole point of the sweep edge_cases: [] constraints: [] depends_on: - CAP-PFM-LLM-007 - CAP-PFM-LLM-003 related: - CAP-PFM-LLM-008 - CAP-PFM-LLM-010 journeys: [] open_questions: [] evidence: - >- services-platform at ad3f6a6582843008ed0f5a17b131878d15829af1: docs/llm.md (Reconciliation, Executions); modules/llm/services/reconciliation/llm-settlement-sweep.ts, reconciliation-pass.ts; aws/logic/llm-reconciliation.lambda.ts; tests integration/services/reconciliation/llm-settlement-sweep.integration.spec.js, llm-execution-sweep.integration.spec.js. controls: [TC-AI-006, TC-AI-007, TC-AVL-001, TC-AZ-001, TC-AZ-002, TC-AZ-003, TC-AZ-004, TC-DAT-001, TC-DAT-002, TC-DAT-003, TC-LOG-005, TC-MON-001, TC-MON-002, TC-PI-001, TC-PI-005, TC-PRV-001, TC-PRV-002, TC-TEN-001, TC-TEN-002, TC-TEN-003, TC-VEN-001, TC-VEN-002, TC-VEN-003] exercised_by: [FAS-PFM-LLM-03-01, FAS-PFM-LLM-03-02, FAS-PFM-LLM-03-03, FAS-PFM-LLM-03-05] suites: [TST-PFM-LLM-028, TST-PFM-LLM-045, TST-PFM-LLM-046] --- # Recover interrupted requests and identify accounting drift ## Purpose Release stranded reservations and make abandoned tasks visible to Rhythm operators. ## How It Works The hourly function visits operational organizations and inspects old nonterminal requests. It leaves recent work alone so normal conversations can finish. A qualifying old request settles at its stored estimate because recovery does not reconstruct measured usage. Recovery then releases the recorded hold from the allowance period. Separate steps mark stale executions abandoned and compare recent execution counters with their requests. Differences are logged rather than silently repaired. The process uses bounded batches and metrics to expose problems. The product should close the transaction gaps that can make terminal requests invisible to this recovery path. ## Relationships - Admit AI work against the organization’s monthly allowance supplies the prerequisite behavior for this outcome. - Keep one account for a multi-request AI task supplies the prerequisite behavior for this outcome.