Rhythm 2.0 Specifications

Appearance

CAP-PFM-LLM-009Recover interrupted requests and identify accounting drift

Markdown

CAP-PFM-LLM-009

Recover interrupted requests and identify accounting drift

Markdown
  • DocumentDraft
  • Product intentProposed
  • ImplementationObserved

Planning

Feature
FEAT-PFM-LLM-03
Exercised by
FAS-PFM-LLM-03-01, FAS-PFM-LLM-03-02, FAS-PFM-LLM-03-03, FAS-PFM-LLM-03-05
Key concepts
CON-PFM-LLM-001, CON-PFM-LLM-003
Actors
ACT-PFM-012
Depends on
CAP-PFM-LLM-007, CAP-PFM-LLM-003
Related
CAP-PFM-LLM-008, CAP-PFM-LLM-010
Proved by
ATS-PFM-LLM-009

Implementation

Executable suites
TST-PFM-LLM-028, TST-PFM-LLM-045, TST-PFM-LLM-046

KindCapabilityProduct areaPlatformModuleAI Requests, Usage and the Jazz AI TeammateRevision1

Read atservices-platform ad3f6a65

Recover interrupted requests and identify accounting drift ​

Purpose ​

Release stranded reservations and make abandoned tasks visible to Rhythm operators.

How It Works ​

The hourly function visits operational organizations and inspects old nonterminal requests. It leaves recent work alone so normal conversations can finish. A qualifying old request settles at its stored estimate because recovery does not reconstruct measured usage. Recovery then releases the recorded hold from the allowance period. Separate steps mark stale executions abandoned and compare recent execution counters with their requests. Differences are logged rather than silently repaired. The process uses bounded batches and metrics to expose problems. The product should close the transaction gaps that can make terminal requests invisible to this recovery path.

Relationships ​

  • Admit AI work against the organization’s monthly allowance supplies the prerequisite behavior for this outcome.
  • Keep one account for a multi-request AI task supplies the prerequisite behavior for this outcome.

The Formal Contract

The narrative above explains; the sections below are authoritative — the rules, scenarios and tests this document binds.

Outcomes
Recovery settles eligible stranded requests and releases reservations.Stalled tasks receive an abandoned outcome.Task accounting drift is reported for investigation.
Surfaces
job
Postconditions
Eligible stranded work receives recovery outcomes.

Controls

23 controls apply, from the corpus control catalogue.

TC-AI-006 TC-AI-007 TC-AVL-001 TC-AZ-001 TC-AZ-002 TC-AZ-003 TC-AZ-004 TC-DAT-001 TC-DAT-002 TC-DAT-003 TC-LOG-005 TC-MON-001 TC-MON-002 TC-PI-001 TC-PI-005 TC-PRV-001 TC-PRV-002 TC-TEN-001 TC-TEN-002 TC-TEN-003 TC-VEN-001 TC-VEN-002 TC-VEN-003

Business Rules

7 rules, 5 to be proved before approval. A rule's product intent is derived from its origin: 0 confirmed by a product decision, the rest inheriting this capability's proposed.

IdRuleKindEnforced atProvenance
01must prove
Proposed
Hourly reconciliation visits operational organizations and settles requests older than twenty-four hours at their stored estimate without a money figure, releasing their recorded holds.
proved by ATS-PFM-LLM-007-12, ATS-PFM-LLM-008-10, ATS-PFM-LLM-009-01, ATS-PFM-LLM-009-06, TST-PFM-LLM-028, TST-PFM-LLM-046
lifecycleserviceobserved
services-platform:modules/llm/aws/lambda/RECONCILIATION.handler.ts
02must prove
Proposed
Requests inside the recovery window keep their state and reservations.
proved by ATS-PFM-LLM-009-02, TST-PFM-LLM-046
policyservicetest
services-platform:modules/llm/tests/integration/services/reconciliation/llm-settlement-sweep.integration.spec.js#leaves the hold in place, so a live call keeps the credits it reserved
03must prove
Proposed
Repeating successful recovery does not add another charge or another past-window alert for the same request.
proved by ATS-PFM-LLM-009-03, TST-PFM-LLM-046
invariantservice, databaseobserved
services-platform:modules/llm/services/reconciliation/llm-settlement-sweep.ts
04
Proposed
Each organization’s recovery pass handles a bounded batch and defers remaining requests to another run.
policyservicetest
services-platform:modules/llm/tests/integration/services/reconciliation/llm-settlement-sweep.integration.spec.js#settles only as many as the batch allows, so one tenant cannot make a run unbounded
05must prove
Proposed
Stale running executions become abandoned with the detection time, no duration and their existing settlement totals.
proved by ATS-PFM-LLM-003-08, ATS-PFM-LLM-009-04, TST-PFM-LLM-045
lifecycleservice, databaseobserved
services-platform:modules/llm/services/reconciliation/llm-settlement-sweep.ts
06must prove
Proposed
The drift check compares recently closed executions from the last seven days with settled requests, logs both figures and identities on disagreement, and changes no totals.
proved by ATS-PFM-LLM-009-05, TST-PFM-LLM-045
policyserviceobserved
services-platform:modules/llm/services/reconciliation/llm-settlement-sweep.ts
07
Proposed
A request beyond the recovery window contributes a once-only alert measure, and missing reconciliation runs have a separate alarm.
policyserviceobserved
services-platform:modules/llm/services/reconciliation/llm-settlement-sweep.ts

Scenarios

IdGivenWhenThenProvenance
01A worker stops after provider dispatch and leaves a reservation.Recovery processes the eligible request.Fallback settlement releases its hold and records the recovery charge.test
services-platform:modules/llm/tests/integration/services/reconciliation/llm-settlement-sweep.integration.spec.js#releases the hold, which is the whole point of the sweep

The Markdown files under specs/ in rhythmsoftware/rhythm-v2-docs are the canonical source of truth. This reader is a generated, read-only projection of them.

Rhythm Software