Rhythm 2.0 Specifications

Appearance

ATS-PFM-LLM-001undefined

Markdown

ATS-PFM-LLM-001

undefined

Markdown
  • DocumentProposed

Planning

Proves capability
CAP-PFM-LLM-001

Design

Executed by
TST-PFM-LLM-018, TST-PFM-LLM-020, TST-PFM-LLM-061, TST-PFM-LLM-063, TST-PFM-LLM-075, TST-PFM-LLM-076, TST-PFM-LLM-086

KindAcceptance test setProduct areaPlatformModuleAI Requests, Usage and the Jazz AI TeammateRevision2

Read atservices-platform ad3f6a65react-console baf6dae3

undefined ​

Coverage ​

The scenarios identify the rules or journey assertions they are intended to prove. A test binding names existing evidence; an unbound proposal remains work to verify.

ScenarioTitleLevelProvenance
ATS-PFM-LLM-001-01Catalog admission rejects unknown or unpriced workapiproposed
ATS-PFM-LLM-001-02Interrupted work remains discoverableapiproposed
ATS-PFM-LLM-001-03Settlement distinguishes measured, unknown and absent usageapitest
ATS-PFM-LLM-001-04Persistence gates normal transitions and permits terminal settlementapiproposed
ATS-PFM-LLM-001-05Request attribution comes from contextapiproposed
ATS-PFM-LLM-001-06Refusal response categories remain distinctunittest
ATS-PFM-LLM-001-07Exhausted turns return a bounded resultapitest
ATS-PFM-LLM-001-08Provider failure retains a Rhythm explanationapiproposed
ATS-PFM-LLM-001-09Only the permitted adapter crosses the provider boundaryunittest
ATS-PFM-LLM-001-10Absent credentials have a configuration resultunittest

The interaction register links concrete experiments with bound ai conversations and handle recoverable failures; keep one account for a multi-request ai task; retain redacted ai evidence and remove it safely; publish ai operation definitions and historical prices; check ai behavior with scripted models and deliberate live runs; admit ai work against the organization’s monthly allowance; set an organization’s ai access and recurring limits; give people understandable ai refusal messages; use jazz ai conversations and project folders; refine queries and directory filters with jazz ai. Each experiment checks the outcome at the shared boundary; an unbound experiment remains proposed.

Interruption proof is missing between request creation, reservation metadata, capture metadata, settlement and the separate period movement. Recovery from every early state and a losing cleanup writer need explicit accounting checks.

Coverage

  • Rule gap: FND-PFM-LLM-060
  • Transition gap: FND-PFM-LLM-060
  • Validation covered
  • Permission not-applicable
  • Edge covered
  • Failure covered
  • Boundary gap: FND-PFM-LLM-060
  • Concurrency gap: FND-PFM-LLM-060
  • Invariant gap: FND-PFM-LLM-060
  • Interaction covered

Test Scenarios

22 scenarios proving CAP-PFM-LLM-001; 10 bound to an existing test.

01 Catalog admission rejects unknown or unpriced workvalidationapinegativeproposed
Proves
BR-CAP-PFM-LLM-001-01
Preconditions
A catalog contains a priced operation and a model without a rate.
Steps
Execute the priced operation, resolve the unpriced model and submit an unknown operation.
Expected
Priced work can run; missing prices and unknown keys prevent admission, with the unknown key creating no request.
Data
Authorized and unauthorized fixtures for the scenario, with controlled request and usage records.
Source
services-platform:modules/llm/services/execution/run-operation.ts:runOperation
02 Interrupted work remains discoverablefailureapiproposed
Proves
BR-CAP-PFM-LLM-001-02 INV-DOM-PFM-LLM-001-01
Preconditions
A scripted worker stops after creating its usage record.
Steps
Interrupt the request after creation and invoke recovery when eligible.
Expected
The early-state record survives and recovery can settle it.
Data
Fixtures representing a scripted worker stops after creating its usage record.
Source
services-platform:modules/llm/services/execution/run-operation.ts:runOperation
03 Settlement distinguishes measured, unknown and absent usageruleapitest
Proves
BR-CAP-PFM-LLM-001-03 INV-DOM-PFM-LLM-001-04
Preconditions
Scripts respectively report usage, answer without usage or never answer.
Steps
Execute all three conversations and inspect their charges and holds.
Expected
Measured usage is priced, unknown usage charges the estimate without cents, and unanswered work releases its hold with no charge.
Data
Fixtures representing scripts respectively report usage, answer without usage or never answer.
Bound to
Source
services-platform:modules/llm/tests/integration/lifecycle/run-operation.integration.spec.js#records the invocation as settled on the actual usage
04 Persistence gates normal transitions and permits terminal settlementtransitionapinegativeproposed
Proves
BR-CAP-PFM-LLM-001-04 INV-DOM-PFM-LLM-001-02
Preconditions
Requests occupy normal and terminal persistence states.
Steps
Exercise compare-and-set transitions and settlement from each state.
Expected
Normal transitions require their expected state, while settlement accepts any nonterminal state and rejects a terminal request.
Data
Fixtures representing the declared transition catalog is available.
Source
services-platform:modules/llm/services/execution/run-operation.ts:runOperation
05 Request attribution comes from contextinvariantapiproposed
Proves
BR-CAP-PFM-LLM-001-05 INV-DOM-PFM-LLM-001-06
Preconditions
A signed caller context encloses a declared execution.
Steps
Make a model request with the execution context.
Expected
The request identifies its application, feature, access type, enclosing request and execution from the authoritative context and routing.
Data
Fixtures representing a signed caller context encloses a declared execution.
Source
services-platform:modules/llm/services/execution/run-operation.ts:runOperation
06 Refusal response categories remain distinctruleunitnegativetest
Proves
BR-CAP-PFM-LLM-001-06
Preconditions
Instances represent each supported refusal classification.
Steps
Convert each refusal to its API response.
Expected
Allowance and concurrency map to rate limits, disabled features to conflict, provider problems to distinct unavailable codes and depleted supplier credit to server failure.
Data
Fixtures representing instances represent each supported refusal classification.
Bound to
Source
services-platform:modules/llm/tests/unit/services/llm-metering-errors.spec.ts#answers 429, because it is this tenant’s limit and not a failure
07 Exhausted turns return a bounded resultedgeapitest
Proves
BR-CAP-PFM-LLM-001-07
Preconditions
A scripted conversation continues beyond its allowed turns.
Steps
Execute it under the declared turn budget.
Expected
The feature receives the limit outcome without a final answer and pays for usage already consumed.
Data
Fixtures representing a scripted conversation continues beyond its allowed turns.
Bound to
Source
services-platform:modules/llm/tests/integration/lifecycle/run-operation.integration.spec.js#returns rather than throwing
08 Provider failure retains a Rhythm explanationfailureapinegativeproposed
Proves
BR-CAP-PFM-LLM-001-08
Preconditions
The provider never produces an answer.
Steps
Execute the request and inspect the returned error.
Expected
Rhythm wording identifies the operation and usage request rather than exposing supplier wording.
Data
Fixtures representing the provider never produces an answer.
Source
services-platform:modules/llm/services/execution/run-operation.ts:runOperation
09 Only the permitted adapter crosses the provider boundaryinvariantunitnegativetest
Proves
BR-CAP-PFM-LLM-001-09
Preconditions
Tracked Platform source files are available.
Steps
Run the SDK and provider-host boundary scan across those files.
Expected
The Platform repository permits its designated adapter only; prohibited SDK imports and extra direct provider boundaries are refused. The console prototype remains a separately recorded gap.
Data
Fixtures representing tracked Platform source files are available.
Bound to
Source
services-platform:tests/unit/llm-provider-boundary.spec.ts#reaches the anthropic sdk and its host from the adapter alone
10 Absent credentials have a configuration resultvalidationunitnegativetest
Proves
BR-CAP-PFM-LLM-001-10
Preconditions
Configuration omits the model secret or its usable value.
Steps
Resolve model credentials.
Expected
The result is the dedicated not-configured refusal rather than an allowance classification.
Data
Fixtures representing configuration omits the model secret or its usable value.
Bound to
Source
services-platform:modules/llm/tests/unit/runtime/anthropic-credentials.spec.ts#answers llm_not_configured in the body, so a console says "nobody configured this" and not "try again"
11 Bounded answers retain their chargeinteractionapitest
Proves
BR-CAP-PFM-LLM-001-07 BR-CAP-PFM-LLM-002-01
Preconditions
A metered operation has a two-turn limit and a scripted model that never accepts a final answer.
Steps
Run the operation to its turn limit and read its request and period.
Expected
The bounded outcome and recorded turns agree; settlement charges reported usage and releases the hold.
Data
An isolated workspace with the records, clock instants and caller authority stated in the preconditions.
Bound to
Source
services-platform:modules/llm/tests/integration/lifecycle/run-operation.integration.spec.js#returns rather than throwing,services-platform:modules/llm/tests/integration/lifecycle/run-operation.integration.spec.js#charges for every turn the model did take, because they happened
12 Task totals agree with request settlementinteractionapitest
Proves
BR-CAP-PFM-LLM-001-03 BR-CAP-PFM-LLM-003-05
Preconditions
A declared task runs three metered requests on its supplied context.
Steps
Complete the requests and compare task, request and period totals.
Expected
Only associated settlements contribute to the task; the period charge equals the sum of those requests.
Data
An isolated workspace with the records, clock instants and caller authority stated in the preconditions.
Bound to
Source
services-platform:modules/llm/tests/integration/lifecycle/executions.integration.spec.js#totals the credits the three settlements charged,services-platform:modules/llm/tests/integration/lifecycle/executions.integration.spec.js#charges the period exactly what the execution totalled
13 Capture follows metered executioninteractionapiproposed
Proves
BR-CAP-PFM-LLM-001-02 BR-CAP-PFM-LLM-004-01
Preconditions
A scripted request contains a credential field and ordinary question text.
Steps
Run it and inspect both capture results through an eligible diagnostic reader.
Expected
The request and response are captured around dispatch; the credential is absent and ordinary text remains.
Data
An isolated workspace with the records, clock instants and caller authority stated in the preconditions.
Source
services-platform:docs/llm.md#The order of the steps, which is the design,services-platform:modules/llm/services/execution/run-operation.ts:recordCapture
14 Published prices determine the next requestinteractionapiproposed
Proves
BR-CAP-PFM-LLM-001-01 BR-CAP-PFM-LLM-005-05
Preconditions
An operation has a published rate and one settled request.
Steps
Publish a changed rate, resolve the operation afresh and run the same reported usage again.
Expected
The earlier request keeps its rate reference; the new request uses the new rate after a fresh resolution.
Data
An isolated workspace with the records, clock instants and caller authority stated in the preconditions.
Source
services-platform:modules/llm/services/catalog/catalog-authority.ts:resolveLlmOperation,services-platform:modules/llm/tests/integration/services/persistence/rate-versions.integration.spec.js#leaves the old version at the old price, because an invocation settled under it still names it
15 Scripted execution uses production accountinginteractionapiproposed
Proves
BR-CAP-PFM-LLM-001-03 BR-CAP-PFM-LLM-006-03
Preconditions
Stub mode is enabled before collection and a declared operation has a script.
Steps
Execute the script through the shared request boundary and read settlement.
Expected
The script supplies model usage without a network call and settlement uses the published prices.
Data
An isolated workspace with the records, clock instants and caller authority stated in the preconditions.
Source
services-platform:modules/llm/services/execution/run-operation.ts:settleInvocation,services-platform:modules/llm/tests/integration/lifecycle/run-operation.integration.spec.js#charges what that usage costs at the published rate, not the estimate
16 Refusal creates history without spendinginteractionapinegativetest
Proves
BR-CAP-PFM-LLM-001-06 BR-CAP-PFM-LLM-007-03
Preconditions
A first scripted request consumes the available allowance.
Steps
Run a second request and inspect its record, period and unused script.
Expected
The refusal retains needed and remaining capacity without consuming the next turn, adding a hold or charging credits.
Data
An isolated workspace with the records, clock instants and caller authority stated in the preconditions.
Bound to
Source
services-platform:modules/llm/tests/integration/lifecycle/run-operation.integration.spec.js#never asked the provider, so a tenant out of credit costs nothing to refuse,services-platform:modules/llm/tests/integration/lifecycle/run-operation.integration.spec.js#holds nothing for the refused call
17 Global disablement stops metered dispatchinteractionapinegativeproposed
Proves
BR-CAP-PFM-LLM-001-06 BR-CAP-PFM-LLM-013-09
Preconditions
The operation has an active exemption but organization-wide AI is disabled.
Steps
Invoke the operation and inspect refusal history and the period.
Expected
Global disablement prevents dispatch and records the disabled-feature refusal without holding credits.
Data
An isolated workspace with the records, clock instants and caller authority stated in the preconditions.
Source
services-platform:modules/llm/services/metering/llm-metering-errors.ts,services-platform:modules/llm/tests/integration/services/limits/tenant-limits.integration.spec.js#refuses it when AI is off, whatever exemption is in force
18 Refusal wording follows the metering classificationinteractionapiproposed
Proves
BR-CAP-PFM-LLM-001-06 BR-CAP-PFM-LLM-018-01
Preconditions
Allowance, running-slot and unavailable-provider refusals are represented.
Steps
Run each refusal through the customer sentence helper.
Expected
The customer receives the corresponding renewal, short-wait or temporary-service guidance without supplier details.
Data
An isolated workspace with the records, clock instants and caller authority stated in the preconditions.
Source
services-platform:modules/llm/services/metering/llm-metering-errors.ts,services-platform:modules/llm/tests/unit/services/usage/usage-sentences.spec.ts#tells an organization that has used its allowance when it resumes
19 General chat exposes the unmetered boundaryinteractionbrowsernegativeproposed
Proves
BR-CAP-PFM-LLM-001-09 BR-CAP-PFM-LLM-019-02
Preconditions
A signed-in person has a general conversation while organization AI is off.
Steps
Send a general question and inspect Platform request history before and after.
Expected
The prototype reply can complete without a Platform request, hold or settlement; this remains a release gap.
Data
An isolated workspace with the records, clock instants and caller authority stated in the preconditions.
Source
services-platform:tests/unit/llm-provider-boundary.spec.ts#refuses an import of the openai sdk even in the adapter,react-console:app/api/agent/conversations/[id]/messages/route.ts:MODEL
20 Live query refusal preserves unsaved workinteractionbrowsernegativeproposed
Proves
BR-CAP-PFM-LLM-001-06 BR-CAP-PFM-LLM-020-02
Preconditions
A live query has chosen columns and insufficient allowance for interpretation.
Steps
Ask scoped Jazz AI to refine it and inspect both the query and metered request.
Expected
The refusal has a usage record while the query remains unchanged and unsaved.
Data
An isolated workspace with the records, clock instants and caller authority stated in the preconditions.
Source
services-platform:modules/llm/services/metering/llm-metering-errors.ts,react-console:components/queries/query-workspace.tsx:refine
21 Terminal request states have no successorstransitionunittest
Proves
BR-CAP-PFM-LLM-001-04
Preconditions
The suite owns isolated fixtures representing the valid control and the contrasting case described by its steps.
Steps
Inspect successors of settled and refused requests in the declared move list.
Expected
Neither terminal state allows another normal transition.
Data
Suite-owned fixtures, with a fixed clock where dates or grant windows matter.
Bound to
Source
services-platform:modules/llm/tests/unit/db/invocation-transitions.spec.ts#lets nothing follow settled, so a settled call can never be charged a second time
22 Allowance and provider faults have separate measuresruleunittest
Proves
BR-CAP-PFM-LLM-001-13
Preconditions
The suite owns isolated fixtures representing the valid control and the contrasting case described by its steps.
Steps
Emit recorded allowance and provider refusal measures through recording telemetry.
Expected
Each measure retains its own name, value and operational properties.
Data
Suite-owned fixtures, with a fixed clock where dates or grant windows matter.
Bound to
Source
services-platform:modules/llm/tests/unit/services/llm-metrics.spec.ts#counts the refusal under its reason

The Markdown files under specs/ in rhythmsoftware/rhythm-v2-docs are the canonical source of truth. This reader is a generated, read-only projection of them.

Rhythm Software