Rhythm 2.0 Specifications

Appearance

CAP-PFM-LLM-006Check AI behavior with scripted models and deliberate live runs

Markdown

CAP-PFM-LLM-006

Check AI behavior with scripted models and deliberate live runs

Markdown
  • DocumentDraft
  • Product intentProposed
  • ImplementationObserved

Planning

Feature
FEAT-PFM-LLM-02
Exercised by
FAS-PFM-LLM-02-01, FAS-PFM-LLM-02-02, FAS-PFM-LLM-02-03, FAS-PFM-LLM-02-04, FAS-PFM-LLM-02-05
Key concepts
CON-PFM-LLM-002
Actors
ACT-PFM-034
Depends on
CAP-PFM-LLM-005
Related
CAP-PFM-LLM-001
Proved by
ATS-PFM-LLM-006

Implementation

Executable suites
TST-PFM-LLM-063, TST-PFM-LLM-064, TST-PFM-LLM-066, TST-PFM-LLM-069, TST-PFM-LLM-087

KindCapabilityProduct areaPlatformModuleAI Requests, Usage and the Jazz AI TeammateRevision1

Read atservices-platform ad3f6a65

Check AI behavior with scripted models and deliberate live runs ​

Purpose ​

Let engineers prove accounting and conversation behavior without unintentionally buying model calls.

How It Works ​

The integration and API test tiers select a scripted provider before collecting their cases. Engineers supply the replies that each test expects the stand-in to produce. The accounting path still uses published prices, allowing tests to exercise real arithmetic. A live provider refuses construction in a stubbed run. Deliberate smoke checks and evaluations require explicit live configuration. The evaluation compares resulting records rather than demanding identical generated query text. Deployed credentials come from the environment’s designated secret and refresh after a short cache period. These controls establish repeatable engineering evidence without claiming customer acceptance.

Relationships ​

  • Publish AI operation definitions and historical prices supplies the prerequisite behavior for this outcome.

The Formal Contract

The narrative above explains; the sections below are authoritative — the rules, scenarios and tests this document binds.

Outcomes
Ordinary verification uses scripted provider responses.Accounting tests exercise production pricing calculations.Live checks require a deliberate opt-in and disclose skipped runs.
Surfaces
job
Intent sources
services-platform:docs/llm.md#The live runs

Controls

46 controls apply, from the corpus control catalogue.

TC-AI-001 TC-AI-002 TC-AI-003 TC-AI-004 TC-AI-006 TC-AI-007 TC-AZ-001 TC-AZ-002 TC-AZ-003 TC-AZ-004 TC-AZ-006 TC-CHG-001 TC-CHG-002 TC-CHG-003 TC-CHG-004 TC-CHG-005 TC-CHG-006 TC-CRY-001 TC-CRY-002 TC-CRY-003 TC-CRY-004 TC-DAT-001 TC-DAT-002 TC-DAT-003 TC-DAT-006 TC-IAM-001 TC-IAM-002 TC-IAM-004 TC-IAM-005 TC-LOG-001 TC-LOG-004 TC-LOG-005 TC-PI-001 TC-PI-005 TC-PRV-001 TC-PRV-002 TC-SEC-001 TC-SEC-002 TC-SEC-003 TC-SEC-004 TC-TEN-001 TC-TEN-002 TC-TEN-003 TC-VEN-001 TC-VEN-002 TC-VEN-003

Business Rules

7 rules, 4 to be proved before approval. A rule's product intent is derived from its origin: 0 confirmed by a product decision, the rest inheriting this capability's proposed.

IdRuleKindEnforced atProvenance
01must prove
Proposed
Integration and end-to-end test scripts enable stub mode before test collection, preventing those tiers from making real model requests.
proved by ATS-PFM-LLM-006-01, ATS-PFM-LLM-006-05, TST-PFM-LLM-066, TST-PFM-LLM-087
policyservicetest
services-platform:tests/unit/llm-stub-mode-config.spec.ts#forces stub mode in the integration tier
02must prove
Proposed
The live adapter refuses construction in stub mode or without credentials instead of deferring the failure to a provider request.
proved by ATS-PFM-LLM-005-09, ATS-PFM-LLM-006-02, TST-PFM-LLM-064
validationservicetest
services-platform:modules/llm/tests/unit/runtime/anthropic-provider.spec.ts#refuses, rather than reaching a provider a test never meant to call
03
Proposed
Scripted requests use the declared model’s published prices, so ledger tests exercise the production cost calculation.
proved by ATS-PFM-LLM-001-15
calculationservicetest
services-platform:modules/llm/tests/integration/lifecycle/run-operation.integration.spec.js#charges what that usage costs at the published rate, not the estimate
04
Proposed
The scripted provider rejects an operation with no script and any requested turn beyond the prepared sequence.
proved by ATS-PFM-LLM-006-06, TST-PFM-LLM-069
validationservicetest
services-platform:modules/llm/tests/unit/runtime/stub-provider.spec.ts#refuses to answer at all, rather than inventing a turn
05must prove
Proposed
Live smoke and evaluation commands require deliberate enablement, a credential source and a test database without stub mode; missing prerequisites produce a did-not-run result without reading or printing a key.
proved by ATS-PFM-LLM-006-03
validationservicedocumented
services-platform:docs/llm.md#The live runs
06
Proposed
Evaluation compares the records returned by the generated query with expected records, and its exit status describes harness execution rather than model correctness.
policyservicedocumented
services-platform:docs/llm.md#The live runs
07must prove
Proposed
Each deployed environment obtains its model key from the same designated secret name within its own account, and warm functions refresh that secret after the credential cache expires.
proved by ATS-PFM-LLM-006-04, TST-PFM-LLM-063
policyservicetest
services-platform:modules/llm/tests/unit/runtime/anthropic-credentials.spec.ts#refuses that variable inside a deployed function, where the secret is the only source

Scenarios

IdGivenWhenThenProvenance
01Live model use has not been enabled.The verification run reaches its live check.The check reports that it did not run and explains why.documented
services-platform:docs/llm.md#The live runs

The Markdown files under specs/ in rhythmsoftware/rhythm-v2-docs are the canonical source of truth. This reader is a generated, read-only projection of them.

Rhythm Software