--- id: ATS-PFM-LLM-006 scope: CAP-PFM-LLM-006 product_area: platform module: llm revision: 2 status: proposed source_sha: services-platform: ad3f6a6582843008ed0f5a17b131878d15829af1 scenarios: - id: ATS-PFM-LLM-006-01 title: Test-tier scripts select stub mode early level: unit proves: - BR-CAP-PFM-LLM-006-01 preconditions: Integration and API-tier scripts are available. steps: Inspect their provider-mode setup and collection order. expected: Both set stub mode before collecting tests. negative: false data_needs: Fixtures representing integration and API-tier scripts are available. bound_to: - TST-PFM-LLM-087 provenance: test source: 'services-platform:tests/unit/llm-stub-mode-config.spec.ts#forces stub mode in the integration tier' category: rule test_sources: - 'services-platform:tests/unit/llm-stub-mode-config.spec.ts#forces stub mode in the integration tier' - 'services-platform:tests/unit/llm-stub-mode-config.spec.ts#forces stub mode in the end-to-end tier' - >- services-platform:tests/unit/llm-stub-mode-config.spec.ts#sets the flag before mocha, so it is in the environment of every spec the tier collects - id: ATS-PFM-LLM-006-02 title: Live adapter construction rejects unsafe setup level: unit proves: - BR-CAP-PFM-LLM-006-02 preconditions: One run is stubbed and another has no key. steps: Construct the real provider in each. expected: Both fail at construction. negative: true data_needs: Fixtures representing one run is stubbed and another has no key. bound_to: - TST-PFM-LLM-064 provenance: test source: >- services-platform:modules/llm/tests/unit/runtime/anthropic-provider.spec.ts#refuses, rather than reaching a provider a test never meant to call category: validation test_sources: - >- services-platform:modules/llm/tests/unit/runtime/anthropic-provider.spec.ts#refuses, rather than reaching a provider a test never meant to call - >- services-platform:modules/llm/tests/unit/runtime/anthropic-provider.spec.ts#refuses at construction rather than on the first request - id: ATS-PFM-LLM-006-03 title: Unrequested live checks decline to run level: unit proves: - BR-CAP-PFM-LLM-006-05 preconditions: The live-check command has no explicit live enablement. steps: Invoke that command. expected: 'It explains that it did not run, returns the configuration result and reads no key.' negative: true data_needs: Fixtures representing the live-check command has no explicit live enablement. bound_to: [] provenance: proposed source: 'services-platform:docs/llm.md#The live runs' category: validation test_sources: [] - id: ATS-PFM-LLM-006-04 title: Warm deployed credentials refresh through the secret level: unit proves: - BR-CAP-PFM-LLM-006-07 preconditions: 'A deployed environment supplies an environment key, and another uses a secret across the cache window.' steps: Resolve credentials before and after cache expiry. expected: Deployed environment-key use is refused and secret reads refresh after the window. negative: true data_needs: >- Fixtures representing a deployed environment supplies an environment key, and another uses a secret across the cache window. bound_to: - TST-PFM-LLM-063 provenance: test source: >- services-platform:modules/llm/tests/unit/runtime/anthropic-credentials.spec.ts#refuses that variable inside a deployed function, where the secret is the only source category: rule test_sources: - >- services-platform:modules/llm/tests/unit/runtime/anthropic-credentials.spec.ts#refuses that variable inside a deployed function, where the secret is the only source - >- services-platform:modules/llm/tests/unit/runtime/anthropic-credentials.spec.ts#reads it again once the window has passed, which is how a rotation reaches a warm function - id: ATS-PFM-LLM-006-05 title: Script mode refuses production configuration category: validation level: unit proves: - BR-CAP-PFM-LLM-006-01 preconditions: >- The suite owns isolated fixtures representing the valid control and the contrasting case described by its steps. steps: 'Resolve stub mode in production, staging and development.' expected: Production and staging refuse the configuration while development permits scripts. negative: true data_needs: 'Suite-owned fixtures, with a fixed clock where dates or grant windows matter.' bound_to: - TST-PFM-LLM-066 provenance: test source: >- services-platform:modules/llm/tests/unit/runtime/llm-mode.spec.ts#throws in production rather than quietly answering from a script test_sources: - >- services-platform:modules/llm/tests/unit/runtime/llm-mode.spec.ts#throws in production rather than quietly answering from a script - id: ATS-PFM-LLM-006-06 title: An exhausted script cannot invent another answer category: edge level: unit proves: - BR-CAP-PFM-LLM-006-04 preconditions: >- The suite owns isolated fixtures representing the valid control and the contrasting case described by its steps. steps: Consume the prepared turns and request another turn. expected: The scripted provider refuses rather than inventing a model answer. negative: false data_needs: 'Suite-owned fixtures, with a fixed clock where dates or grant windows matter.' bound_to: - TST-PFM-LLM-069 provenance: test source: >- services-platform:modules/llm/tests/unit/runtime/stub-provider.spec.ts#refuses the turn past its end, naming the turn that was asked for test_sources: - >- services-platform:modules/llm/tests/unit/runtime/stub-provider.spec.ts#refuses the turn past its end, naming the turn that was asked for evidence: - >- services-platform at ad3f6a6582843008ed0f5a17b131878d15829af1: bound or cited docs/llm.md; modules/llm/tests/unit/runtime/anthropic-credentials.spec.ts; modules/llm/tests/unit/runtime/anthropic-provider.spec.ts; tests/unit/llm-stub-mode-config.spec.ts. coverage: rule: 'gap: FND-PFM-LLM-065' transition: not-applicable validation: covered permission: not-applicable edge: covered failure: 'gap: FND-PFM-LLM-065' boundary: 'gap: FND-PFM-LLM-065' concurrency: 'gap: FND-PFM-LLM-065' invariant: 'gap: FND-PFM-LLM-065' interaction: covered coverage_notes: transition: This capability defines read-only presentation or validation and introduces no owned lifecycle transition. permission: >- This capability declares no permission key; caller authorization belongs to its consuming operation or a related capability. interaction: >- The related interaction register cites a concrete experiment for every dependency and related capability, including cross-capability workflow seams. invariant: >- The domain-invariant register maps each relevant invariant to a scenario or to the capability coverage finding; an example does not establish every invariant. rule: >- All must-prove rules have scenario references, but the coverage finding retains untested clauses and combined outcomes; scenario presence alone is not exhaustive rule proof. related_interactions: - with: CAP-PFM-LLM-001 scenarios: - ATS-PFM-LLM-001-15 source: 'services-platform:modules/llm/services/catalog/catalog-authority.ts:resolveLlmOperation' - with: CAP-PFM-LLM-005 scenarios: - ATS-PFM-LLM-005-09 source: 'services-platform:modules/llm/catalog/pricing.ts' state_models: [] coverage_obligations: must_prove: - rule: BR-CAP-PFM-LLM-006-01 scenarios: - ATS-PFM-LLM-006-01 - ATS-PFM-LLM-006-05 source: 'services-platform:tests/unit/llm-stub-mode-config.spec.ts#forces stub mode in the integration tier' coverage: specified - rule: BR-CAP-PFM-LLM-006-02 scenarios: - ATS-PFM-LLM-006-02 source: >- services-platform:modules/llm/tests/unit/runtime/anthropic-provider.spec.ts#refuses, rather than reaching a provider a test never meant to call coverage: specified - rule: BR-CAP-PFM-LLM-006-05 scenarios: - ATS-PFM-LLM-006-03 source: 'services-platform:docs/llm.md#The live runs' coverage: partial gap: FND-PFM-LLM-065 - rule: BR-CAP-PFM-LLM-006-07 scenarios: - ATS-PFM-LLM-006-04 source: >- services-platform:modules/llm/tests/unit/runtime/anthropic-credentials.spec.ts#refuses that variable inside a deployed function, where the secret is the only source coverage: partial gap: FND-PFM-LLM-065 state_transitions: [] domain_invariants: - invariant: INV-DOM-PFM-LLM-002-05 scenarios: [] gap: FND-PFM-LLM-065 provenance: proposed source: 'services-platform:modules/llm/catalog/pricing.ts:reservationCredits' - invariant: INV-DOM-PFM-LLM-002-06 scenarios: [] gap: FND-PFM-LLM-065 provenance: proposed source: 'services-platform:modules/llm/catalog/pricing.ts:convertUsageToCredits' coverage_limits: finding: FND-PFM-LLM-065 statement: >- Deliberate smoke and evaluation enablement, absent prerequisites, output safety and live-run boundaries need executable proof beyond adapter construction and script configuration. provenance: proposed source: - 'services-platform:tests/unit/llm-stub-mode-config.spec.ts#forces stub mode in the integration tier' - >- services-platform:modules/llm/tests/unit/runtime/anthropic-provider.spec.ts#refuses, rather than reaching a provider a test never meant to call - 'services-platform:docs/llm.md#The live runs' - >- services-platform:modules/llm/tests/unit/runtime/anthropic-credentials.spec.ts#refuses that variable inside a deployed function, where the secret is the only source - 'services-platform:modules/llm/catalog/pricing.ts:reservationCredits' - 'services-platform:modules/llm/catalog/pricing.ts:convertUsageToCredits' invariants_without_scenarios: - INV-DOM-PFM-LLM-002-05 - INV-DOM-PFM-LLM-002-06 --- # Check AI behavior with scripted models and deliberate live runs: Capability Test Specification ## Coverage | Rule | Scenario | |---|---| | `BR-CAP-PFM-LLM-006-01` | `ATS-PFM-LLM-006-01` | | `BR-CAP-PFM-LLM-006-02` | `ATS-PFM-LLM-006-02` | | `BR-CAP-PFM-LLM-006-05` | `ATS-PFM-LLM-006-03` | | `BR-CAP-PFM-LLM-006-07` | `ATS-PFM-LLM-006-04` | ## Interactions With Related Capabilities The interaction register links concrete experiments with run ai requests with consistent attribution and accounting; publish ai operation definitions and historical prices. Each experiment checks the outcome at the shared boundary; an unbound experiment remains proposed. Deliberate smoke and evaluation enablement, absent prerequisites, output safety and live-run boundaries need executable proof beyond adapter construction and script configuration.