ATS-PFM-LLM-006 Check AI behavior with scripted models and deliberate live runs: Capability Test Specification
Markdown Design / Platform / AI Requests, Usage and the Jazz AI Teammate / AI Catalog and Pricing / Capability Test Specification Coverage Rule gap: FND-PFM-LLM-065 Transition not-applicable Validation covered Permission not-applicable Edge covered Failure gap: FND-PFM-LLM-065 Boundary gap: FND-PFM-LLM-065 Concurrency gap: FND-PFM-LLM-065 Invariant gap: FND-PFM-LLM-065 Interaction covered Test Scenarios 6 scenarios proving CAP-PFM-LLM-006
01 Test-tier scripts select stub mode early rule unit test
Proves BR-CAP-PFM-LLM-006-01Preconditions Integration and API-tier scripts are available. Steps Inspect their provider-mode setup and collection order. Expected Both set stub mode before collecting tests. Data Fixtures representing integration and API-tier scripts are available. Bound to Source services-platform:tests/unit/llm-stub-mode-config.spec.ts #forces stub mode in the integration tier 03 Unrequested live checks decline to run validation unit negative proposed
Proves BR-CAP-PFM-LLM-006-05Preconditions The live-check command has no explicit live enablement. Steps Invoke that command. Expected It explains that it did not run, returns the configuration result and reads no key. Data Fixtures representing the live-check command has no explicit live enablement. Source services-platform:docs/llm.md #The live runs 04 Warm deployed credentials refresh through the secret rule unit negative test
Proves BR-CAP-PFM-LLM-006-07Preconditions A deployed environment supplies an environment key, and another uses a secret across the cache window. Steps Resolve credentials before and after cache expiry. Expected Deployed environment-key use is refused and secret reads refresh after the window. Data Fixtures representing a deployed environment supplies an environment key, and another uses a secret across the cache window. Bound to Source services-platform:modules/llm/tests/unit/runtime/anthropic-credentials.spec.ts #refuses that variable inside a deployed function, where the secret is the only source 05 Script mode refuses production configuration validation unit negative test
Proves BR-CAP-PFM-LLM-006-01Preconditions The suite owns isolated fixtures representing the valid control and the contrasting case described by its steps. Steps Resolve stub mode in production, staging and development. Expected Production and staging refuse the configuration while development permits scripts. Data Suite-owned fixtures, with a fixed clock where dates or grant windows matter. Bound to Source services-platform:modules/llm/tests/unit/runtime/llm-mode.spec.ts #throws in production rather than quietly answering from a script 06 An exhausted script cannot invent another answer edge unit test
Proves BR-CAP-PFM-LLM-006-04Preconditions The suite owns isolated fixtures representing the valid control and the contrasting case described by its steps. Steps Consume the prepared turns and request another turn. Expected The scripted provider refuses rather than inventing a model answer. Data Suite-owned fixtures, with a fixed clock where dates or grant windows matter. Bound to Source services-platform:modules/llm/tests/unit/runtime/stub-provider.spec.ts #refuses the turn past its end, naming the turn that was asked for ← Previous Capability Test Specification ATS-PFM-LLM-005 · Publish AI operation definitions and historical prices: Capability Test Specification Next Capability Test Specification → ATS-PFM-LLM-007 · undefined