---
id: CAP-PFM-LLM-006
title: Check AI behavior with scripted models and deliberate live runs
feature: FEAT-PFM-LLM-02
product_area: platform
module: llm
revision: 1
status: draft
product_intent: proposed
implementation_evidence: observed
supersedes: []
source_sha:
  services-platform: ad3f6a6582843008ed0f5a17b131878d15829af1
intent_sources:
  - 'services-platform:docs/llm.md#The live runs'
purpose: Let engineers prove accounting and conversation behavior without unintentionally buying model calls.
outcomes:
  - Ordinary verification uses scripted provider responses.
  - Accounting tests exercise production pricing calculations.
  - Live checks require a deliberate opt-in and disclose skipped runs.
concepts:
  - CON-PFM-LLM-002
actors:
  - ACT-PFM-034
surfaces:
  - job
preconditions: []
postconditions: []
permissions: []
rules:
  - id: BR-CAP-PFM-LLM-006-01
    proved_by: [ATS-PFM-LLM-006-01, ATS-PFM-LLM-006-05, TST-PFM-LLM-066, TST-PFM-LLM-087]
    statement: >-
      Integration and end-to-end test scripts enable stub mode before test collection, preventing those tiers
      from making real model requests.
    kind: policy
    enforced_at:
      - service
    must_prove: true
    provenance: test
    source: 'services-platform:tests/unit/llm-stub-mode-config.spec.ts#forces stub mode in the integration tier'
    origin: none
  - id: BR-CAP-PFM-LLM-006-02
    proved_by: [ATS-PFM-LLM-005-09, ATS-PFM-LLM-006-02, TST-PFM-LLM-064]
    statement: >-
      The live adapter refuses construction in stub mode or without credentials instead of deferring the
      failure to a provider request.
    kind: validation
    enforced_at:
      - service
    must_prove: true
    provenance: test
    source: >-
      services-platform:modules/llm/tests/unit/runtime/anthropic-provider.spec.ts#refuses, rather than
      reaching a provider a test never meant to call
    origin: none
  - id: BR-CAP-PFM-LLM-006-03
    proved_by: [ATS-PFM-LLM-001-15]
    statement: >-
      Scripted requests use the declared model’s published prices, so ledger tests exercise the production
      cost calculation.
    kind: calculation
    enforced_at:
      - service
    must_prove: false
    provenance: test
    source: >-
      services-platform:modules/llm/tests/integration/lifecycle/run-operation.integration.spec.js#charges what
      that usage costs at the published rate, not the estimate
    origin: none
  - id: BR-CAP-PFM-LLM-006-04
    proved_by: [ATS-PFM-LLM-006-06, TST-PFM-LLM-069]
    statement: >-
      The scripted provider rejects an operation with no script and any requested turn beyond the prepared
      sequence.
    kind: validation
    enforced_at:
      - service
    must_prove: false
    provenance: test
    source: >-
      services-platform:modules/llm/tests/unit/runtime/stub-provider.spec.ts#refuses to answer at all, rather
      than inventing a turn
    origin: none
  - id: BR-CAP-PFM-LLM-006-05
    proved_by: [ATS-PFM-LLM-006-03]
    statement: >-
      Live smoke and evaluation commands require deliberate enablement, a credential source and a test
      database without stub mode; missing prerequisites produce a did-not-run result without reading or
      printing a key.
    kind: validation
    enforced_at:
      - service
    must_prove: true
    provenance: documented
    source: 'services-platform:docs/llm.md#The live runs'
    origin: none
  - id: BR-CAP-PFM-LLM-006-06
    proved_by: []
    statement: >-
      Evaluation compares the records returned by the generated query with expected records, and its exit
      status describes harness execution rather than model correctness.
    kind: policy
    enforced_at:
      - service
    must_prove: false
    provenance: documented
    source: 'services-platform:docs/llm.md#The live runs'
    origin: none
  - id: BR-CAP-PFM-LLM-006-07
    proved_by: [ATS-PFM-LLM-006-04, TST-PFM-LLM-063]
    statement: >-
      Each deployed environment obtains its model key from the same designated secret name within its own
      account, and warm functions refresh that secret after the credential cache expires.
    kind: policy
    enforced_at:
      - service
    must_prove: true
    provenance: test
    source: >-
      services-platform:modules/llm/tests/unit/runtime/anthropic-credentials.spec.ts#refuses that variable
      inside a deployed function, where the secret is the only source
    origin: none
scenarios:
  - id: SC-CAP-PFM-LLM-006-01
    given: Live model use has not been enabled.
    when: The verification run reaches its live check.
    then: The check reports that it did not run and explains why.
    provenance: documented
    source: 'services-platform:docs/llm.md#The live runs'
edge_cases: []
constraints: []
depends_on:
  - CAP-PFM-LLM-005
related:
  - CAP-PFM-LLM-001
journeys: []
open_questions: []
evidence:
  - >-
    services-platform at ad3f6a6582843008ed0f5a17b131878d15829af1: docs/llm.md (Stub mode, The live runs);
    modules/llm/runtime/llm-mode.ts, stub-provider.ts, provider-factory.ts, anthropic.ts,
    anthropic-credentials.ts; tests tests/unit/llm-stub-mode-config.spec.ts,
    unit/runtime/stub-provider.spec.ts, anthropic-provider.spec.ts, anthropic-credentials.spec.ts. The
    live-run gate and scorer live in the query layer's tests and are cited from the module guide only.
controls: [TC-AI-001, TC-AI-002, TC-AI-003, TC-AI-004, TC-AI-006, TC-AI-007, TC-AZ-001, TC-AZ-002, TC-AZ-003, TC-AZ-004, TC-AZ-006, TC-CHG-001, TC-CHG-002, TC-CHG-003, TC-CHG-004, TC-CHG-005, TC-CHG-006, TC-CRY-001, TC-CRY-002, TC-CRY-003, TC-CRY-004, TC-DAT-001, TC-DAT-002, TC-DAT-003, TC-DAT-006, TC-IAM-001, TC-IAM-002, TC-IAM-004, TC-IAM-005, TC-LOG-001, TC-LOG-004, TC-LOG-005, TC-PI-001, TC-PI-005, TC-PRV-001, TC-PRV-002, TC-SEC-001, TC-SEC-002, TC-SEC-003, TC-SEC-004, TC-TEN-001, TC-TEN-002, TC-TEN-003, TC-VEN-001, TC-VEN-002, TC-VEN-003]
exercised_by: [FAS-PFM-LLM-02-01, FAS-PFM-LLM-02-02, FAS-PFM-LLM-02-03, FAS-PFM-LLM-02-04, FAS-PFM-LLM-02-05]
suites: [TST-PFM-LLM-063, TST-PFM-LLM-064, TST-PFM-LLM-066, TST-PFM-LLM-069, TST-PFM-LLM-087]
---

# Check AI behavior with scripted models and deliberate live runs

## Purpose

Let engineers prove accounting and conversation behavior without unintentionally buying model calls.

## How It Works

The integration and API test tiers select a scripted provider before collecting their cases. Engineers supply the replies that each test expects the stand-in to produce. The accounting path still uses published prices, allowing tests to exercise real arithmetic. A live provider refuses construction in a stubbed run. Deliberate smoke checks and evaluations require explicit live configuration. The evaluation compares resulting records rather than demanding identical generated query text. Deployed credentials come from the environment’s designated secret and refresh after a short cache period. These controls establish repeatable engineering evidence without claiming customer acceptance.

## Relationships

- Publish AI operation definitions and historical prices supplies the prerequisite behavior for this outcome.