--- id: ATS-PFM-LLM-002 scope: CAP-PFM-LLM-002 product_area: platform module: llm revision: 2 status: proposed source_sha: services-platform: ad3f6a6582843008ed0f5a17b131878d15829af1 scenarios: - id: ATS-PFM-LLM-002-01 title: Turn budget ends an unfinished conversation level: unit proves: - BR-CAP-PFM-LLM-002-01 preconditions: A scripted model never offers an acceptable final answer. steps: Run with a small turn allowance. expected: Exactly the allowed requests execute before the loop returns its turn-limit outcome. negative: true data_needs: Fixtures representing a scripted model never offers an acceptable final answer. bound_to: - TST-PFM-LLM-068 provenance: test source: >- services-platform:modules/llm/tests/unit/runtime/run-tool-loop.spec.ts#stops a conversation that will not converge category: rule test_sources: - >- services-platform:modules/llm/tests/unit/runtime/run-tool-loop.spec.ts#stops a conversation that will not converge - >- services-platform:modules/llm/tests/unit/runtime/run-tool-loop.spec.ts#made exactly the allowed number of provider requests - id: ATS-PFM-LLM-002-02 title: Oversized tool batches run no handlers level: unit proves: - BR-CAP-PFM-LLM-002-02 preconditions: A nonfinal turn asks for more tools than remain. steps: Execute that conversation. expected: The batch ends the loop and none of its handlers runs. negative: true data_needs: Fixtures representing a nonfinal turn asks for more tools than remain. bound_to: - TST-PFM-LLM-068 provenance: test source: >- services-platform:modules/llm/tests/unit/runtime/run-tool-loop.spec.ts#runs no handler for the turn it refused, rather than half of it category: edge test_sources: - >- services-platform:modules/llm/tests/unit/runtime/run-tool-loop.spec.ts#runs no handler for the turn it refused, rather than half of it - id: ATS-PFM-LLM-002-03 title: Credit guard uses strict excess level: unit proves: - BR-CAP-PFM-LLM-002-03 preconditions: Two nonfinal tool turns together cost more than the hold. steps: Execute both turns and inspect the ending and usage. expected: 'The loop stops after strict excess, preserving the charge of the crossing turn.' negative: true data_needs: Fixtures representing two nonfinal tool turns together cost more than the hold. bound_to: - TST-PFM-LLM-068 provenance: test source: >- services-platform:modules/llm/tests/unit/runtime/run-tool-loop.spec.ts#stops when the reservation is spent, whether or not the next turn would have converged category: boundary test_sources: - >- services-platform:modules/llm/tests/unit/runtime/run-tool-loop.spec.ts#stops when the reservation is spent, whether or not the next turn would have converged - >- services-platform:modules/llm/tests/unit/runtime/run-tool-loop.spec.ts#reports what was actually spent, which is over the limit by the turn that crossed it - id: ATS-PFM-LLM-002-04 title: No remaining time means no provider request level: unit proves: - BR-CAP-PFM-LLM-002-04 preconditions: The next turn has no conversation time left. steps: Start the loop under that time budget. expected: No request is sent for the expired turn. negative: true data_needs: Fixtures representing the next turn has no conversation time left. bound_to: - TST-PFM-LLM-068 provenance: test source: >- services-platform:modules/llm/tests/unit/runtime/run-tool-loop.spec.ts#stops before a turn it has no time left for category: boundary test_sources: - >- services-platform:modules/llm/tests/unit/runtime/run-tool-loop.spec.ts#stops before a turn it has no time left for - >- services-platform:modules/llm/tests/unit/runtime/run-tool-loop.spec.ts#makes no provider request at all in that case - id: ATS-PFM-LLM-002-05 title: Retry classification separates recoverable faults level: unit proves: - BR-CAP-PFM-LLM-002-05 preconditions: >- Provider examples include timeout, rate limit, server errors, malformed and unauthorized requests, connection reset and abort. steps: Classify each example with the retry policy. expected: Transient faults permit retry while permanent request or authority faults do not. negative: false data_needs: >- Fixtures representing provider examples include timeout, rate limit, server errors, malformed and unauthorized requests, connection reset and abort. bound_to: - TST-PFM-LLM-067 provenance: test source: >- services-platform:modules/llm/tests/unit/runtime/retry-policy.spec.ts#retries a 529, which is the provider telling us it is overloaded category: failure test_sources: - >- services-platform:modules/llm/tests/unit/runtime/retry-policy.spec.ts#retries a 529, which is the provider telling us it is overloaded - >- services-platform:modules/llm/tests/unit/runtime/retry-policy.spec.ts#does not retry a 400, because the request itself is wrong - >- services-platform:modules/llm/tests/unit/runtime/retry-policy.spec.ts#retries an abort, and records that the attempt ran out of time - id: ATS-PFM-LLM-002-06 title: Depleted provider credit does not retry level: unit proves: - BR-CAP-PFM-LLM-002-06 preconditions: A provider rate-limit response describes an exhausted balance. steps: Resolve its retry classification. expected: It is credit exhaustion with retry disabled. negative: true data_needs: Fixtures representing a provider rate-limit response describes an exhausted balance. bound_to: - TST-PFM-LLM-067 provenance: test source: >- services-platform:modules/llm/tests/unit/runtime/retry-policy.spec.ts#is not retried, because retrying it reads a purchase problem as load category: failure test_sources: - >- services-platform:modules/llm/tests/unit/runtime/retry-policy.spec.ts#is not retried, because retrying it reads a purchase problem as load - >- services-platform:modules/llm/tests/unit/runtime/retry-policy.spec.ts#is classified as credit exhaustion, so a caller can branch without knowing the provider’s strings - id: ATS-PFM-LLM-002-07 title: Retries retain their own usage prices level: api proves: - BR-CAP-PFM-LLM-002-07 preconditions: The first provider attempt fails and its retry succeeds. steps: Execute the request and inspect both attempt records. expected: >- Both attempts share a turn with distinct retry numbers; missing usage costs zero and reported retry usage receives its own price. negative: false data_needs: Fixtures representing the first provider attempt fails and its retry succeeds. bound_to: - TST-PFM-LLM-018 - TST-PFM-LLM-020 provenance: test source: >- services-platform:modules/llm/tests/integration/lifecycle/executions.integration.spec.js#prices the failed attempt, which reported nothing, at zero category: rule test_sources: - >- services-platform:modules/llm/tests/integration/lifecycle/executions.integration.spec.js#prices the failed attempt, which reported nothing, at zero - >- services-platform:modules/llm/tests/integration/lifecycle/executions.integration.spec.js#prices the retry from its own reported usage at the invocation’s rate version - >- services-platform:modules/llm/tests/integration/lifecycle/run-operation.integration.spec.js#numbers the retry as a retry of the same turn, not as a second turn - id: ATS-PFM-LLM-002-08 title: Rejected answers can be repaired level: unit proves: - BR-CAP-PFM-LLM-002-10 preconditions: The feature rejects one final answer and accepts the next. steps: Run the loop with that answer judge. expected: >- The model receives the rejection as a tool error, the rejected call uses tool budget and the accepted repair completes. negative: false data_needs: Fixtures representing the feature rejects one final answer and accepts the next. bound_to: - TST-PFM-LLM-068 provenance: test source: >- services-platform:modules/llm/tests/unit/runtime/run-tool-loop.spec.ts#answers the rejected call with what the consumer gave it, as that call’s own result category: validation test_sources: - >- services-platform:modules/llm/tests/unit/runtime/run-tool-loop.spec.ts#answers the rejected call with what the consumer gave it, as that call’s own result - >- services-platform:modules/llm/tests/unit/runtime/run-tool-loop.spec.ts#counts the rejected call against the tool-call budget, because it was answered like any other - >- services-platform:modules/llm/tests/unit/runtime/run-tool-loop.spec.ts#reads the throw as a rejection rather than losing the account of what the turns cost - id: ATS-PFM-LLM-002-09 title: Cache marking remains within the provider bound level: unit proves: - BR-CAP-PFM-LLM-002-12 preconditions: Stable content requests more cache markers than the provider accepts. steps: Construct the provider request. expected: At most four markers remain and the marked content text is preserved. negative: false data_needs: Fixtures representing stable content requests more cache markers than the provider accepts. bound_to: - TST-PFM-LLM-064 provenance: test source: >- services-platform:modules/llm/tests/unit/runtime/anthropic-provider.spec.ts#never sends more than the four Anthropic takes, however many blocks asked to be cached category: boundary test_sources: - >- services-platform:modules/llm/tests/unit/runtime/anthropic-provider.spec.ts#never sends more than the four Anthropic takes, however many blocks asked to be cached - >- services-platform:modules/llm/tests/unit/runtime/anthropic-provider.spec.ts#sends the marked block’s text exactly as it was, so marking it changes nothing the model reads - id: ATS-PFM-LLM-002-10 title: Final acceptance precedes later budget guards level: unit proves: - BR-CAP-PFM-LLM-002-14 preconditions: A turn offers an acceptable final answer alongside excess cost and requested tools. steps: Execute the turn and accept its final answer. expected: >- Completion wins over the later tool and credit checks; ordinary tools do not run and the accepted answer adds no answered-tool count. negative: false data_needs: >- Fixtures representing a turn offers an acceptable final answer alongside excess cost and requested tools. bound_to: [] provenance: proposed source: 'services-platform:modules/llm/runtime/run-tool-loop.ts:runToolLoop' category: edge test_sources: [] - id: ATS-PFM-LLM-002-11 title: Missing tool handlers return a repairable error level: unit proves: - BR-CAP-PFM-LLM-002-15 preconditions: A model asks for a tool with no handler before offering an answer. steps: Execute with that handler absent. expected: A tool error reaches the model and a later turn can complete within the remaining budgets. negative: false data_needs: Fixtures representing a model asks for a tool with no handler before offering an answer. bound_to: [] provenance: proposed source: 'services-platform:modules/llm/runtime/run-tool-loop.ts:runToolLoop' category: failure test_sources: [] - id: ATS-PFM-LLM-002-12 title: Historical budgets explain a bounded conversation category: interaction level: api proves: - BR-CAP-PFM-LLM-002-01 - BR-CAP-PFM-LLM-005-03 exercises: - CAP-PFM-LLM-002 - CAP-PFM-LLM-005 preconditions: A request ends at its published turn limit and the next publication increases that limit. steps: Read the earlier request and run a freshly resolved scripted request. expected: The earlier ending retains its original routing budget; the later request uses the changed budget. negative: false data_needs: 'An isolated workspace with the records, clock instants and caller authority stated in the preconditions.' bound_to: [] provenance: proposed source: - >- services-platform:modules/llm/tests/unit/runtime/run-tool-loop.spec.ts#stops a conversation that will not converge - 'services-platform:modules/llm/services/catalog/materialize-catalog.ts' - id: ATS-PFM-LLM-002-13 title: Attempt history explains retry outcomes category: interaction level: api proves: - BR-CAP-PFM-LLM-002-07 - BR-CAP-PFM-LLM-012-06 exercises: - CAP-PFM-LLM-002 - CAP-PFM-LLM-012 preconditions: A scripted request first receives overload and then succeeds; another ends at its turn limit. steps: Read retained attempts and explain the limited request. expected: >- Retry numbering and costs remain distinct from turns, and the explanation names the retained routing limit. negative: false data_needs: 'An isolated workspace with the records, clock instants and caller authority stated in the preconditions.' bound_to: [] provenance: proposed source: - 'services-platform:modules/llm/services/persistence/invocations.ts:recordInvocationAttempt' - 'services-platform:modules/llm/services/explain/explain-invocation.ts' - id: ATS-PFM-LLM-002-14 title: Retry exhaustion reports every attempt category: failure level: unit proves: - BR-CAP-PFM-LLM-002-05 preconditions: >- The suite owns isolated fixtures representing the valid control and the contrasting case described by its steps. steps: Script two transient failures followed by a success under the default retry budget. expected: >- All three attempts are reported in order, with failure classifications and increasing backoff before success. negative: true data_needs: 'Suite-owned fixtures, with a fixed clock where dates or grant windows matter.' bound_to: - TST-PFM-LLM-065 provenance: test source: >- services-platform:modules/llm/tests/unit/runtime/execute-with-llm-retry.spec.ts#reports all three, so the two that were billed are not lost test_sources: - >- services-platform:modules/llm/tests/unit/runtime/execute-with-llm-retry.spec.ts#reports all three, so the two that were billed are not lost evidence: - >- services-platform at ad3f6a6582843008ed0f5a17b131878d15829af1: bound or cited modules/llm/tests/integration/lifecycle/executions.integration.spec.js; modules/llm/tests/integration/lifecycle/run-operation.integration.spec.js; modules/llm/tests/unit/runtime/anthropic-provider.spec.ts; modules/llm/tests/unit/runtime/retry-policy.spec.ts; modules/llm/tests/unit/runtime/run-tool-loop.spec.ts. coverage: rule: 'gap: FND-PFM-LLM-061' transition: 'gap: FND-PFM-LLM-061' validation: covered permission: not-applicable edge: covered failure: covered boundary: covered concurrency: 'gap: FND-PFM-LLM-061' invariant: 'gap: FND-PFM-LLM-061' interaction: covered coverage_notes: transition: >- Existing transition scenarios are retained; the model transition register below records the remaining exhaustive mapping obligation. permission: >- This capability declares no permission key; caller authorization belongs to its consuming operation or a related capability. interaction: >- The related interaction register cites a concrete experiment for every dependency and related capability, including cross-capability workflow seams. invariant: >- The domain-invariant register maps each relevant invariant to a scenario or to the capability coverage finding; an example does not establish every invariant. rule: >- All must-prove rules have scenario references, but the coverage finding retains untested clauses and combined outcomes; scenario presence alone is not exhaustive rule proof. related_interactions: - with: CAP-PFM-LLM-001 scenarios: - ATS-PFM-LLM-001-11 source: 'services-platform:modules/llm/services/catalog/catalog-authority.ts:resolveLlmOperation' - with: CAP-PFM-LLM-005 scenarios: - ATS-PFM-LLM-002-12 source: 'services-platform:modules/llm/catalog/pricing.ts' - with: CAP-PFM-LLM-012 scenarios: - ATS-PFM-LLM-002-13 source: >- services-platform:modules/llm/tests/e2e/llm-adjustments-and-organization.e2e.spec.js#is refused 403 for a customer’s administrator, because the route is system state_models: - STATE-PFM-LLM-001 coverage_obligations: must_prove: - rule: BR-CAP-PFM-LLM-002-01 scenarios: - ATS-PFM-LLM-002-01 - ATS-PFM-LLM-002-12 source: >- services-platform:modules/llm/tests/unit/runtime/run-tool-loop.spec.ts#stops a conversation that will not converge coverage: specified - rule: BR-CAP-PFM-LLM-002-02 scenarios: - ATS-PFM-LLM-002-02 source: >- services-platform:modules/llm/tests/unit/runtime/run-tool-loop.spec.ts#runs no handler for the turn it refused, rather than half of it coverage: specified - rule: BR-CAP-PFM-LLM-002-03 scenarios: - ATS-PFM-LLM-002-03 source: 'services-platform:modules/llm/runtime/run-tool-loop.ts:runToolLoop' coverage: partial gap: FND-PFM-LLM-061 - rule: BR-CAP-PFM-LLM-002-04 scenarios: - ATS-PFM-LLM-002-04 source: 'services-platform:modules/llm/runtime/run-tool-loop.ts' coverage: partial gap: FND-PFM-LLM-061 - rule: BR-CAP-PFM-LLM-002-05 scenarios: - ATS-PFM-LLM-002-05 - ATS-PFM-LLM-002-14 source: 'services-platform:modules/llm/runtime/execute-with-llm-retry.ts' coverage: specified - rule: BR-CAP-PFM-LLM-002-06 scenarios: - ATS-PFM-LLM-002-06 source: >- services-platform:modules/llm/tests/unit/runtime/retry-policy.spec.ts#is not retried, because retrying it reads a purchase problem as load coverage: specified - rule: BR-CAP-PFM-LLM-002-07 scenarios: - ATS-PFM-LLM-002-07 - ATS-PFM-LLM-002-13 source: 'services-platform:modules/llm/services/persistence/invocations.ts:recordInvocationAttempt' coverage: partial gap: FND-PFM-LLM-061 - rule: BR-CAP-PFM-LLM-002-10 scenarios: - ATS-PFM-LLM-002-08 source: 'services-platform:modules/llm/runtime/run-tool-loop.ts' coverage: partial gap: FND-PFM-LLM-061 - rule: BR-CAP-PFM-LLM-002-12 scenarios: - ATS-PFM-LLM-002-09 source: >- services-platform:modules/llm/tests/unit/runtime/anthropic-provider.spec.ts#never sends more than the four Anthropic takes, however many blocks asked to be cached coverage: specified - rule: BR-CAP-PFM-LLM-002-14 scenarios: - ATS-PFM-LLM-002-10 source: 'services-platform:modules/llm/runtime/run-tool-loop.ts:runToolLoop' coverage: partial gap: FND-PFM-LLM-061 - rule: BR-CAP-PFM-LLM-002-15 scenarios: - ATS-PFM-LLM-002-11 source: 'services-platform:modules/llm/runtime/run-tool-loop.ts:runToolLoop' coverage: partial gap: FND-PFM-LLM-061 state_transitions: - model: STATE-PFM-LLM-001 transition: 1 from: estimated to: refused trigger: The access check rejects the AI or operation switch. gap: FND-PFM-LLM-061 provenance: proposed source: 'services-platform:modules/llm/services/execution/run-operation.ts:runOperation' - model: STATE-PFM-LLM-001 transition: 2 from: estimated to: refused trigger: Admission cannot fit the reservation or allocate a running slot. gap: FND-PFM-LLM-061 provenance: proposed source: 'services-platform:modules/llm/services/execution/run-operation.ts:runOperation' - model: STATE-PFM-LLM-001 transition: 3 from: estimated to: reserved trigger: Admission successfully reserves credits and a slot. gap: FND-PFM-LLM-061 provenance: proposed source: 'services-platform:modules/llm/services/persistence/invocations.ts:markInvocationReserved' - model: STATE-PFM-LLM-001 transition: 4 from: reserved to: dispatched trigger: Request capture has returned and its metadata was written. gap: FND-PFM-LLM-061 provenance: proposed source: 'services-platform:modules/llm/services/persistence/invocations.ts:markInvocationDispatched' - model: STATE-PFM-LLM-001 transition: 5 from: dispatched to: completed trigger: The loop returns an outcome other than failed. gap: FND-PFM-LLM-061 provenance: proposed source: 'services-platform:modules/llm/services/persistence/invocations.ts:markInvocationCompleted' - model: STATE-PFM-LLM-001 transition: 6 from: dispatched to: failed trigger: The loop reports failure after provider execution or attempt recording. gap: FND-PFM-LLM-061 provenance: proposed source: 'services-platform:modules/llm/services/persistence/invocations.ts:markInvocationFailed' - model: STATE-PFM-LLM-001 transition: 7 from: completed to: pending trigger: A completed loop result contains no reported usage. gap: FND-PFM-LLM-061 provenance: proposed source: 'services-platform:modules/llm/services/persistence/invocations.ts:markInvocationPending' - model: STATE-PFM-LLM-001 transition: 8 from: completed to: settled trigger: Runtime accounting closes a completed result. gap: FND-PFM-LLM-061 provenance: proposed source: 'services-platform:modules/llm/services/execution/run-operation.ts:settleInvocation' - model: STATE-PFM-LLM-001 transition: 9 from: pending to: settled trigger: Runtime or stale-request recovery closes an unmeasured result. gap: FND-PFM-LLM-061 provenance: proposed source: 'services-platform:modules/llm/services/execution/run-operation.ts:settleInvocation' - model: STATE-PFM-LLM-001 transition: 10 from: failed to: settled trigger: Runtime accounting closes a failed loop result. gap: FND-PFM-LLM-061 provenance: proposed source: 'services-platform:modules/llm/services/execution/run-operation.ts:settleInvocation' - model: STATE-PFM-LLM-001 transition: 11 from: dispatched to: settled trigger: An exception escapes the protected execution block. gap: FND-PFM-LLM-061 provenance: proposed source: 'services-platform:modules/llm/services/execution/run-operation.ts:runOperation' - model: STATE-PFM-LLM-001 transition: 12 from: estimated to: settled trigger: Recovery selects a stale estimated request. gap: FND-PFM-LLM-061 provenance: proposed source: >- services-platform:modules/llm/services/reconciliation/llm-settlement-sweep.ts:sweepUnsettledInvocations - model: STATE-PFM-LLM-001 transition: 13 from: reserved to: settled trigger: Recovery selects a stale reserved request. gap: FND-PFM-LLM-061 provenance: proposed source: >- services-platform:modules/llm/services/reconciliation/llm-settlement-sweep.ts:sweepUnsettledInvocations - model: STATE-PFM-LLM-001 transition: 14 from: dispatched to: settled trigger: Recovery selects a stale dispatched request. gap: FND-PFM-LLM-061 provenance: proposed source: >- services-platform:modules/llm/tests/integration/services/reconciliation/llm-settlement-sweep.integration.spec.js#settles it at its estimate, because its real cost was never learned domain_invariants: - invariant: INV-DOM-PFM-LLM-001-08 scenarios: [] gap: FND-PFM-LLM-061 provenance: proposed source: 'services-platform:modules/llm/db/llm-invocation-attempts.table.ts:MAX_ATTEMPTS_PER_INVOCATION' coverage_limits: finding: FND-PFM-LLM-061 statement: >- Final-answer precedence, exact-credit equality, attempt truncation and slow handler or answer-judge timing still need focused boundary proof together with request-state effects. provenance: proposed source: - >- services-platform:modules/llm/tests/unit/runtime/run-tool-loop.spec.ts#stops a conversation that will not converge - >- services-platform:modules/llm/tests/unit/runtime/run-tool-loop.spec.ts#runs no handler for the turn it refused, rather than half of it - 'services-platform:modules/llm/runtime/run-tool-loop.ts:runToolLoop' - 'services-platform:modules/llm/runtime/run-tool-loop.ts' - 'services-platform:modules/llm/runtime/execute-with-llm-retry.ts' - >- services-platform:modules/llm/tests/unit/runtime/retry-policy.spec.ts#is not retried, because retrying it reads a purchase problem as load - 'services-platform:modules/llm/services/persistence/invocations.ts:recordInvocationAttempt' - 'services-platform:modules/llm/runtime/run-tool-loop.ts' - >- services-platform:modules/llm/tests/unit/runtime/anthropic-provider.spec.ts#never sends more than the four Anthropic takes, however many blocks asked to be cached - 'services-platform:modules/llm/runtime/run-tool-loop.ts:runToolLoop' - 'services-platform:modules/llm/runtime/run-tool-loop.ts:runToolLoop' - 'services-platform:modules/llm/db/llm-invocation-attempts.table.ts:MAX_ATTEMPTS_PER_INVOCATION' invariants_without_scenarios: - INV-DOM-PFM-LLM-001-08 --- # Bound AI conversations and handle recoverable failures: Capability Test Specification ## Coverage | Rule | Scenario | |---|---| | `BR-CAP-PFM-LLM-002-01` | `ATS-PFM-LLM-002-01` | | `BR-CAP-PFM-LLM-002-02` | `ATS-PFM-LLM-002-02` | | `BR-CAP-PFM-LLM-002-03` | `ATS-PFM-LLM-002-03` | | `BR-CAP-PFM-LLM-002-04` | `ATS-PFM-LLM-002-04` | | `BR-CAP-PFM-LLM-002-05` | `ATS-PFM-LLM-002-05` | | `BR-CAP-PFM-LLM-002-06` | `ATS-PFM-LLM-002-06` | | `BR-CAP-PFM-LLM-002-07` | `ATS-PFM-LLM-002-07` | | `BR-CAP-PFM-LLM-002-10` | `ATS-PFM-LLM-002-08` | | `BR-CAP-PFM-LLM-002-12` | `ATS-PFM-LLM-002-09` | | `BR-CAP-PFM-LLM-002-14` | `ATS-PFM-LLM-002-10` | | `BR-CAP-PFM-LLM-002-15` | `ATS-PFM-LLM-002-11` | ## Interactions With Related Capabilities The interaction register links concrete experiments with run ai requests with consistent attribution and accounting; publish ai operation definitions and historical prices; explain an ai refusal or unsuccessful conversation. Each experiment checks the outcome at the shared boundary; an unbound experiment remains proposed. Final-answer precedence, exact-credit equality, attempt truncation and slow handler or answer-judge timing still need focused boundary proof together with request-state effects.