ATS-PFM-LLM-002 Bound AI conversations and handle recoverable failures: Capability Test Specification
Markdown Design / Platform / AI Requests, Usage and the Jazz AI Teammate / Metered AI Requests / Capability Test Specification Coverage Rule gap: FND-PFM-LLM-061 Transition gap: FND-PFM-LLM-061 Validation covered Permission not-applicable Edge covered Failure covered Boundary covered Concurrency gap: FND-PFM-LLM-061 Invariant gap: FND-PFM-LLM-061 Interaction covered Test Scenarios 14 scenarios proving CAP-PFM-LLM-002
01 Turn budget ends an unfinished conversation rule unit negative test
Proves BR-CAP-PFM-LLM-002-01Preconditions A scripted model never offers an acceptable final answer. Steps Run with a small turn allowance. Expected Exactly the allowed requests execute before the loop returns its turn-limit outcome. Data Fixtures representing a scripted model never offers an acceptable final answer. Bound to Source services-platform:modules/llm/tests/unit/runtime/run-tool-loop.spec.ts #stops a conversation that will not converge 02 Oversized tool batches run no handlers edge unit negative test
Proves BR-CAP-PFM-LLM-002-02Preconditions A nonfinal turn asks for more tools than remain. Steps Execute that conversation. Expected The batch ends the loop and none of its handlers runs. Data Fixtures representing a nonfinal turn asks for more tools than remain. Bound to Source services-platform:modules/llm/tests/unit/runtime/run-tool-loop.spec.ts #runs no handler for the turn it refused, rather than half of it 03 Credit guard uses strict excess boundary unit negative test
Proves BR-CAP-PFM-LLM-002-03Preconditions Two nonfinal tool turns together cost more than the hold. Steps Execute both turns and inspect the ending and usage. Expected The loop stops after strict excess, preserving the charge of the crossing turn. Data Fixtures representing two nonfinal tool turns together cost more than the hold. Bound to Source services-platform:modules/llm/tests/unit/runtime/run-tool-loop.spec.ts #stops when the reservation is spent, whether or not the next turn would have converged 05 Retry classification separates recoverable faults failure unit test
Proves BR-CAP-PFM-LLM-002-05Preconditions Provider examples include timeout, rate limit, server errors, malformed and unauthorized requests, connection reset and abort. Steps Classify each example with the retry policy. Expected Transient faults permit retry while permanent request or authority faults do not. Data Fixtures representing provider examples include timeout, rate limit, server errors, malformed and unauthorized requests, connection reset and abort. Bound to Source services-platform:modules/llm/tests/unit/runtime/retry-policy.spec.ts #retries a 529, which is the provider telling us it is overloaded 06 Depleted provider credit does not retry failure unit negative test
Proves BR-CAP-PFM-LLM-002-06Preconditions A provider rate-limit response describes an exhausted balance. Steps Resolve its retry classification. Expected It is credit exhaustion with retry disabled. Data Fixtures representing a provider rate-limit response describes an exhausted balance. Bound to Source services-platform:modules/llm/tests/unit/runtime/retry-policy.spec.ts #is not retried, because retrying it reads a purchase problem as load 07 Retries retain their own usage prices rule api test
Proves BR-CAP-PFM-LLM-002-07Preconditions The first provider attempt fails and its retry succeeds. Steps Execute the request and inspect both attempt records. Expected Both attempts share a turn with distinct retry numbers; missing usage costs zero and reported retry usage receives its own price. Data Fixtures representing the first provider attempt fails and its retry succeeds. Bound to Source services-platform:modules/llm/tests/integration/lifecycle/executions.integration.spec.js #prices the failed attempt, which reported nothing, at zero 08 Rejected answers can be repaired validation unit test
Proves BR-CAP-PFM-LLM-002-10Preconditions The feature rejects one final answer and accepts the next. Steps Run the loop with that answer judge. Expected The model receives the rejection as a tool error, the rejected call uses tool budget and the accepted repair completes. Data Fixtures representing the feature rejects one final answer and accepts the next. Bound to Source services-platform:modules/llm/tests/unit/runtime/run-tool-loop.spec.ts #answers the rejected call with what the consumer gave it, as that call’s own result 09 Cache marking remains within the provider bound boundary unit test
Proves BR-CAP-PFM-LLM-002-12Preconditions Stable content requests more cache markers than the provider accepts. Steps Construct the provider request. Expected At most four markers remain and the marked content text is preserved. Data Fixtures representing stable content requests more cache markers than the provider accepts. Bound to Source services-platform:modules/llm/tests/unit/runtime/anthropic-provider.spec.ts #never sends more than the four Anthropic takes, however many blocks asked to be cached 10 Final acceptance precedes later budget guards edge unit proposed
Proves BR-CAP-PFM-LLM-002-14Preconditions A turn offers an acceptable final answer alongside excess cost and requested tools. Steps Execute the turn and accept its final answer. Expected Completion wins over the later tool and credit checks; ordinary tools do not run and the accepted answer adds no answered-tool count. Data Fixtures representing a turn offers an acceptable final answer alongside excess cost and requested tools. Source services-platform:modules/llm/runtime/run-tool-loop.ts :runToolLoop 11 Missing tool handlers return a repairable error failure unit proposed
Proves BR-CAP-PFM-LLM-002-15Preconditions A model asks for a tool with no handler before offering an answer. Steps Execute with that handler absent. Expected A tool error reaches the model and a later turn can complete within the remaining budgets. Data Fixtures representing a model asks for a tool with no handler before offering an answer. Source services-platform:modules/llm/runtime/run-tool-loop.ts :runToolLoop 12 Historical budgets explain a bounded conversation interaction api proposed
Proves BR-CAP-PFM-LLM-002-01BR-CAP-PFM-LLM-005-03Preconditions A request ends at its published turn limit and the next publication increases that limit. Steps Read the earlier request and run a freshly resolved scripted request. Expected The earlier ending retains its original routing budget; the later request uses the changed budget. Data An isolated workspace with the records, clock instants and caller authority stated in the preconditions. Source services-platform:modules/llm/tests/unit/runtime/run-tool-loop.spec.ts #stops a conversation that will not converge,services-platform:modules/llm/services/catalog/materialize-catalog.ts 13 Attempt history explains retry outcomes interaction api proposed
Proves BR-CAP-PFM-LLM-002-07BR-CAP-PFM-LLM-012-06Preconditions A scripted request first receives overload and then succeeds; another ends at its turn limit. Steps Read retained attempts and explain the limited request. Expected Retry numbering and costs remain distinct from turns, and the explanation names the retained routing limit. Data An isolated workspace with the records, clock instants and caller authority stated in the preconditions. Source services-platform:modules/llm/services/persistence/invocations.ts:recordInvocationAttempt,services-platform:modules/llm/services/explain/explain-invocation.ts 14 Retry exhaustion reports every attempt failure unit negative test
Proves BR-CAP-PFM-LLM-002-05Preconditions The suite owns isolated fixtures representing the valid control and the contrasting case described by its steps. Steps Script two transient failures followed by a success under the default retry budget. Expected All three attempts are reported in order, with failure classifications and increasing backoff before success. Data Suite-owned fixtures, with a fixed clock where dates or grant windows matter. Bound to Source services-platform:modules/llm/tests/unit/runtime/execute-with-llm-retry.spec.ts #reports all three, so the two that were billed are not lost ← Previous Capability Test Specification ATS-PFM-LLM-001 · undefined Next Capability Test Specification → ATS-PFM-LLM-003 · undefined