Examination Findings and Product Questions
Findings
| id | severity | statement | affects | evidence | kind |
|---|---|---|---|---|---|
FND-PFM-LLM-001 | high | The console calls a model directly for general Jazz AI replies. Those replies bypass Platform admission, metering, evidence capture and feature switches. They send conversation text and the configured workspace name through a provider path not covered by the Platform boundary or recorded provider assessment. | CAP-PFM-LLM-019, BR-CAP-PFM-LLM-019-02, BR-CAP-PFM-LLM-001-09, OQ-PFM-LLM-003 | react-console:app/api/agent/conversations/[id]/messages/route.ts:MODEL; services-platform:tests/unit/llm-provider-boundary.spec.ts#refuses an import of the openai sdk even in the adapter; services-platform:docs/compliance/exceptions.md#EX-AI-003 | inconsistency |
FND-PFM-LLM-002 | high | The prototype stores conversations and projects in process memory with sample data and a fixed owner. It has no person or workspace partition, shares records within a server instance and loses them on restart. | CAP-PFM-LLM-019, CAP-PFM-LLM-020, BR-CAP-PFM-LLM-019-07, BR-CAP-PFM-LLM-020-01, JNY-PFM-LLM-005 | react-console:lib/agent/store.ts (seed, db, owned); lib/queries/store.ts:CURRENT_USER; lib/queries/registry.ts:MOCK_CURRENT_USER | defect |
FND-PFM-LLM-003 | medium | Admission requires room for the request’s maximum reservation, while summaries and notices use different charged or committed figures. A positive remaining balance can therefore coexist with a refusal and no pause notice. The exhaustion decision remains open. | BR-CAP-PFM-LLM-007-02, BR-CAP-PFM-LLM-008-02, BR-CAP-PFM-LLM-010-03, BR-CAP-PFM-LLM-017-02, OQ-PFM-LLM-001 | services-platform:docs/milestones/plans/PLAT-M1-plan.md#D1; docs/milestones/PLAT-M3-ai-usage-organization-page.md#State found on 2026-10-19 (re-inspected); modules/llm/services/usage-reports/usage-status.ts:deriveUsageStatus | inconsistency |
FND-PFM-LLM-004 | medium | Feature availability does not account for active operation exemptions. A feature can appear disabled while exempted requests pass admission. | BR-CAP-PFM-LLM-017-05, CAP-PFM-LLM-017 | services-platform:modules/llm/services/usage/organization-usage.ts:restrictionOf; docs/milestones/PLAT-M3-ai-usage-organization-page.md | defect |
FND-PFM-LLM-005 | medium | Repeating a grant or correction creates another independently audited record. The reported proposal calls for idempotent HQ writes, and correction idempotency remains recorded debt. Limit updates use version checks rather than creating duplicate grants. | BR-CAP-PFM-LLM-014-10, BR-CAP-PFM-LLM-015-07, JNY-PFM-LLM-003 | services-platform:docs/debt/llm-adjustment-idempotency.md; docs/llm.md#Overrides; docs/proposals/llm-observability/11-decisions.md#11.1 (Idempotency of HQ writes) | debt |
FND-PFM-LLM-006 | medium | The service implements limit changes, grants, revocation, corrections and evidence reads. Their console controls remain disabled or absent, and the control milestone remains proposed. | CAP-PFM-LLM-013, CAP-PFM-LLM-014, CAP-PFM-LLM-015, CAP-PFM-LLM-016, BR-CAP-PFM-LLM-013-11, BR-CAP-PFM-LLM-014-12, BR-CAP-PFM-LLM-015-09, BR-CAP-PFM-LLM-016-07 | react-console:components/console/hq/tenant-ai-usage-card.tsx; components/console/hq/ai-usage/request/evidence-card.tsx; services-platform:docs/milestones/PLAT-M2-ai-usage-hq-controls.md | debt |
FND-PFM-LLM-007 | medium | The customer AI Usage dashboard and its navigation entry are absent. The recorded placement decision exists, while the summary, availability and reporting services already supply its underlying data. | BR-CAP-PFM-LLM-017-09, CAP-PFM-LLM-017, JNY-PFM-LLM-004 | services-platform:docs/milestones/PLAT-M3-ai-usage-organization-page.md; react-console:components/console/nav-config.ts (Dashboards planned) | debt |
FND-PFM-LLM-008 | low | Corrections and allowance reductions can cross usage thresholds without creating notices. The plan treats this behavior as an unapproved assumption. | BR-CAP-PFM-LLM-008-10, CAP-PFM-LLM-015 | services-platform:docs/milestones/plans/PLAT-M1-plan.md#4. Assumptions (A10); docs/milestones/PLAT-M2-ai-usage-hq-controls.md | undocumented |
FND-PFM-LLM-009 | low | The early-warning milestone is incomplete. Usage signals remain unavailable, the outage refusal is unused, and declared spike and raw-retention settings have no implementation. Request history has no implemented monthly partition or retention cleanup. | CAP-PFM-LLM-010, CAP-PFM-LLM-011, CON-PFM-LLM-004 | services-platform:docs/milestones/PLAT-M4-ai-usage-early-warning.md; modules/llm/llm.settings.ts (LLM_RAW_RETENTION_MONTHS, spike_factor); services/usage-reports/tenant-usage.ts (signals) | debt |
FND-PFM-LLM-010 | low | Source guides and comments lag implementation. They describe implemented routes as deferred, understate permission and concurrency enforcement, and give query budgets and grant ceilings that differ from current declarations. | MOD-PFM-LLM, BR-CAP-PFM-LLM-017-01, BR-CAP-PFM-LLM-007-04, BR-CAP-PFM-LLM-014-05 | services-platform:modules/llm/README.md#What is built, what is deferred; docs/llm.md#The HQ surface, #The closed set grows, #What is declared, and what is published, #Overrides; aws/platform.service.ts (LLM_USAGE_PERMISSIONS comment); modules/llm/catalog/llm-operations.ts; llm.settings.ts | inconsistency |
FND-PFM-LLM-011 | low | The runtime distinguishes provider account exhaustion, but the shared server-error response omits that classification. The recorded debt prevents clients from reliably using the code. | BR-CAP-PFM-LLM-001-12 | services-platform:docs/debt/llm-server-error-code-not-in-body.md | debt |
FND-PFM-LLM-012 | medium | AI notices currently target the built-in Administrator role. The later direction for responsibility-based operational notifications has not been implemented here; the interim recipient decision remains pending. | BR-CAP-PFM-LLM-008-03, BR-CAP-PFM-LLM-014-08, OQ-PFM-LLM-002 | services-platform:docs/proposals/llm-observability/11-decisions.md#D-4; docs/milestones/plans/PLAT-M1-plan.md#D2 | debt |
FND-PFM-LLM-013 | medium | The records do not establish complete customer acceptance. The plan awaits approval, HQ browser tests use mocks or rendering checks, real-inbox delivery is not demonstrated, and evidence access lacks an HTTP end-to-end test. | JNY-PFM-LLM-001, JNY-PFM-LLM-002, JNY-PFM-LLM-003, CAP-PFM-LLM-016 | services-platform:docs/milestones/PLAT-M1-ai-usage-metered-and-visible.md#Acceptance scenarios; docs/milestones/PLAT-M2-ai-usage-hq-controls.md; react-console:e2e/mock/hq-ai-usage.mock.spec.ts; e2e/dev/hq-ai-usage.dev.spec.ts | debt |
FND-PFM-LLM-014 | medium | The outbound-model compliance exception remains open. The records do not establish provider subprocessor status, retention or training treatment, despite the separate decision governing Rhythm’s captured evidence. | CAP-PFM-LLM-004, BR-CAP-PFM-LLM-004-01 | services-platform:docs/compliance/exceptions.md#EX-AI-003 | debt |
FND-PFM-LLM-015 | low | The directory assistant uses fixed client-side interpretation rather than a model. Its replies identify the behavior as a prototype. | BR-CAP-PFM-LLM-020-05 | react-console:components/record-explorer/use-explorer-assistant.ts | undocumented |
FND-PFM-LLM-016 | low | The original module registry omitted specific records and several console source surfaces. The examined registry now names the sources needed for this module’s documented scope. | MOD-PFM-LLM | specs/PRODUCT-AREAS.md (MOD-PFM-LLM row); react-console tree at baf6dae3 | inconsistency |
FND-PFM-LLM-017 | low | Usage reporting groups UTC dates while allowance periods follow the organization’s calendar. Month-boundary differences remain an unapproved reporting assumption. | BR-CAP-PFM-LLM-010-02, BR-CAP-PFM-LLM-017-08 | services-platform:docs/milestones/plans/PLAT-M1-plan.md#4. Assumptions (A13); docs/llm.md#Reporting | undocumented |
FND-PFM-LLM-018 | low | A failed restoration notice is logged and counted but is not retried or retained as delivery history available to HQ. Threshold notices have a different recovery path. | BR-CAP-PFM-LLM-014-11 | services-platform:modules/llm/services/notices/override-notice.ts | defect |
FND-PFM-LLM-019 | low | The console exposes Jazz AI entry points without an AI-specific permission check or organization AI-switch evaluation. Product access policy remains unresolved. | BR-CAP-PFM-LLM-019-01, OQ-PFM-LLM-003 | react-console:components/console/nav-config.ts:unlistedNav; components/console/site-header.tsx | undocumented |
FND-PFM-LLM-020 | low | The request boundary stores caller-supplied person attribution without checking that the caller represents that person. | BR-CAP-PFM-LLM-001-05 | services-platform:docs/llm.md#Attribution | undocumented |
FND-PFM-LLM-021 | low | The implementation contains unapproved plan assumptions: the default allowance and local month, maximum reservations, admitted-request overshoot, customer wording and UTC reporting days. Observed behavior does not approve these choices. | BR-CAP-PFM-LLM-007-01, BR-CAP-PFM-LLM-007-07, BR-CAP-PFM-LLM-018-01, BR-CAP-PFM-LLM-018-02, BR-CAP-PFM-LLM-018-03, BR-CAP-PFM-LLM-018-04 | services-platform:docs/milestones/plans/PLAT-M1-plan.md#4. Assumptions | undocumented |
FND-PFM-LLM-022 | low | The reconciliation guide describes charging reported attempt usage, but the recovery implementation and tests use the operation estimate with unknown money cost. | BR-CAP-PFM-LLM-009-01 | services-platform:docs/llm.md#Reconciliation; modules/llm/tests/integration/services/reconciliation/llm-settlement-sweep.integration.spec.js#settles it at its estimate, because its real cost was never learned | inconsistency |
FND-PFM-LLM-023 | low | The foundation record approves a work plan while identifying configuration choices as reversible author decisions. Later planning still lists those behaviors as unapproved assumptions. Earlier budgets also differ from the current catalog, so work-plan approval does not confirm current settings. | BR-CAP-PFM-LLM-007-01, BR-CAP-PFM-LLM-007-02, BR-CAP-PFM-LLM-005-01, CAP-PFM-LLM-001 | services-platform:docs/llm/llm-foundation-and-generate-query.md#4.5 Model and Configuration; #9. Work Plan (approved by Andrew 2026-10-10, "go"); docs/milestones/plans/PLAT-M1-plan.md#4. Assumptions | inconsistency |
FND-PFM-LLM-024 | high | The runtime takes allowance credits and writes request-capture metadata before entering its cleanup block. A failed reservation-status write, capture-metadata write or dispatch-status write can leave credits held until recovery, and a failure before the request records its hold leaves recovery without that held amount. | CAP-PFM-LLM-001 | services-platform:modules/llm/services/execution/run-operation.ts:runOperation; services-platform:modules/llm/services/reconciliation/llm-settlement-sweep.ts:sweepUnsettledInvocations | defect |
FND-PFM-LLM-025 | high | The runtime and recovery sweep mark a request settled before updating its allowance period in a separate transaction. Outside a caller-supplied encompassing transaction, an interruption between these writes leaves a terminal request with a held or uncharged allowance balance; the sweep excludes terminal requests. Fault-injection proof remains needed. | CAP-PFM-LLM-001, CAP-PFM-LLM-009 | services-platform:modules/llm/services/execution/run-operation.ts:settleInvocation; services-platform:modules/llm/services/persistence/invocations.ts:markInvocationSettled; services-platform:modules/llm/services/reconciliation/llm-settlement-sweep.ts:sweepUnsettledInvocations | defect |
FND-PFM-LLM-026 | medium | An accepted final answer returns before the tool and credit guards. The credit guard uses greater-than and rounded turn costs, not greater-than-or-equal or the eventual once-rounded charge; tool handlers and answer validation have no enforced wall-clock timeout. These bounds need an explicit product interpretation and focused acceptance proof. | CAP-PFM-LLM-002 | services-platform:modules/llm/runtime/run-tool-loop.ts:runToolLoop, decideFinal, answerToolCall; services-platform:modules/llm/catalog/pricing.ts:convertUsageToCredits | undocumented |
FND-PFM-LLM-027 | medium | Some draft acceptance bindings prove individual backend results or mock rendering rather than the complete journey or broader expected result. Complete-journey bindings are removed; the remaining focused tests are useful evidence but do not establish customer acceptance. | JNY-PFM-LLM-001, JNY-PFM-LLM-002, JNY-PFM-LLM-003, JNY-PFM-LLM-004 | react-console:e2e/mock/hq-ai-usage.mock.spec.ts; services-platform:modules/llm/tests/e2e/allowance-warning.e2e.spec.js; services-platform:modules/llm/tests/e2e/llm-adjustments-and-organization.e2e.spec.js | debt |
FND-PFM-LLM-028 | medium | Content reads audit missing-grant refusals and successful responses, but permission and input rejections, store errors and malformed stored content can fail without a content-read audit entry. Missing-request reads with a valid grant also intentionally leave no entry. The product should define the required audit boundary before content-reading controls ship. | CAP-PFM-LLM-016 | services-platform:modules/llm/services/evidence/evidence-access.ts:readInvocationEvidence, answerFor; services-platform:modules/llm/aws/api/llm-usage.routes.ts:getHqTenantLlmUsageInvocationEvidence | undocumented |
FND-PFM-LLM-029 | low | Allowance refunds and charge additions are limited to ten million credits in either direction. A negative correction to a current month with no period bypasses the service refund check; persistence creates the period and clamps its balance to zero while retaining the negative correction. The product should require a matching charged balance for this path. | CAP-PFM-LLM-015 | services-platform:modules/llm/services/adjustments/adjust-allowance.ts:requireAdjustable; services-platform:modules/llm/services/persistence/allowance-periods.ts:applyAllowanceAdjustment | defect |
FND-PFM-LLM-030 | low | The draft operations denominator omits ten console conversation API operations, and its integration test total exceeds the forty-two specification files actually present. The registry also omits several cited console helpers and source records. The examination expands the declared read scope and distinguishes code inventory coverage from tested behavior. | MOD-PFM-LLM | react-console:app/api/agent; specs/platform/llm/MODULE.md:coverage_catalog; specs/PRODUCT-AREAS.md:MOD-PFM-LLM | inconsistency |
FND-PFM-LLM-031 | medium | Threshold email delivery precedes the completion stamp. An interruption after send can leave the claim retryable and send the same notice again after the lease expires. The product should choose and verify the delivery guarantee rather than infer exactly-once delivery from the racing-sender test. | CAP-PFM-LLM-008, ATS-PFM-LLM-008-09 | services-platform:modules/llm/services/notices/threshold-notice.ts:sendThresholdNotice; services-platform:modules/llm/services/persistence/allowance-periods.ts:stampThresholdNotified | defect |
FND-PFM-LLM-032 | high | Hourly recovery settles every request left unsettled for a day at its full estimate, whatever state it reached. A request whose process died before admission completed (still estimated, nothing held, nothing sent) and one whose provider never answered are both charged the estimate; the period's running count is decremented for a request that never took a slot, and a hold the request never recorded is not released. Design finding; the charging policy is an open product question. | CAP-PFM-LLM-009, CAP-PFM-LLM-001, WF-PFM-LLM-004, STATE-PFM-LLM-001, ADR-PFM-LLM-004, OQ-PFM-LLM-004 | services-platform:modules/llm/services/reconciliation/llm-settlement-sweep.ts:sweepUnsettledInvocations; services-platform:modules/llm/services/persistence/invocations.ts:listUnsettledInvocations; services-platform:modules/llm/services/persistence/allowance-periods.ts:settleCredits | defect |
FND-PFM-LLM-033 | low | Planning rule BR-CAP-PFM-LLM-015-09 says the console's charge-correction menu item is disabled and visible to usage managers, but the organization card's action menu has only Edit Limits and Grant Override; no charge-correction or grant-revocation control exists, and the card lists neither corrections nor threshold crossings although the service offers both. A planning correction for Product to make; the design describes the screen as built. | BR-CAP-PFM-LLM-015-09, CAP-PFM-LLM-014, CAP-PFM-LLM-015, SCR-PFM-LLM-007 | react-console:components/console/hq/tenant-ai-usage-card.tsx:AiUsageMenu; react-console:lib/hq/use-hq-ai-usage.ts | inconsistency |
FND-PFM-LLM-034 | medium | Threshold and capacity-restored notices are sent directly through the shared e-mail helper rather than the Platform notification pipeline (registry, queue and delivery records) that the notifications standard now defines, so they leave no notification record, follow no personal preference and are not listed as a producer. | CAP-PFM-LLM-008, CAP-PFM-LLM-014, WF-PFM-LLM-005, ADR-PFM-LLM-011, OQ-PFM-LLM-005 | services-platform:modules/llm/services/notices/threshold-notice.ts:sendThresholdNotice; services-platform:lib/notices/send-notice.ts:sendNoticeToRecipients; services-platform:docs/notifications.md#9. Producers | inconsistency |
FND-PFM-LLM-035 | low | A threshold crossing is recorded in its own transaction after the settlement that caused it has committed. An interruption between them leaves no crossing, and so no notice, until a later settlement in the same month evaluates thresholds again; if none follows, the month never records it. | CAP-PFM-LLM-008, WF-PFM-LLM-005, STATE-PFM-LLM-006 | services-platform:modules/llm/services/metering/llm-credit-ledger.ts:evaluateCrossings; services-platform:modules/llm/services/persistence/allowance-periods.ts:recordThresholdCrossing | defect |
FND-PFM-LLM-036 | low | A scoped Jazz AI refinement applies the change on screen before the exchange is appended, and Undo reverts before appending its note. When the append fails the person sees an error with the description restored while the change stays applied, and sending again applies it twice; the conversation and the screen then disagree. | CAP-PFM-LLM-020, WF-PFM-LLM-014, SCR-PFM-LLM-002 | react-console:components/console/agent-chat.tsx:AgentChat; react-console:components/queries/query-workspace.tsx:refine | defect |
FND-PFM-LLM-037 | low | The daily rollup does not advance its progress mark at all while any operational organization's changed days cannot be read. One persistently failing organization makes every run recompute the same days for everyone and lets lag grow until the alarm, although other organizations' days are still rebuilt. | CAP-PFM-LLM-010, WF-PFM-LLM-006 | services-platform:modules/llm/services/reconciliation/rollup-settled-invocations.ts:rollUpSettledInvocations | debt |
FND-PFM-LLM-038 | high | The request runtime's cleanup block releases the reservation whenever it did not mark the request settled, without checking whether its own settlement applied. If the period was already charged and a later step throws (for example recording a threshold crossing), the hold is released a second time; because releases are floored rather than checked, this takes held credits and a running slot from other running requests of the organization. A failed write of the response capture status inside the block also settles the request as released, so a conversation that spent real usage is charged nothing. | CAP-PFM-LLM-001, CAP-PFM-LLM-007, WF-PFM-LLM-001, STATE-PFM-LLM-001, ADR-PFM-LLM-003 | services-platform:modules/llm/services/execution/run-operation.ts:runOperation; services-platform:modules/llm/services/persistence/allowance-periods.ts:releaseCredits; services-platform:modules/llm/services/metering/llm-credit-ledger.ts:settle | defect |
FND-PFM-LLM-039 | medium | Catalogue validation runs before writes, but publication commits each append separately. Warm caches do not refresh when a cached version is deleted; the resolver retries only an absent cache result. The design now states both limits. | DOM-PFM-LLM-002, WF-PFM-LLM-003, SVC-PFM-LLM-llm | services-platform:modules/llm/services/catalog/materialize-catalog.ts:materializeLlmCatalog; modules/llm/services/catalog/catalog-authority.ts:resolveLlmOperation | inconsistency |
FND-PFM-LLM-040 | medium | The tool loop catches provider-execution and attempt-recording errors and returns a failed result with in-memory attempts. Dispatch precedes provider construction, and settled status does not prove that the period movement succeeded. Corrected design guarantees require fault-injection proof. | WF-PFM-LLM-001, STATE-PFM-LLM-001, DOM-PFM-LLM-001 | services-platform:modules/llm/runtime/run-tool-loop.ts:runToolLoop; modules/llm/services/execution/run-operation.ts:runOperation | inconsistency |
FND-PFM-LLM-041 | medium | Threshold notices can exhaust their attempts without delivery. A send that outlasts its ten-minute lease can overlap a later claimant, and completion stamping does not verify lease ownership. The design no longer claims guaranteed delivery or exclusion of every overlapping send. | STATE-PFM-LLM-006, WF-PFM-LLM-005, ADR-PFM-LLM-011 | services-platform:modules/llm/services/persistence/allowance-periods.ts:claimThresholdNotice; modules/llm/services/persistence/allowance-periods.ts:stampThresholdNotified | defect |
FND-PFM-LLM-042 | medium | Query generation has separate inference artifacts for consuming surfaces in this module, Metadata and Queries. They describe the same task rather than separate model questions. Caller links for the foundation and request workflow are corrected here; canonical ownership across modules remains unresolved within this examination scope. | IOP-PFM-LLM-002, SVC-PFM-LLM-llm, WF-PFM-LLM-001 | specs/platform/llm/design/inference/IOP-PFM-LLM-002-jazz-ai-query-refinement.md; specs/platform/queries/design/inference; specs/platform/metadata/design/inference | inconsistency |
FND-PFM-LLM-043 | medium | Approval of the foundation work plan does not confirm the current maximum-reservation policy or catalogue settings. The admission decision record is downgraded to proposed while retaining its original decision sources. | ADR-PFM-LLM-002 | services-platform:docs/llm/llm-foundation-and-generate-query.md#4.5 Model and Configuration; docs/milestones/plans/PLAT-M1-plan.md#4. Assumptions; FND-PFM-LLM-023 | inconsistency |
FND-PFM-LLM-044 | medium | Planning describes a disabled charge-correction menu item, but the pinned organization card exposes only disabled limits and grant items. The screen contract follows the implementation; the planning rule needs a later product-reviewed correction. | SCR-PFM-LLM-007, BR-CAP-PFM-LLM-015-09 | react-console:components/console/hq/tenant-ai-usage-card.tsx:AiUsageMenu | inconsistency |
FND-PFM-LLM-045 | medium | The named design brief is missing from this checkout, and the planning artifacts remain draft with proposed intent and a registry entry awaiting approval. Examination uses those records as the documented baseline and does not establish approval. | MOD-PFM-LLM | rhythm-v2-docs:briefs/design/platform-llm.brief.md (absent at 68e00372); specs/platform/llm/planning; specs/PRODUCT-AREAS.md | undocumented |
FND-PFM-LLM-046 | low | The conversation lifecycle omitted supported actions on empty conversations and implied state restrictions that the prototype does not enforce. Empty conversations can be renamed, archived or deleted; requests can append messages while archived; restart loss affects every state. The rewritten model records these limits. | STATE-PFM-LLM-008 | react-console:lib/agent/store.ts:updateConversation; lib/agent/store.ts:appendMessages; lib/agent/store.ts:db | inconsistency |
FND-PFM-LLM-047 | medium | Refund eligibility reads the period without a row lock before applying a floored update. Concurrent refunds can pass against the same earlier balance, preserving signed correction records whose sum exceeds the actual returned capacity. A concurrency acceptance case is needed. | WF-PFM-LLM-009, DOM-PFM-LLM-003 | services-platform:modules/llm/services/adjustments/adjust-allowance.ts:requireAdjustable; modules/llm/services/persistence/allowance-periods.ts:applyAllowanceAdjustment | defect |
FND-PFM-LLM-048 | low | Task detail shows start, end, person, request count and spend but does not render duration. The dashboard does not render a costliest-task card. Corrected screen contracts retain only implemented displays and navigation; task detail includes associated requests regardless of settlement. | SCR-PFM-LLM-003, SCR-PFM-LLM-006 | react-console:components/console/hq/ai-usage/ai-usage-dashboard.tsx:AiUsageDashboard; components/console/hq/ai-usage/ai-usage-execution-detail.tsx:ExecutionBody | inconsistency |
FND-PFM-LLM-051 | high | General Jazz AI replies use a direct provider path outside the shared admission, metering, evidence and feature controls. | TC-AI-006, TC-AI-007, TC-VEN-001, TC-AZ-002, CAP-PFM-LLM-019, BR-CAP-PFM-LLM-019-02, BR-CAP-PFM-LLM-001-09 | Structured control evidence | control-gap |
FND-PFM-LLM-052 | high | Prototype conversation storage does not establish isolation by the requesting user and workspace. | TC-TEN-001, TC-TEN-002, TC-TEN-003, CAP-PFM-LLM-019, CAP-PFM-LLM-020, BR-CAP-PFM-LLM-019-07, BR-CAP-PFM-LLM-020-01 | Structured control evidence | control-gap |
FND-PFM-LLM-053 | medium | Repeated AI capacity grants or charge corrections create separate records rather than reuse the original decision. | TC-PI-002, BR-CAP-PFM-LLM-014-10, BR-CAP-PFM-LLM-015-07 | Structured control evidence | control-gap |
FND-PFM-LLM-054 | low | Raw request-history retention and the declared early-warning signals remain unimplemented. | TC-DAT-003, TC-MON-002, CAP-PFM-LLM-010, CAP-PFM-LLM-011 | Structured control evidence | control-gap |
FND-PFM-LLM-055 | medium | The open model-provider exception leaves subprocessor, training and provider-retention treatment unestablished. | TC-AI-007, TC-VEN-001, TC-VEN-003, CAP-PFM-LLM-004, BR-CAP-PFM-LLM-004-01 | Structured control evidence | control-gap |
FND-PFM-LLM-056 | high | AI request settlement can commit before allowance movement, leaving a terminal request outside the recovery sweep while its balance remains wrong. | TC-AVL-004, TC-PI-004, CAP-PFM-LLM-001, CAP-PFM-LLM-009 | Structured control evidence | control-gap |
FND-PFM-LLM-057 | medium | Captured-content read failures do not all produce the sensitive-read audit required by the evidence boundary. | TC-LOG-001, CAP-PFM-LLM-016 | Structured control evidence | control-gap |
FND-PFM-LLM-058 | high | AI cleanup can release a hold twice after partial settlement or record no charge after failed capture persistence, disturbing another running request’s account. | TC-PI-002, TC-PI-003, TC-PI-004, CAP-PFM-LLM-001, CAP-PFM-LLM-007, WF-PFM-LLM-001 | Structured control evidence | control-gap |
FND-PFM-LLM-059 | medium | Concurrent refunds can both pass an unlocked balance check and retain corrections exceeding the capacity actually returned. | TC-PI-003, WF-PFM-LLM-009 | Structured control evidence | control-gap |
| FND-PFM-LLM-049 | low | The general-reply call does not set a retry count or timeout. Its effective SDK defaults were not established by this examination, so the inference contract makes no guarantee about either behavior. Establish these defaults and prove the failure boundary before customer preparation. | IOP-PFM-LLM-001, WF-PFM-LLM-013 | react-console:app/api/agent/conversations/[id]/messages/route.ts:generateText | undocumented | | FND-PFM-LLM-050 | low | The ten console route entries have local contract labels containing a method and route, but their handlers declare no operation identifiers. Existing labels remain permanent references; they must not be interpreted as native route operation identifiers. | SVC-PFM-LLM-console-agent | react-console:app/api/agent/projects/route.ts; app/api/agent/projects/[id]/route.ts; app/api/agent/conversations/route.ts; app/api/agent/conversations/[id]/route.ts; app/api/agent/conversations/[id]/messages/route.ts | undocumented |
| FND-PFM-LLM-060 | high | Interruption proof is missing between request creation, reservation metadata, capture metadata, settlement and the separate period movement. Recovery from every early state and a losing cleanup writer need explicit accounting checks. | ATS-PFM-LLM-001, CAP-PFM-LLM-001, STATE-PFM-LLM-001 | services-platform:modules/llm/services/catalog/catalog-authority.ts:resolveLlmOperation; services-platform:docs/llm.md#The order of the steps, which is the design; services-platform:modules/llm/services/execution/run-operation.ts:settleInvocation | coverage-gap | | FND-PFM-LLM-061 | medium | Final-answer precedence, exact-credit equality, attempt truncation and slow handler or answer-judge timing still need focused boundary proof together with request-state effects. | ATS-PFM-LLM-002, CAP-PFM-LLM-002, STATE-PFM-LLM-001 | services-platform:modules/llm/tests/unit/runtime/run-tool-loop.spec.ts#stops a conversation that will not converge; services-platform:modules/llm/tests/unit/runtime/run-tool-loop.spec.ts#runs no handler for the turn it refused, rather than half of it; services-platform:modules/llm/runtime/run-tool-loop.ts:runToolLoop | coverage-gap | | FND-PFM-LLM-062 | medium | Running and closed replay races, late settlements and simultaneous close versus abandonment need complete task-state coverage. Failed close persistence needs a caller-error preservation case. | ATS-PFM-LLM-003, CAP-PFM-LLM-003, STATE-PFM-LLM-002 | services-platform:modules/llm/services/execution/with-llm-execution.ts; services-platform:modules/llm/services/execution/execution-scope.ts; services-platform:modules/llm/services/execution/with-llm-execution.ts | coverage-gap | | FND-PFM-LLM-063 | medium | Capture skip and serialization failures, metadata-write interruption, retention expiry and purge while traffic continues still need explicit outcomes. Purge quiescence is an external precondition. | ATS-PFM-LLM-004, CAP-PFM-LLM-004, STATE-PFM-LLM-003 | services-platform:modules/llm/services/execution/run-operation.ts:recordCapture; services-platform:modules/llm/services/evidence/evidence-redaction.ts; services-platform:modules/llm/services/persistence/invocations.ts:recordInvocationAttempt | coverage-gap | | FND-PFM-LLM-064 | medium | Changed and reverted publication needs complete lifecycle coverage; different-content racing publication, partial persistence failure and warm-cache convergence remain unproved. | ATS-PFM-LLM-005, CAP-PFM-LLM-005, STATE-PFM-LLM-004 | services-platform:modules/llm/catalog/pricing.ts; services-platform:modules/llm/services/catalog/materialize-catalog.ts; services-platform:modules/llm/services/catalog/materialize-catalog.ts | coverage-gap | | FND-PFM-LLM-065 | medium | Deliberate smoke and evaluation enablement, absent prerequisites, output safety and live-run boundaries need executable proof beyond adapter construction and script configuration. | ATS-PFM-LLM-006, CAP-PFM-LLM-006 | services-platform:tests/unit/llm-stub-mode-config.spec.ts#forces stub mode in the integration tier; services-platform:modules/llm/tests/unit/runtime/anthropic-provider.spec.ts#refuses, rather than reaching a provider a test never meant to call; services-platform:docs/llm.md#The live runs | coverage-gap | | FND-PFM-LLM-066 | medium | Complete displayed-position transitions, month-boundary races, changes during admission and interrupted period movement need additional proof beyond the atomic reservation race. | ATS-PFM-LLM-007, CAP-PFM-LLM-007, STATE-PFM-LLM-005, STATE-PFM-LLM-007 | services-platform:modules/llm/services/persistence/allowance-periods.ts; services-platform:modules/llm/services/persistence/allowance-periods.ts; services-platform:modules/llm/services/execution/run-operation.ts | coverage-gap | | FND-PFM-LLM-067 | medium | Partial delivery, lease expiry during a slow sender, post-send stamp interruption and successful delivery suppression need individual cases. The current racing-sender test covers only a live lease. | ATS-PFM-LLM-008, CAP-PFM-LLM-008, STATE-PFM-LLM-006 | services-platform:modules/llm/services/persistence/allowance-periods.ts; services-platform:modules/llm/services/metering/llm-credit-ledger.ts:evaluateCrossings; services-platform:modules/llm/tests/integration/services/notices/threshold-notice.integration.spec.js#sends one message to each administrator, and to neither the suspended one nor the administrator of another organization | coverage-gap | | FND-PFM-LLM-068 | high | Every early request state, terminal-request accounting drift and failure between recovery settlement and period movement need fault injection. Sweeping and rollup follow-up failures need combined verification. | ATS-PFM-LLM-009, CAP-PFM-LLM-009, STATE-PFM-LLM-001, STATE-PFM-LLM-002, STATE-PFM-LLM-006 | services-platform:modules/llm/aws/lambda/RECONCILIATION.handler.ts; services-platform:modules/llm/tests/integration/services/reconciliation/llm-settlement-sweep.integration.spec.js#leaves the hold in place, so a live call keeps the credits it reserved; services-platform:modules/llm/services/reconciliation/llm-settlement-sweep.ts | coverage-gap | | FND-PFM-LLM-069 | medium | Unavailable-section rendering, independent live position reads during changes and unreadable-workspace reporting progress need feature-level proof. The browser probe verifies loading rather than accounting agreement. | ATS-PFM-LLM-010, CAP-PFM-LLM-010, STATE-PFM-LLM-005 | services-platform:modules/llm/tests/e2e/llm-usage.e2e.spec.js#refuses a customer’s administrator, whatever they hold at their customer’s scope; services-platform:modules/llm/services/usage-reports/platform-usage.ts; services-platform:modules/llm/services/usage-reports/usage-status.ts | coverage-gap | | FND-PFM-LLM-070 | medium | All independent section failures, complete task-to-request navigation and concurrent account changes need additional cases. Mocked browser data does not establish agreement with real ledger reads. | ATS-PFM-LLM-011, CAP-PFM-LLM-011, STATE-PFM-LLM-005, STATE-PFM-LLM-007 | services-platform:modules/llm/tests/e2e/llm-usage.e2e.spec.js#refuses a customer’s administrator a tenant’s usage even in their own tenant; services-platform:modules/llm/tests/e2e/llm-usage.e2e.spec.js#answers 404 for a request of another tenant, the tenant policy binding the read; services-platform:modules/llm/services/usage-reports/tenant-usage.ts | coverage-gap | | FND-PFM-LLM-071 | medium | Concurrent changes while the present-position blocks are assembled and snapshot-free or truncated explanations need combined boundary proof. A complete support journey remains unexecuted. | ATS-PFM-LLM-012, CAP-PFM-LLM-012 | services-platform:modules/llm/tests/e2e/llm-adjustments-and-organization.e2e.spec.js#is refused 403 for a customer’s administrator, because the route is system; services-platform:modules/llm/services/explain/explain-invocation.ts; services-platform:modules/llm/tests/integration/services/explain/explain-invocation.integration.spec.js#says so when an override the snapshot named cannot be read back, rather than dropping it | coverage-gap | | FND-PFM-LLM-072 | medium | Audit rollback, identical first-write races and full live-position transitions after configuration replacement need explicit proof. The console save remains disabled. | ATS-PFM-LLM-013, CAP-PFM-LLM-013, STATE-PFM-LLM-005 | services-platform:modules/llm/tests/e2e/llm-limits.e2e.spec.js#is refused the write before a handler runs, whatever the customer’s scope grants; services-platform:modules/llm/services/limits/tenant-limits.ts; services-platform:modules/llm/services/limits/tenant-limits.ts | coverage-gap | | FND-PFM-LLM-073 | medium | Simultaneous overlap grants, scheduled revocation, notice failure and duplicate creates need complete lifecycle and failure verification. The console has no working create or revoke flow. | ATS-PFM-LLM-014, CAP-PFM-LLM-014, STATE-PFM-LLM-007 | services-platform:modules/llm/tests/e2e/llm-overrides.e2e.spec.js#is refused the grant before a handler runs, whatever the customer’s scope grants; services-platform:modules/llm/services/overrides/allowance-overrides.ts; services-platform:modules/llm/services/overrides/allowance-overrides.ts | coverage-gap | | FND-PFM-LLM-074 | high | Refund eligibility during competing changes, missing-current-period refunds, audit rollback and repeated corrections need focused scenarios. A correction does not generate a threshold notice. | ATS-PFM-LLM-015, CAP-PFM-LLM-015 | services-platform:modules/llm/tests/e2e/llm-adjustments-and-organization.e2e.spec.js#is refused 403 for a Support Representative, who holds the read key and not the manage key; services-platform:modules/llm/services/adjustments/adjust-allowance.ts:requireAdjustable; services-platform:modules/llm/tests/integration/services/adjustments/adjust-allowance.integration.spec.js#refuses a period the organization has no row for and that is not this month, naming the period | coverage-gap | | FND-PFM-LLM-075 | high | Route authorization, invalid body kinds, failed audit, malformed stored content and access revocation during storage reads need explicit HTTP and timing proof. Console content reads remain disabled. | ATS-PFM-LLM-016, CAP-PFM-LLM-016 | services-platform:modules/llm/services/evidence/evidence-access.ts; services-platform:modules/llm/tests/integration/services/evidence/evidence-access.integration.spec.js#refuses the holder once the grant’s window has closed, and records the refusal; services-platform:modules/llm/services/evidence/evidence-access.ts:readInvocationEvidence | coverage-gap | | FND-PFM-LLM-076 | medium | Every live-position transition and the complete customer dashboard journey remain unproved. Exemption availability disagreement needs an executable paired admission and reporting check. | ATS-PFM-LLM-017, CAP-PFM-LLM-017, STATE-PFM-LLM-005 | services-platform:modules/llm/tests/e2e/llm-adjustments-and-organization.e2e.spec.js#is refused to Rhythm staff holding every HQ key, because the routes are console; services-platform:modules/llm/services/usage/organization-usage.ts; services-platform:modules/llm/tests/e2e/llm-adjustments-and-organization.e2e.spec.js#carries no key naming money, a provider, a model, an override or a limit on the summary | coverage-gap | | FND-PFM-LLM-077 | medium | Customer wording at missing reset dates, internal server classifications and full notice-to-console agreement need complete interaction proof beyond individual sentence helpers. | ATS-PFM-LLM-018, CAP-PFM-LLM-018 | services-platform:modules/llm/tests/unit/services/usage/usage-sentences.spec.ts#tells an organization that has used its allowance when it resumes; services-platform:modules/llm/tests/unit/services/usage/usage-sentences.spec.ts#asks an organization with too many requests running to try again in a few seconds; services-platform:modules/llm/tests/unit/services/usage/usage-sentences.spec.ts#tells an organization with a feature turned off to ask Rhythm support | coverage-gap | | FND-PFM-LLM-078 | high | All empty and archived conversation transitions, restart loss, simultaneous sends and deletion during generation lack executable coverage. Prototype entry and sample-owner sharing do not prove customer isolation. | ATS-PFM-LLM-019, CAP-PFM-LLM-019, STATE-PFM-LLM-008 | react-console:components/console/nav-config.ts:unlistedNav; react-console:app/api/agent/conversations/[id]/messages/route.ts:MODEL; react-console:lib/agent/schemas.ts | coverage-gap | | FND-PFM-LLM-079 | high | Applied-work versus failed history-write timing, failed undo-note writes, scope identity races and live refusal need browser verification. Local interpretation is distinct from metered generation. | ATS-PFM-LLM-020, CAP-PFM-LLM-020, STATE-PFM-LLM-008 | react-console:lib/agent/store.ts:createConversation; react-console:components/queries/query-workspace.tsx:refine; react-console:components/console/agent-chat.tsx:undo | coverage-gap | | FND-PFM-LLM-080 | medium | Several inherited test-title citations use a different spelling or a shortened title. The exemption scenario previously cited only a disabled-operation error mapper, which cannot prove the exemption and global-switch experiment; the broad experiment now cites the actual exemption and global-switch integration assertions, while the error mapper binds only its narrower conflict-response experiment. | ATS-PFM-LLM-001, ATS-PFM-LLM-009, ATS-PFM-LLM-013 | services-platform:modules/llm/tests/unit/services/llm-operation-disabled.spec.ts; services-platform:modules/llm/tests/integration/services/reconciliation/llm-settlement-sweep.integration.spec.js; services-platform:modules/llm/services/limits/tenant-limits.ts | inconsistency | | FND-PFM-LLM-081 | high | Most feature experiments have no existing executable suite that proves their entire chain. Component bindings remain useful prerequisites, but mock rendering, local dispatch and a skippable development read probe do not establish complete customer acceptance. No registered suite is executed by this documentation back-fill. | FAS-PFM-LLM-01, FAS-PFM-LLM-02, FAS-PFM-LLM-03, FAS-PFM-LLM-04, FAS-PFM-LLM-05, FAS-PFM-LLM-06, FAS-PFM-LLM-07 | react-console:e2e/mock/hq-ai-usage.mock.spec.ts; react-console:e2e/dev/hq-ai-usage.dev.spec.ts; services-platform:modules/llm/tests/README.md | coverage-gap |
Open Questions
| id | question | why_it_matters | options | recommendation | affects |
|---|---|---|---|---|---|
OQ-PFM-LLM-001 | When should monthly AI capacity count as exhausted? | Reservation-based refusals can precede the status and notice boundary. | (a) Align status and notices with affordability of the next request. (b) Admit while any capacity remains and permit bounded overshoot. (c) Retain admission and qualify the customer wording. | The product should align exhaustion wording and notices with reservation affordability, then define the reservation used for that comparison. | CAP-PFM-LLM-007, CAP-PFM-LLM-008, CAP-PFM-LLM-010, CAP-PFM-LLM-017, FND-PFM-LLM-003 |
OQ-PFM-LLM-002 | Which recipients and delivery path should receive interim allowance notices? | Administrator-free workspaces receive no warning, and the responsibility-based path is absent. | (a) Retain administrator email temporarily with migration debt. (b) Wait for the notification infrastructure. (c) Implement an AI usage responsibility first. | The product should retain administrator email as an explicit interim arrangement and specify migration to responsibility-based recipients. | CAP-PFM-LLM-008, CAP-PFM-LLM-014, FND-PFM-LLM-012 |
OQ-PFM-LLM-003 | What boundary must Jazz AI meet before customer use? | General replies bypass metering and controls, while prototype conversation storage mixes users and workspaces. | (a) Move replies and isolated conversation storage into Platform. (b) Restrict the prototype to demonstration use until that boundary exists. (c) Retain the prototype behavior. | The product should use Platform for metered replies and isolated durable conversations, with prototype access restricted until those requirements are met. | CAP-PFM-LLM-019, CAP-PFM-LLM-020, CAP-PFM-LLM-001, FND-PFM-LLM-001, FND-PFM-LLM-002, FND-PFM-LLM-019 |
OQ-PFM-LLM-004 | What should recovery charge for a request whose real cost was never learned? | Today every request unsettled after a day is charged its full estimate, including requests that never held credits or never reached the provider, which overcharges organizations for Rhythm's own interruptions. | (a) Release requests that never reached the provider and charge the estimate only for a conversation that completed without usage. (b) Charge from the usage the recorded attempts reported, falling back to the estimate. (c) Keep charging the estimate for every stale request. | The product should adopt (a) with (b) for requests that recorded usage, so recovery never charges more than the evidence supports; the period's running count and holds should be released only for what the request recorded. | CAP-PFM-LLM-009, WF-PFM-LLM-004, ADR-PFM-LLM-004, FND-PFM-LLM-032, FND-PFM-LLM-022 |
OQ-PFM-LLM-005 | Should allowance notices move onto the Platform notification pipeline now, before responsibilities exist? | The notices bypass the registry, queue and delivery records every other Platform producer is expected to use, and the interim recipient rule is tied to that direct path. | (a) Move now, keeping the Administrator-role recipients, so deliveries are recorded and retried by the pipeline. (b) Move only when the AI usage responsibility exists. (c) Keep the direct path. | The product should choose (a): delivery history and preferences matter more than the recipient rule, and the recipient resolver can still be swapped for a responsibility later. | CAP-PFM-LLM-008, CAP-PFM-LLM-014, ADR-PFM-LLM-011, FND-PFM-LLM-034, OQ-PFM-LLM-002 |