Rhythm 2.0 Specifications

Appearance

CAP-PFM-LLM-010Monitor AI usage across organizations

Markdown

CAP-PFM-LLM-010

Monitor AI usage across organizations

Markdown
  • DocumentDraft
  • Product intentProposed
  • ImplementationObserved

Planning

Feature
FEAT-PFM-LLM-04
Exercised by
FAS-PFM-LLM-04-01, FAS-PFM-LLM-04-02, FAS-PFM-LLM-04-03, FAS-PFM-LLM-04-04, FAS-PFM-LLM-04-05
Key concepts
CON-PFM-LLM-004, CON-PFM-LLM-003
Actors
ACT-PFM-033, ACT-PFM-004, ACT-PFM-012
Journeys
JNY-PFM-LLM-002
Depends on
CAP-PFM-LLM-007, CAP-PFM-LLM-009
Related
CAP-PFM-LLM-005, CAP-PFM-LLM-011
Proved by
ATS-PFM-LLM-010

Implementation

Executable suites
TST-PFM-LLM-003, TST-PFM-LLM-009, TST-PFM-LLM-017, TST-PFM-LLM-044, TST-PFM-LLM-047, TST-PFM-LLM-048, TST-PFM-LLM-049, TST-PFM-LLM-054, TST-PFM-LLM-081, TST-PFM-LLM-082

KindCapabilityProduct areaPlatformModuleAI Requests, Usage and the Jazz AI TeammateRevision1

Read atservices-platform ad3f6a65react-console baf6dae3

Monitor AI usage across organizations ​

Purpose ​

Help Rhythm staff compare cost, activity and allowance pressure across customers.

How It Works ​

Authorized staff open the system-wide AI Usage view and choose a reporting window. Settled daily figures support its totals, trends and breakdowns. Live allowance positions supply the organization statuses beside those historical figures. Staff can sort organizations and follow a row into a particular customer’s usage. Smaller trend series fold into an Other series without losing their totals. Costly closed executions provide another way to identify expensive work. A failed section reports its unavailable state while independent sections can continue. The hourly rebuild maintains a progress watermark, and operators receive an alarm when that work falls behind.

Relationships ​

  • Admit AI work against the organization’s monthly allowance supplies the prerequisite behavior for this outcome.
  • Recover interrupted requests and identify accounting drift supplies the prerequisite behavior for this outcome.

The Formal Contract

The narrative above explains; the sections below are authoritative — the rules, scenarios and tests this document binds.

Outcomes
Usage reports compare activity and charges across reporting windows.Organization status highlights capacity concerns.Daily aggregates retain organization and usage dimensions.
Surfaces
consoleapijob
Constraints
Reporting uses UTC days while allowance periods follow the organization’s calendar.
Intent sources
services-platform:docs/proposals/llm-observability/11-decisions.md#D-7services-platform:docs/milestones/plans/PLAT-M1-plan.md#4. Assumptions

Controls

21 controls apply, from the corpus control catalogue.

TC-AVL-001 TC-AZ-001 TC-AZ-002 TC-AZ-003 TC-AZ-004 TC-AZ-005 TC-IAM-003 TC-LOG-001 TC-LOG-004 TC-LOG-005 TC-MON-001 TC-MON-002 TC-PI-001 TC-PI-003 TC-PI-005 TC-TEN-001 TC-TEN-002 TC-TEN-003 TC-VEN-001 TC-VEN-002 TC-VEN-003

Permissions

KeyChecked atScope
hq.llm-usage.readapiglobal
hq.llm-usage.readconsoleglobal

Business Rules

12 rules, 8 to be proved before approval. A rule's product intent is derived from its origin: 0 confirmed by a product decision, the rest inheriting this capability's proposed.

IdRuleKindEnforced atProvenance
01must prove
Proposed
Only staff with the HQ AI usage read key can read system-wide usage; customer-scoped authority and unrelated HQ keys cannot authorize it.
proved by ATS-PFM-LLM-010-01, ATS-PFM-LLM-010-09, TST-PFM-LLM-009, TST-PFM-LLM-054
permissionapitest
services-platform:modules/llm/tests/e2e/llm-usage.e2e.spec.js#refuses a customer’s administrator, whatever they hold at their customer’s scope
02must prove
Proposed
Reporting accepts month to date or thirty or ninety UTC days including today and compares with the immediately preceding equal-length period; unknown windows are rejected.
proved by ATS-PFM-LLM-010-02, TST-PFM-LLM-009, TST-PFM-LLM-049, TST-PFM-LLM-082
calculationapi, serviceobserved
services-platform:modules/llm/services/usage-reports/platform-usage.ts
03must prove
Proposed
Live status is disabled for an inactive organization or disabled AI, otherwise stopped when committed usage reaches allowance, warned from 80%, and within below that; later capacity can improve status.
proved by ATS-PFM-LLM-007-13, ATS-PFM-LLM-010-03, TST-PFM-LLM-081
calculationserviceobserved
services-platform:modules/llm/services/usage-reports/usage-status.ts
04must prove
Proposed
Utilisation divides charged plus held credits by base plus active capacity, rounds to one decimal, may exceed 100% and has no percentage for a zero allowance.
proved by ATS-PFM-LLM-010-04, TST-PFM-LLM-081
calculationserviceobserved
services-platform:modules/llm/services/usage-reports/usage-status.ts
05must prove
Proposed
Trends can group by organization, application, model, provider or operation, retain ten largest series and aggregate the remainder exactly as Other, omitting empty dates.
proved by ATS-PFM-LLM-010-05, TST-PFM-LLM-049
calculationserviceobserved
services-platform:modules/llm/services/usage-reports/platform-usage.ts
06
Proposed
Organization rows combine spend, requests, refusals, failures, utilisation, allowance, capacity and status, with cost, utilisation or refusal sorting and stable pagination.
policyapi, serviceobserved
services-platform:modules/llm/services/usage-reports/platform-usage.ts
07
Proposed
Breakdowns provide group cost, activity, average latency and failure rate; cross-organization execution leaders identify expensive closed tasks and organizations whose reads failed.
proved by ATS-PFM-LLM-010-12, TST-PFM-LLM-048
calculationserviceobserved
services-platform:modules/llm/services/usage-reports/platform-usage.ts
08must prove
Proposed
Hourly rebuilding replaces each affected organization’s UTC-day totals, assigns requests to their start day, excludes running work, and retains refused, failed, released and settled classifications with their applicable charges.
proved by ATS-PFM-LLM-005-11, ATS-PFM-LLM-009-06, ATS-PFM-LLM-010-06, ATS-PFM-LLM-010-10, TST-PFM-LLM-017, TST-PFM-LLM-047
calculationservice, databaseobserved
services-platform:modules/llm/services/reconciliation/rollup-settled-invocations.ts
09must prove
Proposed
Rollup progress trails the run start by five minutes and stops at the earliest unfinished change on error or exhausted time without moving backward; delay beyond two hours alarms.
proved by ATS-PFM-LLM-010-07, ATS-PFM-LLM-010-11, TST-PFM-LLM-044, TST-PFM-LLM-047
invariantservice, databaseobserved
services-platform:modules/llm/services/reconciliation/rollup-settled-invocations.ts
10must prove
Proposed
HQ presents list-price dollars with the equivalent credits alongside.
origin: services-platform:docs/proposals/llm-observability/11-decisions.md#D-7
proved by ATS-PFM-LLM-010-08, TST-PFM-LLM-003
policyconsoletest
react-console:lib/hq/ai-usage-format.test.ts#shows cents as dollars beside the credits
11
Proposed
An unavailable report section shows its failure state independently of successfully read sections.
policyservice, consoleobserved
react-console:components/console/hq/ai-usage/ai-usage-tiles.tsx
12
Proposed
HQ query recipes cover organizational, daily and model spend and organizational refusals; the HQ reporting reader can query daily rollups without gaining access to organization request records.
permissiondatabasedocumented
services-platform:docs/llm.md#Reporting

Scenarios

IdGivenWhenThenProvenance
01Two organizations have requests across a UTC day boundary.The aggregate rebuild runs.Each request contributes to its own organization and start day.test
services-platform:modules/llm/tests/integration/services/reconciliation/rollup-settled-invocations.integration.spec.js#keeps the other organization's calls in its own cell, with the four token sums as reported
02A support reader opens the HQ usage overview.Its mocked reads complete.The page renders metrics, trends, organization states and breakdowns.test
react-console:e2e/mock/hq-ai-usage.mock.spec.ts#shows the tiles, the trend, who needs attention and the organizations with their states

The Markdown files under specs/ in rhythmsoftware/rhythm-v2-docs are the canonical source of truth. This reader is a generated, read-only projection of them.

Rhythm Software