Control capacity protection during database pressure

Purpose

Capacity protection reduces shared workload budgets under database pressure and allows staff to hold a bounded state during an incident.

How It Works

The sentinel periodically reads database pressure and API latency signals. It classifies those readings before deciding whether a state transition is justified. Several consistent readings and a spacing interval prevent rapid oscillation. More severe states reduce bulk and heavy budgets first. Sustained emergency can also reduce ordinary write and read budgets. Authorized staff can pin a known state with an expiry and reason. The pin takes precedence over automatic readings until it expires. Emergency stage changes require a separate switch and declared throttle targets. Independent publication channels distribute the factors. The thresholds remain proposed values whose operational fitness needs load evidence.

Relationships

File admission uses the published bulk factor to reduce its budget.

Show evidence (sources, provenance, origins, pinned commits)

The Formal Contract

The narrative above explains; the sections below are authoritative — the rules, scenarios and tests this document binds.

Outcomes: Repeated pressure readings guide gradual state changes.Bulk work yields capacity before ordinary traffic is reduced.Staff overrides require a reason, expiry and audit record.
Surfaces: apijob
Intent sources: services-platform:docs/workload-governance.md#Capacity sentinel

Controls

23 controls apply, from the corpus control catalogue.

TC-AVL-001 TC-AVL-005 TC-AZ-001 TC-AZ-002 TC-AZ-003 TC-AZ-004 TC-AZ-005 TC-AZ-006 TC-IAM-003 TC-LOG-001 TC-LOG-002 TC-LOG-003 TC-LOG-005 TC-MON-001 TC-PI-001 TC-PI-003 TC-PI-005 TC-TEN-001 TC-TEN-002 TC-TEN-003 TC-VEN-001 TC-VEN-002 TC-VEN-003

Permissions

Key	Checked at	Scope
`hq.capacity.manage`	api	global

Business Rules

9 rules, 8 to be proved before approval. A rule's product intent is derived from its origin: 0 confirmed by a product decision, the rest inheriting this capability's proposed.

Id	Rule	Kind	Enforced at	Provenance
`01`must prove Proposed	Minute readings classify pressure using capacity utilization above 80%, load above 1.5 times the derived processor estimate, connection borrowing above one second, API latency above twice baseline or detected scaling; saturation or borrowing above three seconds is critical. proved by `ATS-PFM-DJE-019-01`, `TST-PFM-DJE-159`	calculation	service	observed services-platform:modules/capacity/services/capacity/decide-state.ts:assessReading
`02`must prove Proposed	Automatic escalation requires three supporting readings, recovery steps down after five calmer readings, and changes are spaced by five minutes; sustained critical readings can then enter emergency, while scaling alone reaches only constrained. proved by `ATS-PFM-DJE-019-02`, `TST-PFM-DJE-159`	calculation	service	test services-platform:modules/capacity/tests/unit/services/capacity/decide-state.spec.ts#makes no second change within five minutes of the last
`03`must prove Proposed	Constrained applies 0.5 bulk and heavy factors; critical applies zero bulk and 0.25 heavy; emergency additionally applies 0.5 write and read factors. proved by `ATS-PFM-DJE-014-12`, `ATS-PFM-DJE-019-03`, `ATS-PFM-DJE-019-10`, `ATS-PFM-DJE-019-11`, `ATS-PFM-DJE-020-05`, `TST-PFM-DJE-159`	calculation	service	test services-platform:modules/capacity/tests/unit/services/capacity/decide-state.spec.ts#stops bulk and quarters heavy in critical, keeping writes and reads whole
`04`must prove Proposed	Capacity inspection and pin management use staff-only routes and capacity authority; customer-scoped administrators cannot use them. proved by `ATS-PFM-DJE-019-04`, `TST-PFM-DJE-152`	permission	api	test services-platform:modules/capacity/tests/e2e/capacity-pins.e2e.spec.js#refuses a customer's administrator, whatever they hold at their customer's scope
`05`must prove Proposed	A capacity pin requires a known state, nonblank reason and future expiry within 24 hours; invalid input names its field without changing the stored pin. proved by `ATS-PFM-DJE-019-05`, `TST-PFM-DJE-153`	validation	api, service, database	test services-platform:modules/capacity/tests/integration/services/persistence/capacity-state-pins.integration.spec.js#refuses an expiry more than a day ahead, naming expires_at
`06`must prove Proposed	Pin and release operations audit their previous and resulting values; the next sentinel tick applies an active pin without the automatic spacing restriction. proved by `ATS-PFM-DJE-019-06`, `TST-PFM-DJE-155`	policy	service	test services-platform:modules/capacity/tests/integration/services/pins/capacity-pins.integration.spec.js#records the pin and its removal, each with what it was and what it became
`07`must prove Proposed	When enabled and emergency readings persist for five further minutes, the sentinel applies declared emergency stage throttles and restores their normal values on recovery. proved by `ATS-PFM-DJE-019-07`, `TST-PFM-DJE-156`, `TST-PFM-DJE-159`	policy	service	documented services-platform:docs/workload-governance.md#Capacity sentinel
`08`must prove Proposed	Without an active pin, a failed signal read holds the previous state for one tick and a second consecutive failure returns it to normal. proved by `ATS-PFM-DJE-019-08`, `TST-PFM-DJE-159`	policy	service	observed services-platform:modules/capacity/services/capacity/decide-state.ts:decideCapacityState
`09` Proposed	The sentinel publishes its state independently to cache and parameter channels so one failed write does not prevent the other. proved by `ATS-PFM-DJE-019-09`	policy	service	documented services-platform:docs/workload-governance.md#Capacity sentinel

Scenarios

Id	Given	When	Then	Provenance
`01`	Three consecutive readings show capacity utilization above 80%.	The sentinel evaluates the new window.	It enters constrained and publishes a bulk factor of one half.	test services-platform:modules/capacity/tests/unit/services/capacity/decide-state.spec.ts#enters constrained on three constrained readings in a row

Planning

Implementation

Control capacity protection during database pressure ​

Purpose ​

How It Works ​

Relationships ​