--- id: STATE-PFM-DJE-006 title: Capacity protection state object: DOM-PFM-DJE-006 product_area: platform module: data-job-engine revision: 1 status: draft design_intent: confirmed origin: >- services-platform:docs/proposals/api-workload-governance/README.md (D1 to D10 approved 2026-10-17; brief 3.2, the capacity sentinel); thresholds are proposed starting values implementation_evidence: observed source_sha: services-platform: ad3f6a6582843008ed0f5a17b131878d15829af1 states: - name: normal meaning: All workload factors remain at their full values. - name: constrained meaning: >- Bulk and heavy factors are one half, leaving ordinary writes and reads whole. - name: critical meaning: >- New bulk work stops and heavy work uses one quarter; writes and reads retain full factors. - name: emergency meaning: >- Bulk remains stopped, heavy stays at one quarter and write/read factors halve; sustained pressure can reduce API stages only with the switch enabled. transitions: - from: normal to: constrained trigger: Evaluate sustained pressure guard: Three supporting readings and automatic spacing permit constrained state side_effects: - 'publish factors; record gauge, change metric and supporting readings' who: Capacity sentinel provenance: test source: >- services-platform:modules/capacity/tests/unit/services/capacity/decide-state.spec.ts#makes no second change within five minutes of the last - from: constrained to: critical trigger: Evaluate sustained critical pressure guard: >- Three critical readings and spacing permit escalation; scaling by itself is insufficient side_effects: - publish a zero bulk factor who: Capacity sentinel provenance: observed source: >- services-platform:modules/capacity/services/capacity/decide-state.ts:assessReading - from: critical to: emergency trigger: Escalate sustained critical state guard: >- Critical has lasted at least three minutes and the five-minute change spacing also permits the move side_effects: - >- halve write and read factors; consider API stage changes after five more emergency minutes who: Capacity sentinel provenance: observed source: >- services-platform:modules/capacity/services/capacity/decide-state.ts:decideCapacityState - from: emergency to: critical trigger: Recover from emergency guard: >- Five below-critical readings and spacing permit recovery; active stage throttles are first restored in a separate tick side_effects: - restore stage settings before stepping down who: Capacity sentinel provenance: documented source: 'services-platform:docs/workload-governance.md#Capacity sentinel' - from: critical to: constrained trigger: Recover from critical guard: Five calmer readings and spacing permit recovery side_effects: - publish half bulk and heavy factors who: Capacity sentinel provenance: observed source: >- services-platform:modules/capacity/services/capacity/decide-state.ts:decideCapacityState - from: constrained to: normal trigger: Recover or fall back to normal guard: 'Recovery satisfies spacing, or two signal reads fail without an active pin' side_effects: - restore full workload factors who: Capacity sentinel provenance: observed source: >- services-platform:modules/capacity/services/capacity/decide-state.ts:decideCapacityState - from: normal to: emergency trigger: Apply a staff override guard: The pin remains unexpired and overrides automatic spacing side_effects: - publish the pinned state; request appropriate stage-throttle handling who: Authorized Rhythm staff and the sentinel provenance: test source: >- services-platform:modules/capacity/tests/integration/services/pins/capacity-pins.integration.spec.js#records the pin and its removal, each with what it was and what it became - from: normal to: critical trigger: Enter critical directly guard: Three critical readings satisfy automatic spacing side_effects: - publish zero bulk and quarter heavy factors who: Capacity sentinel provenance: observed source: >- services-platform:modules/capacity/services/capacity/decide-state.ts:decideCapacityState - from: critical to: normal trigger: Release protection after signal loss guard: Two consecutive reads fail and no active pin overrides them side_effects: - publish full factors and request stage restoration who: Capacity sentinel provenance: observed source: >- services-platform:modules/capacity/services/capacity/decide-state.ts:decideCapacityState - from: emergency to: normal trigger: Release emergency after signal loss guard: Two consecutive reads fail and no active pin overrides them side_effects: - publish full factors and request stage restoration who: Capacity sentinel provenance: observed source: >- services-platform:modules/capacity/services/capacity/decide-state.ts:decideCapacityState invalid: - from: constrained to: emergency why: >- Automatic escalation cannot reach emergency without previous critical state; an active pin can select it directly. concurrency: >- Reserved concurrency allows one sentinel invocation at a time. Each tick reads the current pin and previous memory; unreadable pins act as absent. Release or expiry resumes automatic decisions from the pinned state. Publication channels settle independently and can temporarily disagree. invariants: - INV-DOM-PFM-DJE-006-01 - INV-DOM-PFM-DJE-006-02 - INV-DOM-PFM-DJE-006-03 - INV-DOM-PFM-DJE-006-07 evidence: - >- services-platform at ad3f6a65: modules/capacity/services/capacity/decide-state.ts, run-sentinel.ts, apply-emergency-throttles.ts, publish-state.ts; services/pins/capacity-pins.ts; docs/workload-governance.md. --- # Capacity Protection State ## The Lifecycle ```mermaid stateDiagram-v2 [*] --> normal normal --> constrained: evaluate sustained pressure constrained --> critical: evaluate sustained critical pressure critical --> emergency: escalate sustained critical state emergency --> critical: recover from emergency critical --> constrained: recover from critical constrained --> normal: recover or fall back to normal normal --> emergency: apply a staff override normal --> critical: enter critical directly critical --> normal: release protection after signal loss emergency --> normal: release emergency after signal loss ``` ## Transitions Explained Three supporting readings select their supported pressure level, so normal can move straight to critical. Emergency requires existing critical state and spacing. Five calmer readings recover one state at a time; when emergency stage limits stand, restoration occupies a separate tick first. Pins can select any state immediately on evaluation. Without an active pin, one failed signal read holds protection and a second restores normal, including from critical or emergency. ## What Cannot Happen Automatic escalation cannot jump directly to emergency, and scaling alone cannot classify critical pressure. An active staff pin can still select emergency from any previous state. ## Concurrency and Timing The minute schedule and reserved concurrency serialize decision writers. Consumer caching adds propagation delay. Emergency API stage changes require their disabled-by-default switch and sustained critical assessments.