--- id: WF-PFM-DJE-015 title: Decide and publish the capacity state each minute trigger: The minute sentinel schedule invokes capacity evaluation. product_area: platform module: data-job-engine revision: 1 status: draft design_intent: confirmed origin: >- services-platform:docs/proposals/api-workload-governance/README.md (D1 to D10 approved 2026-10-17; brief 3.2); services-platform:docs/workload-governance.md#Capacity sentinel implementation_evidence: observed inference: [] source_sha: services-platform: ad3f6a6582843008ed0f5a17b131878d15829af1 macroservice-libraries: 54536a441935c6e09fdb7f96f0bec71c52ba4e03 implements: - CAP-PFM-DJE-019 - CAP-PFM-DJE-014 moves: - STATE-PFM-DJE-006 crosses: - SVC-PFM-DJE-capacity - SVC-PFM-DJE-engine hops: - 'n': 1 actor_or_service: Capacity sentinel action: >- Read previous memory, current pin, metric window and an hourly latency baseline. anchor: 'services-platform:modules/capacity/services/capacity/run-sentinel.ts' seam: schedule on_failure: >- Start fresh for unreadable memory or baseline, ignore unreadable pins and pass metric failure to the decision. left_inconsistent: Fallback reads can discard previous comparison context. observer: Sentinel logs. provenance: documented source: >- services-platform:modules/capacity/services/capacity/run-sentinel.ts (module comment, "Failure handling") - 'n': 2 actor_or_service: Capacity decision action: >- Evaluate readings with timing filters, or apply an active pin before signal-failure fallback. anchor: >- services-platform:modules/capacity/services/capacity/decide-state.ts:decideCapacityState seam: sync on_failure: The pure decision handles failed signal input explicitly. left_inconsistent: No external effect has occurred at this step. observer: Transition logs with readings. provenance: test source: >- services-platform:modules/capacity/tests/unit/services/capacity/decide-state.spec.ts#stops bulk and quarters heavy in critical, keeping writes and reads whole - 'n': 3 actor_or_service: Emergency stage throttles action: >- Apply enabled sustained-emergency stage targets, or restore stages when protection relaxes. anchor: >- services-platform:modules/capacity/services/capacity/apply-emergency-throttles.ts:applyStageThrottles seam: sync on_failure: Log unsuccessful targets and retry them; skip malformed declarations. left_inconsistent: Different stages can temporarily carry different throttle settings. observer: Logs and the throttle outcome. provenance: observed source: >- services-platform:modules/capacity/services/capacity/apply-emergency-throttles.ts:CAPACITY_EMERGENCY_THROTTLES_VARIABLE - 'n': 4 actor_or_service: Capacity publication action: Publish to cache and parameter independently and emit decision metrics. anchor: >- services-platform:modules/capacity/services/capacity/publish-state.ts:publishCapacityState seam: event on_failure: >- Log one failed channel; failure of both fails the invocation, with scheduled recovery and dead-letter handling at their configured boundaries. left_inconsistent: Consumers of a failed channel can see an older state. observer: Publish-failure metric and alarms. provenance: documented source: 'services-platform:docs/workload-governance.md#Capacity sentinel' - 'n': 5 actor_or_service: Bulk admission in each adopting service action: >- Read the protection factor and derive the service and workspace admission budgets. anchor: >- macroservice-libraries:packages/data-jobs/src/admission-budget.ts:resolveAdmissionBudget seam: sync on_failure: Use full capacity when factor reading fails and record the read failure. left_inconsistent: Fallback can admit work above the last restrictive factor. observer: Admission logs. provenance: documented source: >- macroservice-libraries:packages/data-jobs/src/admission-budget.ts:resolveAdmissionBudget idempotency: >- Ticks reevaluate current input and memory. Unsuccessful stage changes remain eligible for another attempt; a repeated or missed tick changes timing rather than creating business records. concurrency: >- Reserved concurrency serializes sentinel invocations. Its memory and independent publication channels still have separate persistence outcomes. branches: - >- A valid unexpired staff pin overrides readings and change spacing until release or expiry. waits: - >- Automatic changes respect five-minute spacing; emergency has an additional state timer and stage changes need five further minutes. evidence: - >- services-platform at ad3f6a65: modules/capacity/services/capacity/run-sentinel.ts, decide-state.ts, publish-state.ts, apply-emergency-throttles.ts, capacity-signals.ts, capacity-baseline.ts; aws/logic/capacity-sentinel.lambda.ts; docs/workload-governance.md. - >- macroservice-libraries at 54536a44: packages/data-jobs/src/admission-budget.ts; README.md#The bulk lane. controls: [TC-AVL-001, TC-AVL-004, TC-AVL-005, TC-AZ-001, TC-AZ-002, TC-AZ-003, TC-AZ-004, TC-AZ-005, TC-AZ-006, TC-DAT-001, TC-DAT-003, TC-DAT-006, TC-IAM-001, TC-IAM-002, TC-IAM-003, TC-IAM-004, TC-IAM-005, TC-IAM-006, TC-LOG-001, TC-LOG-002, TC-LOG-003, TC-LOG-004, TC-LOG-005, TC-MON-001, TC-MON-002, TC-PI-001, TC-PI-002, TC-PI-003, TC-PI-004, TC-PI-005, TC-PRV-001, TC-PRV-002, TC-TEN-001, TC-TEN-002, TC-TEN-003, TC-VEN-001, TC-VEN-002, TC-VEN-003] --- # Decide And Publish The Capacity State Each Minute ## What It Does The sentinel reads pressure, prior memory and staff pins, then publishes protection factors through two independent channels. API stage changes are conditional on sustained emergency and a separate disabled switch. ## The Flow ```mermaid flowchart TD H1["Capacity sentinel"] --> H2 H2["Capacity decision"] --> H3 H3["Emergency stage throttles"] --> H4 H4["Capacity publication"] --> H5 H5["Bulk admission in each adopting service"] ``` ## When Something Fails Unreadable signals release protection after a held tick unless an active pin overrides them. Publication failure is independent of decision fallback, and partial stage updates or stale channels remain observable consistency limits. ## Open Design Questions Operational evidence must establish suitable thresholds before emergency stage changes are enabled.