--- id: WF-PFM-OPS-003 title: Settle stalled jobs on the five-minute pass product_area: platform module: operations revision: 1 status: draft design_intent: proposed implementation_evidence: observed inference: [] trigger: The five-minute Platform schedule invokes job reconciliation. source_sha: services-platform: ad3f6a6582843008ed0f5a17b131878d15829af1 macroservice-libraries: 54536a441935c6e09fdb7f96f0bec71c52ba4e03 implements: - CAP-PFM-OPS-002 - CAP-PFM-OPS-003 - CAP-PFM-OPS-004 moves: - STATE-PFM-OPS-001 - STATE-PFM-OPS-002 - STATE-PFM-OPS-003 crosses: - SVC-PFM-OPS-jobs idempotency: >- Reconciliation rereads final status before settling and leaves final jobs closed. Consequences must tolerate repetition. Missing execution probes append their evidence to history. concurrency: >- One reserved reconciler invocation runs at a time, with a four-minute timeout. Task writes can still race it and use version checks. Candidate limits or failures defer work to later passes. hops: - 'n': 1 actor_or_service: Reconciler action: >- Validate invocation, register Platform kinds and select up to two hundred open jobs silent for fifteen minutes. anchor: 'services-platform:modules/jobs/aws/lambda/RECONCILER.handler.ts:handler' seam: schedule on_failure: Fail the invocation and await a later scheduled run. left_inconsistent: Eligible jobs can remain open until another successful selection. observer: Function logs and the alarm for missing scheduled invocations. provenance: observed source: 'services-platform:modules/jobs/aws/logic/jobs-reconciler.lambda.ts:createJobsReconcilerFunction' - 'n': 2 actor_or_service: Reconciler action: Describe candidate execution and follow a recorded continuation within the bounded search. anchor: 'macroservice-libraries:packages/jobs/src/reconciler/job-reconciler.ts:handleCandidate' seam: sync on_failure: Defer transient probe errors and alert for unregistered kinds. left_inconsistent: The candidate retains its existing outcome. observer: Reconciliation totals and operational alerts. provenance: observed source: 'macroservice-libraries:packages/jobs/src/reconciler/job-reconciler.ts:handleCandidate' - 'n': 3 actor_or_service: Reconciler action: >- Repair drift first; otherwise retain running execution, count missing probes, fail unstarted jobs or derive settlement from terminal execution. anchor: 'macroservice-libraries:packages/jobs/src/reconciler/job-reconciler.ts:reconcileStalledJobs' seam: sync on_failure: Defer any unsuccessful candidate processing. left_inconsistent: The job can remain open despite terminal execution. observer: Reconciliation history and recorded probe misses. provenance: test source: >- services-platform:modules/jobs/tests/integration/lifecycle/job-reconciliation/job-reconciliation.integration.spec.js#halts the job only on the third pass - 'n': 4 actor_or_service: Job type registration action: >- Apply the registered effect before explicit failure; apply it after settlement when apparent execution success becomes job failure. anchor: 'macroservice-libraries:packages/jobs/src/engine/job-type-registry.ts:applyJobFailureConsequence' seam: sync on_failure: >- A pre-settlement exception leaves an open candidate; a post-settlement exception does not restore eligibility. left_inconsistent: A final failed job can lack its business consequence or requester attempt. observer: Job detail and the requester-notice outcome. provenance: test source: 'macroservice-libraries:packages/jobs/src/reconciler/job-reconciler.ts:settleFromReconciliation' - 'n': 5 actor_or_service: Job engine action: >- Close running and retrying steps and commit the reconciliation event with root settlement; correct success if either in-flight status remains. anchor: 'macroservice-libraries:packages/jobs/src/engine/job-engine.ts:settleJobFromReconciliation' seam: sync on_failure: Roll back settlement writes and reconsider an open candidate later. left_inconsistent: 'No partial settlement commits, though a prior business consequence can already exist.' observer: Job history is the observation surface. provenance: test source: >- services-platform:modules/jobs/tests/integration/lifecycle/job-reconciliation/job-reconciliation.integration.spec.js#corrects the job to failed, never succeeded, keeping the progress its steps earned - 'n': 6 actor_or_service: Reconciler action: >- Probe recently final jobs with handles and alert when their executions still run, retaining the recorded outcome. anchor: 'macroservice-libraries:packages/jobs/src/reconciler/job-reconciler.ts:reconcileStalledJobs' seam: sync on_failure: Log a failed check and defer it. left_inconsistent: Work may continue behind an already final job account. observer: An operational alert exposes this outcome. provenance: test source: >- services-platform:modules/jobs/tests/integration/lifecycle/job-reconciliation/job-reconciliation.integration.spec.js#raises one alert and changes nothing - 'n': 7 actor_or_service: Workspace lifecycle action: Invoke the workspace owner’s repair for provisioning stranded behind a failed job. anchor: >- services-platform:modules/tenants/services/provisioning/repair-stranded-provisioning.ts:repairStrandedTenantProvisioning seam: sync on_failure: 'After a successful jobs pass, a thrown repair error fails the invocation and skips website repair.' left_inconsistent: Pending website repair waits for another invocation. observer: Function log and error alarm is the observation surface. provenance: observed source: 'services-platform:modules/jobs/aws/lambda/RECONCILER.handler.ts:handler' - 'n': 8 actor_or_service: Website studio action: Invoke website reconciliation for changes that have waited beyond its ten-minute threshold. anchor: 'services-platform:modules/website-studio/services/pipeline/reconcile.ts' seam: sync on_failure: Contain and log the website pass error at its own boundary. left_inconsistent: Pending website changes may remain unresolved. observer: The operation’s own log exposes the outcome. provenance: observed source: 'services-platform:modules/jobs/aws/lambda/RECONCILER.handler.ts:handler' branches: - condition: The jobs pass itself throws path: >- After job-pass failure, the handler attempts provisioning repair and website reconciliation, contains their errors, then rethrows the original pass error. waits: - what: Stall threshold how_long: fifteen minutes of silence - what: Missing execution how_long: three consecutive passes evidence: - >- services-platform at ad3f6a6: modules/jobs/aws/lambda/RECONCILER.handler.ts, aws/logic/jobs-reconciler.lambda.ts, db/cross-job-query.view.ts, tests/integration/lifecycle/job-reconciliation. - >- macroservice-libraries at 54536a4: reconciler/job-reconciler.ts, reconciler/job-reconciler-handler.ts, aws/job-reconciliation.schedule.ts. - >- codex examination 2026-10-23: checked the pinned source anchors, planning references and enforcement boundaries; unresolved limits are recorded in module findings. controls: [TC-AVL-001, TC-AVL-004, TC-AVL-005, TC-AZ-001, TC-AZ-002, TC-AZ-003, TC-AZ-004, TC-AZ-005, TC-DAT-001, TC-DAT-002, TC-DAT-003, TC-DAT-004, TC-IAM-001, TC-IAM-002, TC-IAM-003, TC-IAM-004, TC-IAM-005, TC-IAM-006, TC-LOG-001, TC-LOG-002, TC-LOG-003, TC-LOG-004, TC-LOG-005, TC-MON-001, TC-MON-002, TC-PI-001, TC-PI-002, TC-PI-003, TC-PI-004, TC-PI-005, TC-PRV-001, TC-PRV-002, TC-TEN-001, TC-TEN-002, TC-TEN-003, TC-VEN-001, TC-VEN-002, TC-VEN-003] --- # Settle stalled jobs on the five-minute pass ## What It Does Reconciliation compares silent job accounts with orchestration evidence. It also checks recently final jobs for still-running executions. The Platform invocation hosts workspace provisioning repair and website repair after the shared jobs pass. ## The Flow ```mermaid flowchart TD H0["Validate invocation, register Platform kinds and select up to two hundred open jobs silent for fifteen minutes."] H1["Describe candidate execution and follow a recorded continuation within the bounded search."] H0 --> H1 H2["Repair drift first; otherwise retain running execution, count missing probes, fail unstarted jobs or derive settlement from terminal execution."] H1 --> H2 H3["Apply the registered effect before explicit failure; apply it after settlement when apparent execution success becomes job failure."] H2 --> H3 H4["Close running and retrying steps and commit the reconciliation event with root settlement; correct success if either in-flight status remains."] H3 --> H4 H5["Probe recently final jobs with handles and alert when their executions still run, retaining the recorded outcome."] H4 --> H5 H6["Invoke the workspace owner’s repair for provisioning stranded behind a failed job."] H5 --> H6 H7["Invoke website reconciliation for changes that have waited beyond its ten-minute threshold."] H6 --> H7 ``` ## When Something Fails Probe errors and candidate limits postpone recovery. Explicit failures apply their business effect before settlement, but a success corrected for running or retrying steps applies it afterwards; failure there can leave a final job without recovery. If the jobs pass throws, the handler attempts both other repairs before rethrowing. After a successful pass, a provisioning-repair exception prevents the website pass. The silence alarm measures invocation absence rather than successful completion. ## Open Design Questions The shared invocation couples unrelated recovery paths. The corrected-success consequence gap remains a finding that needs resolution before assuming uniform recovery.