Resolve interrupted background work

Purpose

Close the gap between a job’s stored status and the execution that actually ran.

How It Works

The scheduled process looks for old job records that have stopped receiving updates. It asks the workflow service whether the underlying execution is still active or has ended. The response determines whether the record needs repair, settlement or another lookup later. Business failure consequences run before the failed outcome is stored. That ordering lets the next pass try again if a consequence cannot finish. A settled job stays final even when its execution disagrees. An alert calls attention to that disagreement. Workspace repair and website reconciliation share the scheduled invocation. After a successful jobs pass, repair failure stops the invocation before website reconciliation runs. Transient lookup faults defer settlement, and recorded continuation can redirect the execution lookup.

Relationships

The job record supports reconciliation, notices and both monitors; owning modules retain responsibility for the work itself.

Show evidence (sources, provenance, origins, pinned commits)

The Formal Contract

The narrative above explains; the sections below are authoritative — the rules, scenarios and tests this document binds.

Outcomes: Interrupted work reaches an explainable resolution.Reconciled failure applies the same business consequence as ordinary failure.
Surfaces: job
Preconditions: The reconciler registers the kind and can discover its family through the service’s job views.
Postconditions: The pass repairs, settles or preserves each examined job according to the execution evidence.
Constraints: The reconciler reads its service’s job views, probes workflow executions and changes records through the engine.

Controls

24 controls apply, from the corpus control catalogue.

TC-AVL-001 TC-AVL-004 TC-AZ-001 TC-AZ-002 TC-AZ-003 TC-AZ-004 TC-IAM-001 TC-IAM-006 TC-LOG-002 TC-LOG-003 TC-LOG-005 TC-MON-001 TC-MON-002 TC-PI-001 TC-PI-002 TC-PI-003 TC-PI-004 TC-PI-005 TC-TEN-001 TC-TEN-002 TC-TEN-003 TC-VEN-001 TC-VEN-002 TC-VEN-003

Business Rules

12 rules, 11 to be proved before approval. A rule's product intent is derived from its origin: 0 confirmed by a product decision, the rest inheriting this capability's proposed.

Id	Rule	Kind	Enforced at	Provenance
`01`must prove Proposed	The five-minute reconciliation pass considers jobs silent for fifteen minutes and converts timed-out, failed or unrequested aborted executions to failed jobs, preserving earned progress and failing any running step. proved by `ATS-PFM-OPS-002-01`, `TST-PFM-OPS-069`	lifecycle	service	test services-platform:modules/jobs/tests/integration/lifecycle/job-reconciliation/job-reconciliation.integration.spec.js#settles the job failed, with the reconciled summary and the progress its steps earned
`02`must prove Proposed	A reported successful execution settles failed if a recorded step remains running or retrying. proved by `ATS-PFM-OPS-002-02`, `TST-PFM-OPS-069`	calculation	service	test services-platform:modules/jobs/tests/integration/lifecycle/job-reconciliation/job-reconciliation.integration.spec.js#corrects the job to failed, never succeeded, keeping the progress its steps earned
`03`must prove Proposed	A stalled job with no execution handle ends failed as never started, leaving its steps queued. proved by `ATS-PFM-OPS-001-09`, `ATS-PFM-OPS-002-03`, `ATS-PFM-OPS-002-09`, `ATS-PFM-OPS-003-08`, `TST-PFM-OPS-069`	lifecycle	service	test services-platform:modules/jobs/tests/integration/lifecycle/job-reconciliation/job-reconciliation.integration.spec.js#settles it failed, saying the job never started
`04`must prove Proposed	Three consecutive unsuccessful execution lookups are required before reconciliation fails a missing execution, with each miss added to history. proved by `ATS-PFM-OPS-002-04`, `TST-PFM-OPS-069`	policy	service	test services-platform:modules/jobs/tests/integration/lifecycle/job-reconciliation/job-reconciliation.integration.spec.js#halts the job only on the third pass
`05`must prove Proposed	Reconciliation leaves recently active jobs and genuinely running executions unchanged, including their history. proved by `ATS-PFM-OPS-002-05`, `TST-PFM-OPS-069`	policy	service	test services-platform:modules/jobs/tests/integration/lifecycle/job-reconciliation/job-reconciliation.integration.spec.js#leaves the root exactly as it was, version and all
`06`must prove Proposed	A job settled within the preceding day whose execution still runs raises an operational alert without reopening its record. proved by `ATS-PFM-OPS-002-06`, `TST-PFM-OPS-069`	policy	service	test services-platform:modules/jobs/tests/integration/lifecycle/job-reconciliation/job-reconciliation.integration.spec.js#raises one alert and changes nothing
`07`must prove Proposed	Reconciliation applies the registered failure consequence before settlement, and that consequence must tolerate repeat application. ✓ An unstarted provisioning job moves its workspace into recoverable failure.✗ A dead execution cannot leave provisioning dependent on a failure task that never ran. proved by `ATS-PFM-OPS-002-07`, `ATS-PFM-OPS-002-10`	policy	service	test macroservice-libraries:packages/jobs/src/reconciler/job-reconciler.spec.ts#applies it before the settlement, so a consequence that fails leaves the job for the next pass
`08`must prove Proposed	A mismatch between the job summary and step records causes summary repair rather than settlement. proved by `ATS-PFM-OPS-002-08`, `TST-PFM-OPS-069`	calculation	service	test services-platform:modules/jobs/tests/integration/lifecycle/job-reconciliation/job-reconciliation.integration.spec.js#repairs the summary instead of settling the job
`09`must prove Proposed	The reconciler reserves one concurrent invocation, has a silence alarm, and rejects malformed invocation requests before running a pass. no proof recorded yet	policy	service	observed services-platform:modules/jobs/aws/logic/jobs-reconciler.lambda.ts:createJobsReconcilerFunction
`10` Proposed	The handler also repairs stranded provisioning and reconciles websites; after successful jobs reconciliation, an exception from provisioning repair fails the invocation before the website pass runs.	policy	service	observed services-platform:modules/jobs/aws/lambda/RECONCILER.handler.ts:handler
`11`must prove Proposed	Transient candidate or execution-probe errors defer the job to another pass, while an unregistered kind produces an operational alert. proved by `ATS-PFM-OPS-002-11`, `TST-PFM-OPS-013`	policy	service	observed macroservice-libraries:packages/jobs/src/reconciler/job-reconciler.ts:reconcileStalledJobs
`12`must prove Proposed	When a completed execution has a recorded successor, reconciliation follows the successor before deciding settlement, with a bounded continuation search. proved by `ATS-PFM-OPS-002-12`, `TST-PFM-OPS-004`	policy	service	observed macroservice-libraries:packages/jobs/src/reconciler/job-reconciler.ts:handleCandidate

Scenarios

Id	Given	When	Then	Provenance
`01`	Provisioning committed but no execution started.	Reconciliation examines it after the inactivity threshold.	The job fails as unstarted, the workspace becomes recoverable and the notifying path is invoked.	inferred services-platform:modules/tenants/tests/integration/lifecycle/reconciled-job-failure.integration.spec.js#moves the tenant to provisioning_failed, which is what makes it retryable and deletable again
`02`	Execution lookup cannot find a running job’s workflow.	Three successive passes record the same absence.	Only the last pass fails the job; the earlier passes record misses.	inferred services-platform:modules/jobs/tests/integration/lifecycle/job-reconciliation/job-reconciliation.integration.spec.js#halts the job only on the third pass

Planning

Implementation

Resolve interrupted background work ​

Purpose ​

How It Works ​

Relationships ​