--- id: CAP-PFM-OPS-002 title: Resolve interrupted background work feature: FEAT-PFM-OPS-01 product_area: platform module: operations revision: 1 status: draft product_intent: proposed implementation_evidence: observed supersedes: [] source_sha: services-platform: ad3f6a6582843008ed0f5a17b131878d15829af1 macroservice-libraries: 54536a441935c6e09fdb7f96f0bec71c52ba4e03 intent_sources: [] purpose: Close the gap between a job’s stored status and the execution that actually ran. outcomes: - Interrupted work reaches an explainable resolution. - Reconciled failure applies the same business consequence as ordinary failure. concepts: - CON-PFM-OPS-001 actors: - ACT-PFM-006 surfaces: - job preconditions: - The reconciler registers the kind and can discover its family through the service’s job views. postconditions: - 'The pass repairs, settles or preserves each examined job according to the execution evidence.' permissions: [] rules: - id: BR-CAP-PFM-OPS-002-01 proved_by: [ATS-PFM-OPS-002-01, TST-PFM-OPS-069] statement: >- The five-minute reconciliation pass considers jobs silent for fifteen minutes and converts timed-out, failed or unrequested aborted executions to failed jobs, preserving earned progress and failing any running step. kind: lifecycle enforced_at: - service must_prove: true provenance: test source: >- services-platform:modules/jobs/tests/integration/lifecycle/job-reconciliation/job-reconciliation.integration.spec.js#settles the job failed, with the reconciled summary and the progress its steps earned origin: none - id: BR-CAP-PFM-OPS-002-02 proved_by: [ATS-PFM-OPS-002-02, TST-PFM-OPS-069] statement: A reported successful execution settles failed if a recorded step remains running or retrying. kind: calculation enforced_at: - service must_prove: true provenance: test source: >- services-platform:modules/jobs/tests/integration/lifecycle/job-reconciliation/job-reconciliation.integration.spec.js#corrects the job to failed, never succeeded, keeping the progress its steps earned origin: none - id: BR-CAP-PFM-OPS-002-03 proved_by: [ATS-PFM-OPS-001-09, ATS-PFM-OPS-002-03, ATS-PFM-OPS-002-09, ATS-PFM-OPS-003-08, TST-PFM-OPS-069] statement: 'A stalled job with no execution handle ends failed as never started, leaving its steps queued.' kind: lifecycle enforced_at: - service must_prove: true provenance: test source: >- services-platform:modules/jobs/tests/integration/lifecycle/job-reconciliation/job-reconciliation.integration.spec.js#settles it failed, saying the job never started origin: none - id: BR-CAP-PFM-OPS-002-04 proved_by: [ATS-PFM-OPS-002-04, TST-PFM-OPS-069] statement: >- Three consecutive unsuccessful execution lookups are required before reconciliation fails a missing execution, with each miss added to history. kind: policy enforced_at: - service must_prove: true provenance: test source: >- services-platform:modules/jobs/tests/integration/lifecycle/job-reconciliation/job-reconciliation.integration.spec.js#halts the job only on the third pass origin: none - id: BR-CAP-PFM-OPS-002-05 proved_by: [ATS-PFM-OPS-002-05, TST-PFM-OPS-069] statement: >- Reconciliation leaves recently active jobs and genuinely running executions unchanged, including their history. kind: policy enforced_at: - service must_prove: true provenance: test source: >- services-platform:modules/jobs/tests/integration/lifecycle/job-reconciliation/job-reconciliation.integration.spec.js#leaves the root exactly as it was, version and all origin: none - id: BR-CAP-PFM-OPS-002-06 proved_by: [ATS-PFM-OPS-002-06, TST-PFM-OPS-069] statement: >- A job settled within the preceding day whose execution still runs raises an operational alert without reopening its record. kind: policy enforced_at: - service must_prove: true provenance: test source: >- services-platform:modules/jobs/tests/integration/lifecycle/job-reconciliation/job-reconciliation.integration.spec.js#raises one alert and changes nothing origin: none - id: BR-CAP-PFM-OPS-002-07 proved_by: [ATS-PFM-OPS-002-07, ATS-PFM-OPS-002-10] statement: >- Reconciliation applies the registered failure consequence before settlement, and that consequence must tolerate repeat application. kind: policy enforced_at: - service must_prove: true provenance: test source: >- macroservice-libraries:packages/jobs/src/reconciler/job-reconciler.spec.ts#applies it before the settlement, so a consequence that fails leaves the job for the next pass origin: none examples: positive: An unstarted provisioning job moves its workspace into recoverable failure. negative: A dead execution cannot leave provisioning dependent on a failure task that never ran. - id: BR-CAP-PFM-OPS-002-08 proved_by: [ATS-PFM-OPS-002-08, TST-PFM-OPS-069] statement: A mismatch between the job summary and step records causes summary repair rather than settlement. kind: calculation enforced_at: - service must_prove: true provenance: test source: >- services-platform:modules/jobs/tests/integration/lifecycle/job-reconciliation/job-reconciliation.integration.spec.js#repairs the summary instead of settling the job origin: none - id: BR-CAP-PFM-OPS-002-09 proved_by: [] statement: >- The reconciler reserves one concurrent invocation, has a silence alarm, and rejects malformed invocation requests before running a pass. kind: policy enforced_at: - service must_prove: true provenance: observed source: 'services-platform:modules/jobs/aws/logic/jobs-reconciler.lambda.ts:createJobsReconcilerFunction' origin: none - id: BR-CAP-PFM-OPS-002-10 proved_by: [] statement: >- The handler also repairs stranded provisioning and reconciles websites; after successful jobs reconciliation, an exception from provisioning repair fails the invocation before the website pass runs. kind: policy enforced_at: - service must_prove: false provenance: observed source: 'services-platform:modules/jobs/aws/lambda/RECONCILER.handler.ts:handler' origin: none - id: BR-CAP-PFM-OPS-002-11 proved_by: [ATS-PFM-OPS-002-11, TST-PFM-OPS-013] statement: >- Transient candidate or execution-probe errors defer the job to another pass, while an unregistered kind produces an operational alert. kind: policy enforced_at: - service must_prove: true provenance: observed source: 'macroservice-libraries:packages/jobs/src/reconciler/job-reconciler.ts:reconcileStalledJobs' origin: none - id: BR-CAP-PFM-OPS-002-12 proved_by: [ATS-PFM-OPS-002-12, TST-PFM-OPS-004] statement: >- When a completed execution has a recorded successor, reconciliation follows the successor before deciding settlement, with a bounded continuation search. kind: policy enforced_at: - service must_prove: true provenance: observed source: 'macroservice-libraries:packages/jobs/src/reconciler/job-reconciler.ts:handleCandidate' origin: none scenarios: - id: SC-CAP-PFM-OPS-002-01 given: Provisioning committed but no execution started. when: Reconciliation examines it after the inactivity threshold. then: 'The job fails as unstarted, the workspace becomes recoverable and the notifying path is invoked.' provenance: inferred source: >- services-platform:modules/tenants/tests/integration/lifecycle/reconciled-job-failure.integration.spec.js#moves the tenant to provisioning_failed, which is what makes it retryable and deletable again note: >- This illustrative example combines behavior established by the cited test and the capability’s implementation evidence; it is not an independently bound end-to-end proof. - id: SC-CAP-PFM-OPS-002-02 given: Execution lookup cannot find a running job’s workflow. when: Three successive passes record the same absence. then: Only the last pass fails the job; the earlier passes record misses. provenance: inferred source: >- services-platform:modules/jobs/tests/integration/lifecycle/job-reconciliation/job-reconciliation.integration.spec.js#halts the job only on the third pass note: >- This illustrative example combines behavior established by the cited test and the capability’s implementation evidence; it is not an independently bound end-to-end proof. edge_cases: [] constraints: - >- The reconciler reads its service’s job views, probes workflow executions and changes records through the engine. depends_on: - CAP-PFM-OPS-001 related: - CAP-PFM-OPS-003 - CAP-PFM-SEC-003 journeys: - JNY-PFM-OPS-002 open_questions: [] evidence: - >- services-platform at ad3f6a6: modules/jobs/aws/lambda/RECONCILER.handler.ts, aws/logic/jobs-reconciler.lambda.ts, db/cross-job-query.view.ts; job-reconciliation integration suite and its README; modules/tenants reconciled-job-failure and repair-stranded-provisioning suites; docs/jobs.md §2 step 4, §4. - >- macroservice-libraries at 54536a4: reconciler/job-reconciler.ts, reconciler/job-reconciler-handler.ts (thresholds 15 minutes, three misses, one-day recheck), aws/job-reconciliation.schedule.ts. - 'Schedules in scope: the five-minute reconciler rule and its silence alarm.' - >- Examination: proposed acceptance matrices retain candidate test references; incomplete proof is recorded by FND-PFM-OPS-028. controls: [TC-AVL-001, TC-AVL-004, TC-AZ-001, TC-AZ-002, TC-AZ-003, TC-AZ-004, TC-IAM-001, TC-IAM-006, TC-LOG-002, TC-LOG-003, TC-LOG-005, TC-MON-001, TC-MON-002, TC-PI-001, TC-PI-002, TC-PI-003, TC-PI-004, TC-PI-005, TC-TEN-001, TC-TEN-002, TC-TEN-003, TC-VEN-001, TC-VEN-002, TC-VEN-003] exercised_by: [FAS-PFM-OPS-01-02, FAS-PFM-OPS-01-05] suites: [TST-PFM-OPS-004, TST-PFM-OPS-013, TST-PFM-OPS-069] --- # Resolve interrupted background work ## Purpose Close the gap between a job’s stored status and the execution that actually ran. ## How It Works The scheduled process looks for old job records that have stopped receiving updates. It asks the workflow service whether the underlying execution is still active or has ended. The response determines whether the record needs repair, settlement or another lookup later. Business failure consequences run before the failed outcome is stored. That ordering lets the next pass try again if a consequence cannot finish. A settled job stays final even when its execution disagrees. An alert calls attention to that disagreement. Workspace repair and website reconciliation share the scheduled invocation. After a successful jobs pass, repair failure stops the invocation before website reconciliation runs. Transient lookup faults defer settlement, and recorded continuation can redirect the execution lookup. ## Relationships The job record supports reconciliation, notices and both monitors; owning modules retain responsibility for the work itself.