Rhythm 2.0 Specifications

Appearance

CAP-PFM-OPS-001Complete background work with a durable record

Markdown

CAP-PFM-OPS-001

Complete background work with a durable record

Markdown
  • DocumentDraft
  • Product intentConfirmed
  • ImplementationObserved

Planning

Feature
FEAT-PFM-OPS-01
Exercised by
FAS-PFM-OPS-01-01, FAS-PFM-OPS-01-02, FAS-PFM-OPS-01-03, FAS-PFM-OPS-01-05
Key concepts
CON-PFM-OPS-001
Actors
ACT-PFM-006, ACT-PFM-004
Journeys
JNY-PFM-OPS-002
Related
CAP-PFM-OPS-002, CAP-PFM-OPS-004, CAP-PFM-SEC-003, CAP-PFM-SEC-004
Proved by
ATS-PFM-OPS-001

Implementation

Executable suites
TST-PFM-OPS-004, TST-PFM-OPS-005, TST-PFM-OPS-006, TST-PFM-OPS-007, TST-PFM-OPS-019, TST-PFM-OPS-077

KindCapabilityProduct areaPlatformModuleOperational RecordsRevision1

Read atservices-platform ad3f6a65macroservice-libraries 54536a44

Complete background work with a durable record ​

Purpose ​

Give long-running work a persistent account that operators can inspect and use for recovery.

How It Works ​

The owning module registers the kind of work, its visible steps and its recovery behavior. A submitting operation creates the job beside the records that require it. Execution starts after that transaction commits. The shared engine records step reports and updates the job summary together. A monitor can then show attempts and outcomes without reconstructing them from logs. When execution cannot begin, the committed job remains available for reconciliation. Permanent refusals provide an operator-safe explanation, while unexpected technical details stay outside the public job record. Recovery belongs to the owning workflow and the shared reconciliation contract.

Submission inside a caller transaction can require that caller to retry a conflicting transaction. Progress combines settled steps with declared weights, and execution continuation keeps a traceable successor rather than losing the original handle.

Relationships ​

The job record supports reconciliation, notices and both monitors; owning modules retain responsibility for the work itself.

The Formal Contract

The narrative above explains; the sections below are authoritative — the rules, scenarios and tests this document binds.

Outcomes
Repeated intentions resolve to the existing job rather than another submission.Operators can inspect completed work, interruptions and failure explanations.Failure handling includes the owning record’s declared recovery consequence.
Surfaces
job
Preconditions
The owning module registers the job kind, step catalog, execution workflow, recovery declarations and failure consequence.
Postconditions
Submission preserves the steps and event history and the job eventually records one final status.
Constraints
Cloud workflows own sequencing, retry and timeouts; the engine maintains the observable record.
Intent sources
services-platform:docs/hq/rhythm-hq-plan.md#K8 (plan approved 2026-10-11)services-platform:docs/hq/rhythm-hq-plan.md#L. Verification plan (durable job persistence, retry lineage and idempotent submission)

Controls

41 controls apply, from the corpus control catalogue.

TC-AVL-001 TC-AVL-004 TC-AVL-005 TC-AZ-001 TC-AZ-002 TC-AZ-003 TC-AZ-004 TC-CRY-001 TC-CRY-002 TC-CRY-003 TC-CRY-004 TC-DAT-001 TC-DAT-002 TC-DAT-003 TC-DAT-004 TC-IAM-001 TC-IAM-002 TC-IAM-004 TC-IAM-005 TC-IAM-006 TC-LOG-002 TC-LOG-003 TC-LOG-004 TC-LOG-005 TC-MON-001 TC-MON-002 TC-PI-001 TC-PI-002 TC-PI-003 TC-PI-004 TC-PI-005 TC-SEC-001 TC-SEC-002 TC-SEC-003 TC-SEC-004 TC-TEN-001 TC-TEN-002 TC-TEN-003 TC-VEN-001 TC-VEN-002 TC-VEN-003

Business Rules

12 rules, 10 to be proved before approval. A rule's product intent is derived from its origin: 12 confirmed by a product decision, the rest inheriting this capability's confirmed.

IdRuleKindEnforced atProvenance
01must prove
Confirmed
Repeated intentions identify one job; a concurrent uniqueness conflict resolves to the existing job outside a caller transaction but aborts an open caller transaction for the caller to retry.
✓ A repeated workspace-provisioning submission returns the existing job.✗ A unique submission key prevents two job records for the same intention.
origin: services-platform:docs/hq/rhythm-hq-plan.md#L. Verification plan (approved K8, 2026-10-11)
proved by ATS-PFM-OPS-001-01, TST-PFM-OPS-005
invariantservice, databaseobserved
macroservice-libraries:packages/jobs/src/engine/job-engine.ts:createJob
02must prove
Confirmed
Creating a job records its queued steps and submission event; starting its execution inside the submitting transaction is refused.
proved by ATS-PFM-OPS-001-02
lifecycleservicetest
macroservice-libraries:packages/jobs/src/engine/job-engine.spec.ts#refuses submitJob inside a transaction: the execution starts after the commit
03must prove
Confirmed
When execution cannot start after submission commits, the API-facing start helper leaves the job pending for reconciliation and preserves the successful submission response.
✓ A repeated workspace-provisioning submission returns the existing job.✗ A unique submission key prevents two job records for the same intention.
proved by ATS-PFM-OPS-001-03, ATS-PFM-OPS-001-09
policyserviceobserved
macroservice-libraries:packages/jobs/src/engine/job-engine.ts:startJobOrLeaveToReconciler
04must prove
Confirmed
A final job or step cannot transition to another status; late reports for settled jobs are discarded and logged, and another settlement is refused.
proved by ATS-PFM-OPS-001-04, ATS-PFM-OPS-001-10, ATS-PFM-OPS-001-13, ATS-PFM-OPS-001-14, ATS-PFM-OPS-001-16, ATS-PFM-OPS-003-13, TST-PFM-OPS-005, TST-PFM-OPS-006, TST-PFM-OPS-007
invariantservicetest
macroservice-libraries:packages/jobs/src/engine/job-engine.spec.ts#refuses settling an already terminal job
05must prove
Confirmed
Job progress cannot decrease, and settlement timestamps are present only on final jobs.
proved by ATS-PFM-OPS-001-05, TST-PFM-OPS-005
invariantservice, databasetest
macroservice-libraries:packages/jobs/src/engine/job-engine.spec.ts#refuses progress that would move backwards
06must prove
Confirmed
A permanent task refusal ends the job with its safe explanation; transient failures are retried without copying their details into the job or execution history.
proved by ATS-PFM-OPS-001-06, TST-PFM-OPS-019
policyservicetest
macroservice-libraries:packages/jobs/src/workflow/job-workflow.spec.ts#lets no detail of the transient error reach the job or the execution state
07
Confirmed
Asynchronous job inputs contain identifying references and request context, with secrets and customer content retained outside the execution payload.
policyservicedocumented
services-platform:docs/jobs.md#What may cross the asynchronous boundary
08
Confirmed
The monitor obtains the job’s step catalog from its published history, preserving the definition used for that execution rather than substituting the current declaration.
policyservicetest
services-platform:modules/jobs/tests/integration/services/job-snapshot/job-snapshot.integration.spec.js#takes that catalog from the published event rather than the declaration, which the event itself carries
09must prove
Confirmed
Each registered job kind must declare its business failure consequence or explicitly declare that failure changes nothing outside the job.
proved by ATS-PFM-OPS-001-07, TST-PFM-OPS-077
policyservicetest
services-platform:modules/jobs/tests/unit/job-failure-consequences.spec.js#declares a failure consequence, or states there is none, for every registered kind
10must prove
Confirmed
Workspace deletion removes its job records, steps, events and notices while preserving the deletion’s own scheduled and executing jobs.
proved by ATS-PFM-OPS-001-08, ATS-PFM-OPS-001-11, ATS-PFM-OPS-001-12, ATS-PFM-OPS-001-17
lifecycleservice, databaseobserved
services-platform:modules/jobs/services/persistence/tenant-purges.ts:JOB_PURGE_RESOURCES
11must prove
Confirmed
Default progress rounds the fraction of settled steps to a percentage capped at one hundred; declared progress weights instead use the highest settled-step weight.
no proof recorded yet
calculationserviceobserved
macroservice-libraries:packages/jobs/src/engine/job-engine.ts:deriveDeclaredProgress
12must prove
Confirmed
Execution continuation records the successor handle and continuation history together, accepts replay of the same move and refuses a source execution that does not match the job.
proved by ATS-PFM-OPS-001-15, ATS-PFM-OPS-002-12, TST-PFM-OPS-004
invariantserviceobserved
macroservice-libraries:packages/jobs/src/engine/job-engine.ts:recordJobContinuation

Scenarios

IdGivenWhenThenProvenance
01A provisioning request already has a recorded submission identity.The caller repeats the request using that identity.The existing job is returned without another job record.inferred
macroservice-libraries:packages/jobs/src/engine/job-engine.spec.ts#answers the job the identity already names when the submission is repeated
02An unexpected database fault interrupts a running task.The task exhausts its workflow retries.The recorded failure remains generic and the technical error is confined to logs.inferred
macroservice-libraries:packages/jobs/src/workflow/job-workflow.spec.ts#lets no detail of the transient error reach the job or the execution state

Edge Cases

IdGivenWhenThenProvenance
03A job has already completed successfully.A delayed report marks one of its steps running.The engine ignores and logs the report without adding history or raising a regression alert.inferred
macroservice-libraries:packages/jobs/src/engine/job-engine.spec.ts#raises no terminal-regression alert for a late running report against a succeeded job

The Markdown files under specs/ in rhythmsoftware/rhythm-v2-docs are the canonical source of truth. This reader is a generated, read-only projection of them.

Rhythm Software