--- id: CAP-PFM-DJE-004 title: Interrupt a job and recover its unfinished work feature: FEAT-PFM-DJE-01 product_area: platform module: data-job-engine revision: 1 status: draft product_intent: confirmed implementation_evidence: observed supersedes: - CAP-DJE-ENG-004 - RUL-DJE-ENG-015 - RUL-DJE-ENG-016 source_sha: macroservice-libraries: 54536a441935c6e09fdb7f96f0bec71c52ba4e03 services-crm: a8d1364f1dd44b723bb5f9acba12f983e9ec8d69 react-console: baf6dae382d6f41dce976c7f848667ccc37e076f intent_sources: - 'services-platform:docs/milestones/plans/PLAT-M19-plan.md#S06' - 'services-platform:docs/milestones/plans/PLAT-M19-plan.md#S15' - 'services-crm:docs/milestones/CRM-M7-data-operations.md#D-39' purpose: Administrators control future job processing while retaining the changes that have already committed. outcomes: - Interruption takes effect between batches. - Cancellation distinguishes committed records from work never attempted. - Blocked dependencies can be retried after they arrive. concepts: - CON-PFM-DJE-001 actors: - ACT-PFM-035 - ACT-PFM-036 surfaces: - console - api preconditions: [] postconditions: [] permissions: - key: crm.contacts.bulk where_checked: api scope: tenant - key: crm.organizations.bulk where_checked: api scope: tenant - key: data.jobs.control where_checked: console scope: tenant rules: - id: BR-CAP-PFM-DJE-004-01 proved_by: [ATS-PFM-DJE-004-01, TST-PFM-DJE-014] statement: >- Job state changes follow the declared transitions; final states have no outgoing transition except the supported retry of completed work containing blocked items. kind: lifecycle enforced_at: - service must_prove: true provenance: test source: >- macroservice-libraries:packages/data-jobs/src/data-job-states.spec.ts#leaves every final state with nowhere to go, but for the retry of blocked items the plan allows origin: none - id: BR-CAP-PFM-DJE-004-02 proved_by: [ATS-PFM-DJE-003-12, ATS-PFM-DJE-004-02, ATS-PFM-DJE-004-08, TST-PFM-DJE-101, TST-PFM-DJE-117] statement: >- Pause and cancel prevent later batches from starting while in-flight work settles; committed changes remain, and cancellation reports unattempted records. kind: lifecycle enforced_at: - service must_prove: true provenance: test source: >- services-crm:modules/data-jobs/tests/integration/services/jobs/contact-data-jobs.integration.spec.js#ends the job cancelled, not failed, once the batch it waited for has finished origin: 'services-platform:docs/milestones/plans/PLAT-M19-plan.md#S06' - id: BR-CAP-PFM-DJE-004-03 proved_by: [ATS-PFM-DJE-004-03, TST-PFM-DJE-024] statement: 'Resume queues unfinished batches again, including a file job paused before source reading began.' kind: lifecycle enforced_at: - service must_prove: true provenance: test source: >- macroservice-libraries:packages/data-jobs/src/tests/integration/bulk-data-jobs.integration.spec.ts#holds a job paused before its source was read, and runs it to the end once resumed origin: 'services-platform:docs/milestones/plans/PLAT-M19-plan.md#S15' - id: BR-CAP-PFM-DJE-004-04 proved_by: [ATS-PFM-DJE-004-04, ATS-PFM-DJE-004-10, ATS-PFM-DJE-012-12, TST-PFM-DJE-029, TST-PFM-DJE-120] statement: >- Retry recovers transiently failed batches or a completed job’s blocked items; rejected and conflicted items require a correcting job. kind: policy enforced_at: - service must_prove: true provenance: test source: >- macroservice-libraries:packages/data-jobs/src/tests/integration/keyed-data-jobs.integration.spec.ts#refuses to retry a job whose refusals are not blocked items, because correcting records is a new job origin: 'services-crm:docs/milestones/CRM-M7-data-operations.md#D-39' - id: BR-CAP-PFM-DJE-004-05 proved_by: [ATS-PFM-DJE-004-05, ATS-PFM-DJE-004-09, ATS-PFM-DJE-008-07, TST-PFM-DJE-012, TST-PFM-DJE-032] statement: >- Once a rollback has been applied, the original job refuses retry so blocked items cannot be written after restoration. kind: policy enforced_at: - service must_prove: true provenance: test source: >- macroservice-libraries:packages/data-jobs/src/tests/integration/rollback-and-retry.integration.spec.ts#refuses to retry a job that has been rolled back, so its blocked rows are not written after the undo origin: 'services-crm:docs/milestones/CRM-M7-data-operations.md#D-39' - id: BR-CAP-PFM-DJE-004-06 proved_by: [ATS-PFM-DJE-004-06] statement: >- Job controls require the route’s permissions; the console hides controls from unauthorized users and the backend refuses their requests. kind: permission enforced_at: - api - console must_prove: true provenance: documented source: 'services-crm:docs/data-jobs.md#What a job may be asked now: `actions` (D-39)' origin: 'services-crm:docs/milestones/CRM-M7-data-operations.md#D-39' - id: BR-CAP-PFM-DJE-004-07 proved_by: [ATS-PFM-DJE-004-07, TST-PFM-DJE-024, TST-PFM-DJE-113] statement: >- A failed bulk workflow marks its job failed with the cause and abandons unstarted batches as unattempted. kind: lifecycle enforced_at: - service must_prove: true provenance: test source: >- services-crm:modules/data-jobs/tests/integration/services/jobs/bulk-data-job-failure.integration.spec.js#fails the data job, so its status read answers failed and a Data Run polling it stops origin: none - id: BR-CAP-PFM-DJE-004-08 proved_by: [] statement: Cancelling through the console requires an explicit confirmation. kind: policy enforced_at: - console must_prove: false provenance: observed source: 'react-console:lib/api/data-jobs/rules.ts:DATA_JOB_ACTIONS_NEEDING_CONFIRMATION' origin: none scenarios: - id: SC-CAP-PFM-DJE-004-01 given: A 500-contact update is processing. when: 'The administrator pauses it, resumes it and then confirms cancellation.' then: >- The job settles between batches as cancelled, accounting for all 500 contacts as applied or unattempted. provenance: test source: >- react-console:acceptance/data-jobs/playwright/journeys/foundations/DJE-ACC-0006.spec.ts#DJE-ACC-0006 - id: SC-CAP-PFM-DJE-004-02 given: An import finished with branches blocked because their head office was absent. when: The administrator loads that head office and requests retry. then: The saved branch input can now attach to the head office without another upload. provenance: test source: 'react-console:acceptance/crm/playwright/journeys/data-operations/CRM-ACC-0647.spec.ts#CRM-ACC-0647' edge_cases: [] constraints: [] depends_on: [] related: - CAP-PFM-DJE-003 - CAP-PFM-DJE-008 - CAP-PFM-DJE-012 journeys: - JNY-PFM-DJE-005 open_questions: [] evidence: - >- Reviewed evidence: macroservice-libraries (54536a44): src/data-job-states.ts, job-control.ts, data-job-actions.ts, fail-bulk-data-job.ts; tests data-job-states.spec.ts, data-job-actions.spec.ts, bulk-data-jobs.integration.spec.ts, keyed-data-jobs.integration.spec.ts, rollback-and-retry.integration.spec.ts. - >- Reviewed evidence: services-crm (a8d1364f): docs/data-jobs.md (What a job may be asked now); tests contact-data-jobs.integration.spec.js, bulk-data-job-failure.integration.spec.js. - >- Reviewed evidence: react-console (baf6dae3): lib/api/data-jobs/rules.ts; components/data-management/data-jobs/job-actions.tsx; acceptance/data-jobs/playwright/journeys/foundations/DJE-ACC-0006.spec.ts. controls: [TC-AVL-001, TC-AVL-004, TC-AZ-001, TC-AZ-002, TC-AZ-003, TC-AZ-004, TC-DAT-001, TC-DAT-003, TC-IAM-001, TC-IAM-006, TC-LOG-004, TC-LOG-005, TC-MON-001, TC-PI-001, TC-PI-002, TC-PI-003, TC-PI-004, TC-PI-005, TC-PRV-001, TC-PRV-002, TC-TEN-001, TC-TEN-002, TC-TEN-003, TC-VEN-001, TC-VEN-002, TC-VEN-003] exercised_by: [FAS-PFM-DJE-01-02, FAS-PFM-DJE-01-03] suites: [TST-PFM-DJE-012, TST-PFM-DJE-014, TST-PFM-DJE-024, TST-PFM-DJE-029, TST-PFM-DJE-032, TST-PFM-DJE-101, TST-PFM-DJE-113, TST-PFM-DJE-117, TST-PFM-DJE-120] --- # Interrupt a job and recover its unfinished work ## Purpose Administrators control future job processing while retaining the changes that have already committed. ## How It Works An administrator opens job detail to see the controls appropriate to its current state. Pause holds future processing while in-flight batches settle. Resume makes unfinished work available again. Cancel asks for confirmation and preserves earlier commits. The cancelled job reports both completed work and work never attempted. Retry addresses transient failures or dependencies that were missing when the job ran. Refusals requiring changed input use another submission instead. A completed rollback prevents later retry of the original job. If the bulk workflow fails, the failure handler settles its job rather than leaving unstarted work queued. ## Relationships - Processing supplies the work that controls interrupt. - Rollback prevents further retries of restored jobs. - Keyed imports use retry to recover missing dependencies.