---
id: CAP-PFM-DJE-016
title: Load and reconcile a migration package across its owning applications
feature: FEAT-PFM-DJE-05
product_area: platform
module: data-job-engine
revision: 2
status: draft
product_intent: proposed
implementation_evidence: observed
supersedes: []
source_sha:
  services-platform: ad3f6a6582843008ed0f5a17b131878d15829af1
  services-crm: a8d1364f1dd44b723bb5f9acba12f983e9ec8d69
intent_sources:
  - 'services-platform:docs/milestones/PLAT-M19-data-job-engine-foundations.md#D1'
  - 'services-platform:docs/milestones/PLAT-M19-data-job-engine-foundations.md#D2'
  - 'services-platform:docs/milestones/PLAT-M19-data-job-engine-foundations.md#D5'
  - 'services-platform:docs/milestones/plans/PLAT-M19-plan.md#S22'
  - 'services-platform:docs/milestones/plans/PLAT-M22-plan.md#A2'
  - 'services-platform:docs/milestones/plans/PLAT-M22-plan.md#A4'
  - >-
    services-platform:docs/milestones/PLAT-M22-migration-proof-and-close-out.md#D3
purpose: >-
  Implementation staff coordinate an initial migration across the applications
  that own the records and assess whether the resulting load is ready for
  customer use.
outcomes:
  - Invalid package declarations are caught before dataset jobs begin.
  - Dependent datasets wait for the records they reference.
  - Staff can distinguish a finished load from a load ready for customer use.
concepts:
  - CON-PFM-DJE-005
actors:
  - ACT-PFM-037
surfaces:
  - api
preconditions: []
postconditions: []
permissions:
  - key: hq.data-runs.manage
    where_checked: api
    scope: global
  - key: platform.custom-fields.bulk
    where_checked: service
    scope: tenant
rules:
  - id: BR-CAP-PFM-DJE-016-01
    proved_by: [ATS-PFM-DJE-016-01]
    statement: >-
      Data Run routes require the staff import-management permission and remain
      available during workspace loading.
    kind: permission
    enforced_at:
      - api
    must_prove: true
    provenance: observed
    source: 'services-platform:modules/data-jobs/aws/api/data-runs.routes.ts:manage'
    origin: >-
      services-platform:docs/milestones/PLAT-M19-data-job-engine-foundations.md#D5
  - id: BR-CAP-PFM-DJE-016-02
    proved_by: [ATS-PFM-DJE-016-02, TST-PFM-DJE-176]
    statement: >-
      Registering an initial migration requires both implementation status and
      loading state; other workspace states produce a named refusal.
    kind: validation
    enforced_at:
      - service
    must_prove: true
    provenance: test
    source: >-
      services-platform:modules/data-jobs/tests/integration/services/runs/register-run.integration.spec.js#refuses
      an initial import into a tenant that is not loading
    origin: >-
      services-platform:docs/milestones/PLAT-M19-data-job-engine-foundations.md#D5
  - id: BR-CAP-PFM-DJE-016-03
    proved_by: [ATS-PFM-DJE-011-11, ATS-PFM-DJE-016-03, ATS-PFM-DJE-016-14, TST-PFM-DJE-176]
    statement: >-
      Before saving a run, registration checks completed uploads against
      declared dataset sizes and available digests, requires an upload for every
      known dataset, and checks the destination workspace and any configured
      environment.
    kind: validation
    enforced_at:
      - service
    must_prove: true
    provenance: observed
    source: >-
      services-platform:modules/data-jobs/services/runs/register-run.ts:registerRun
    origin: 'services-platform:docs/milestones/plans/PLAT-M22-plan.md#A4'
  - id: BR-CAP-PFM-DJE-016-04
    proved_by: [ATS-PFM-DJE-012-17, ATS-PFM-DJE-016-04, ATS-PFM-DJE-016-15, TST-PFM-DJE-176, TST-PFM-DJE-189]
    statement: >-
      A package contains at most one hundred datasets, arranged into dependency
      waves that allow independent datasets together; missing dependencies and
      cycles prevent registration.
    kind: calculation
    enforced_at:
      - service
    must_prove: true
    provenance: test
    source: >-
      services-platform:modules/data-jobs/tests/integration/services/runs/register-run.integration.spec.js#refuses
      a package whose datasets cannot be ordered, before any row of it is
      written
    origin: 'services-platform:docs/milestones/plans/PLAT-M19-plan.md#S22'
  - id: BR-CAP-PFM-DJE-016-05
    proved_by: [ATS-PFM-DJE-016-05, TST-PFM-DJE-167, TST-PFM-DJE-176]
    statement: >-
      Registration rejects a package when it finds an unfinished run for that
      package in the same workspace, and allows registration again after that
      run reaches a final state.
    kind: validation
    enforced_at:
      - service
    must_prove: true
    provenance: test
    source: >-
      services-platform:modules/data-jobs/tests/integration/services/runs/register-run.integration.spec.js#refuses
      a second registration of a package whose run has not finished, naming the
      run that holds it
    origin: none
  - id: BR-CAP-PFM-DJE-016-06
    proved_by: [ATS-PFM-DJE-012-17, ATS-PFM-DJE-016-06, ATS-PFM-DJE-016-15, TST-PFM-DJE-167, TST-PFM-DJE-177]
    statement: >-
      Starting, resuming or reading a running run drives progress by copying job
      results, stopping at the first failed dataset, skipping pending work on
      failure and submitting datasets whose dependencies have finished.
    kind: lifecycle
    enforced_at:
      - service
    must_prove: true
    provenance: test
    source: >-
      services-platform:modules/data-jobs/tests/integration/lifecycle/data-run-guarantees.integration.spec.js#leaves
      the run where it stood while nobody reads it, though a dataset finished in
      its service
    origin: 'services-platform:docs/milestones/plans/PLAT-M22-plan.md#A2'
  - id: BR-CAP-PFM-DJE-016-07
    proved_by: [ATS-PFM-DJE-012-17, ATS-PFM-DJE-016-07, ATS-PFM-DJE-016-15, ATS-PFM-DJE-016-18, ATS-PFM-DJE-018-10, TST-PFM-DJE-177]
    statement: >-
      Each dataset becomes a job in its owning application under the requesting
      staff member’s authority; Platform datasets run locally with the same
      access checks, and an access refusal preserves the run state.
    kind: permission
    enforced_at:
      - service
    must_prove: true
    provenance: test
    source: >-
      services-platform:modules/data-jobs/tests/integration/services/runs/run-driver.integration.spec.js#forwards
      the staff caller’s own token with every call, and invents no identity of
      its own
    origin: 'services-platform:docs/milestones/plans/PLAT-M19-plan.md#S22'
  - id: BR-CAP-PFM-DJE-016-08
    proved_by: [ATS-PFM-DJE-012-17, ATS-PFM-DJE-016-08, ATS-PFM-DJE-016-15, TST-PFM-DJE-177]
    statement: >-
      A dataset ending with blocked records receives one further submission
      after its dependencies finish, then remains blocked if unresolved while
      dependent datasets may continue.
    kind: policy
    enforced_at:
      - service
    must_prove: true
    provenance: test
    source: >-
      services-platform:modules/data-jobs/tests/integration/services/runs/run-driver.integration.spec.js#tries
      a blocked dataset a second time, and only a second time
    origin: 'services-platform:docs/milestones/plans/PLAT-M19-plan.md#S22'
  - id: BR-CAP-PFM-DJE-016-09
    proved_by: [ATS-PFM-DJE-016-09, ATS-PFM-DJE-016-17, ATS-PFM-DJE-017-09, TST-PFM-DJE-177]
    statement: >-
      Pause prevents further dataset submissions, Resume requires a paused run
      and Cancel attempts to stop running jobs and skips pending work without
      undoing committed records; final run states permit no further transition.
    kind: lifecycle
    enforced_at:
      - service
    must_prove: true
    provenance: test
    source: >-
      services-platform:modules/data-jobs/tests/integration/services/runs/run-driver.integration.spec.js#cancels
      the jobs a cancelled run had already submitted, in their own services
    origin: none
  - id: BR-CAP-PFM-DJE-016-10
    proved_by: [ATS-PFM-DJE-015-07, ATS-PFM-DJE-016-10, ATS-PFM-DJE-016-16, ATS-PFM-DJE-016-17, ATS-PFM-DJE-017-09, TST-PFM-DJE-192]
    statement: >-
      The run copies each job’s counts and adds them into totals, while its
      separate readiness verdict lists unfinished or validation-only work,
      failed or cancelled runs, failed datasets, unresolved rejections, blocked
      records and shortfalls against declared counts.
    kind: calculation
    enforced_at:
      - service
    must_prove: true
    provenance: test
    source: >-
      services-platform:modules/data-jobs/tests/unit/run-summary.spec.ts#is not
      ready when a dataset’s counts do not add up to what the producer claimed
    origin: 'services-platform:docs/milestones/plans/PLAT-M22-plan.md#A3'
  - id: BR-CAP-PFM-DJE-016-11
    proved_by: [ATS-PFM-DJE-016-11, TST-PFM-DJE-177]
    statement: >-
      A permanent failure to read a running job fails its dataset and then the
      run, whereas a submission exception reaches the caller and can leave the
      run unfinished.
    kind: policy
    enforced_at:
      - service
    must_prove: true
    provenance: observed
    source: >-
      services-platform:modules/data-jobs/services/runs/run-driver.ts:pollStep;
      services-platform:modules/data-jobs/services/runs/run-driver.ts:submitStep
    origin: none
  - id: BR-CAP-PFM-DJE-016-12
    proved_by: [ATS-PFM-DJE-016-18, ATS-PFM-DJE-018-10]
    statement: Data Runs use configured application connections for Platform and CRM.
    kind: policy
    enforced_at:
      - service
    must_prove: false
    provenance: observed
    source: >-
      services-platform:modules/data-jobs/services/runs/data-run-services.ts:DATA_RUN_SERVICES
    origin: none
  - id: BR-CAP-PFM-DJE-016-13
    proved_by: [ATS-PFM-DJE-016-12]
    statement: >-
      A downstream access-denied exception interrupts cancellation, but other
      cancellation exceptions are logged and ignored, allowing the run to become
      cancelled while a child job continues.
    kind: lifecycle
    enforced_at:
      - service
    must_prove: true
    provenance: observed
    source: 'services-platform:modules/data-jobs/services/runs/run-driver.ts:cancelStep'
    origin: none
  - id: BR-CAP-PFM-DJE-016-14
    proved_by: [ATS-PFM-DJE-016-13]
    statement: >-
      A transient job-read failure preserves the step during the thirty-minute
      grace period measured from its last successful progress write; a failed
      read beyond that period fails the step.
    kind: lifecycle
    enforced_at:
      - service
    must_prove: true
    provenance: observed
    source: >-
      services-platform:modules/data-jobs/services/runs/run-driver.ts:pollStep;
      services-platform:modules/data-jobs/services/runs/data-job-port.ts:DATA_JOB_UNREADABLE_GRACE_MS
    origin: none
scenarios:
  - id: SC-CAP-PFM-DJE-016-01
    given: >-
      A test package contains organizations, contacts and a further dataset with
      declared dependencies.
    when: Implementation staff start the run and continue polling its progress.
    then: >-
      The datasets finish in dependency order and the fully accounted test load
      has no readiness blockers.
    provenance: test
    source: >-
      services-platform:modules/data-jobs/tests/integration/lifecycle/data-run-guarantees.integration.spec.js#accounts
      for every record the package claimed, and finds the tenant ready
edge_cases: []
constraints: []
depends_on: []
related:
  - CAP-PFM-DJE-011
  - CAP-PFM-DJE-012
  - CAP-PFM-DJE-015
  - CAP-PFM-DJE-017
journeys:
  - JNY-PFM-DJE-004
open_questions:
  - OQ-PFM-DJE-001
evidence:
  - >-
    services-platform at ad3f6a65: docs/data-jobs/data-runs.md;
    modules/data-jobs/aws/api/data-runs.routes.ts;
    services/runs/register-run.ts, run-driver.ts, run-summary.ts,
    package-manifest.ts, data-run-ports.ts, data-job-port.ts,
    data-run-services.ts; services/persistence/data-runs.ts;
    db/data-runs.table.ts, data-run-steps.table.ts; tests
    register-run.integration.spec.js, run-driver.integration.spec.js,
    data-run-ports.integration.spec.js, data-run-guarantees.integration.spec.js,
    data-runs.e2e.spec.js, unit/run-summary.spec.ts,
    unit/package-manifest.spec.ts; docs/milestones/PLAT-M22 record and plan.
  - >-
    services-crm at a8d1364f: docs/data-jobs.md (The routes: one URL pattern for
    a Data Run).
controls: [TC-AVL-004, TC-AZ-001, TC-AZ-002, TC-AZ-003, TC-AZ-004, TC-AZ-005, TC-CHG-001, TC-CHG-002, TC-CHG-003, TC-CHG-004, TC-CHG-005, TC-CHG-006, TC-CHG-007, TC-DAT-001, TC-DAT-003, TC-DAT-006, TC-IAM-001, TC-IAM-002, TC-IAM-003, TC-IAM-004, TC-IAM-005, TC-LOG-001, TC-LOG-004, TC-LOG-005, TC-MON-001, TC-PI-001, TC-PI-002, TC-PI-003, TC-PI-004, TC-PI-005, TC-PRV-001, TC-PRV-002, TC-TEN-001, TC-TEN-002, TC-TEN-003, TC-VEN-001, TC-VEN-002, TC-VEN-003]
exercised_by: [FAS-PFM-DJE-05-01, FAS-PFM-DJE-05-02, FAS-PFM-DJE-05-03, FAS-PFM-DJE-05-04, FAS-PFM-DJE-05-05]
suites: [TST-PFM-DJE-167, TST-PFM-DJE-176, TST-PFM-DJE-177, TST-PFM-DJE-189, TST-PFM-DJE-192]
---

# Load and reconcile a migration package across its owning applications

## Purpose

Implementation staff coordinate a migration without taking ownership of another application’s records. The run gives them one place to follow the package and assess its results.

## How It Works

Staff upload a manifest and dataset files, then register the package in the workspace being prepared for use. Registration checks the declarations against completed uploads and establishes a dependency order before any dataset job starts. A measured digest is preferred, but a declared digest can also satisfy the current check. Staff start the run using their own authority. Platform loads its datasets locally; other configured applications receive requests carrying the staff member’s token. Reading the running run refreshes dataset progress and submits newly eligible work, so polling also drives the load. Staff can pause further submissions or cancel the run, although a failed downstream cancellation can leave a child job active. Records already committed remain in the workspace. The summary adds the reported counts and presents readiness separately from completion. Submission errors, incomplete progress responses and excess counts remain limits that staff must consider when reviewing a load. The product should give staff a dedicated screen for this work and make unresolved cancellation visible.

## Relationships

- Uploads provide the manifest and datasets used during registration.
- Record imports perform each dataset’s changes in the owning application.
- Reading records by producer key supports reconciliation against the extract.
- Workspace reset provides a clean starting point after a failed load; activation controls customer access.