--- product_area: platform module: data-job-engine revision: 2 status: draft owned_by: MOD-PFM-DJE synonyms: - initial import - migration load supersedes: [] id: CON-PFM-DJE-005 name: Data Run feature: FEAT-PFM-DJE-05 product_intent: confirmed implementation_evidence: partially-observed source_sha: services-platform: ad3f6a6582843008ed0f5a17b131878d15829af1 services-crm: a8d1364f1dd44b723bb5f9acba12f983e9ec8d69 definition: >- A Data Run coordinates the initial migration of a package into a customer workspace, retaining dataset dependencies, application job progress and a separate assessment of readiness. not_to_be_confused_with: - An individual data job that processes one dataset. - 'Workspace provisioning, which establishes the workspace before migration.' facets: - name: Package and manifest meaning: >- The manifest describes the package destination, datasets, file sizes, digests, record counts and dependencies. used_for: Checking a package before its dataset jobs start. - name: Steps and waves meaning: >- Each dataset has a step, and dependency waves distinguish work that can begin together from work that must wait. used_for: Coordinating referenced records across datasets. - name: Run state meaning: >- Registered, running and paused describe unfinished work; succeeded, failed and cancelled are final states. used_for: Following progress and deciding whether work can continue. - name: Summary and readiness meaning: >- Dataset counts feed run totals, while readiness separately identifies the obstacles to a usable load. used_for: Reviewing the load before customer access begins. - name: Loading state meaning: >- Implementation status and loading state identify a workspace undergoing initial migration. used_for: Keeping ordinary customer work outside an unfinished migration. - name: Reset meaning: >- Reset removes the loaded domain records while preserving the agreed configuration and migration evidence. used_for: Repeating a migration from a clean workspace. - name: Activation meaning: >- A separately authorized operator changes the workspace from loading to active. used_for: Opening the workspace to customer staff. relationships: - to: CON-PFM-DJE-001 statement: >- A run submits dataset jobs under the requesting staff member’s authority, forwarding the token to remote applications. provenance: documented source: >- services-platform:docs/data-jobs/data-runs.md#5. What each step asks of the owning service - to: CON-PFM-DJE-003 statement: >- Completed uploads supply the datasets, checked against measured sizes and measured or declared digests. provenance: observed source: >- services-platform:modules/data-jobs/services/runs/register-run.ts:requireFileAsClaimed - to: CON-PFM-DJE-004 statement: >- Producer-key lookups allow staff to compare loaded records with the original extract. provenance: documented source: 'services-crm:docs/data-jobs.md#Reading a load back by source key' lifecycle: - state: registered meaning: 'The package is recorded and ordered, with no dataset job yet submitted.' how_it_changes: Staff start or cancel the run. - state: 'running, paused' meaning: >- The run submits eligible datasets or holds further submissions while paused. how_it_changes: 'Staff drive progress through Start, Resume and reads of the running run.' - state: 'succeeded, failed, cancelled' meaning: The run has ended and its retained results explain the load. how_it_changes: Further migration uses a newly registered run. acted_on_by: - CAP-PFM-DJE-016 - CAP-PFM-DJE-017 visible_to: - ACT-PFM-037 - ACT-PFM-038 evidence: - >- services-platform: inspected manifest parsing, registration, upload comparisons, driver and application ports, summaries, run tables and persistence; checked the cited migration milestone records. - >- services-crm: inspected migration documentation and producer-key reconciliation services. --- # Data Run ## What It Is A Data Run is the continuing account of an initial migration package as its datasets load into Rhythm. Implementation staff use it when an association’s information spans several kinds of record and more than one owning application. The run connects those separate jobs so staff can follow the package as a whole and assess what still needs attention. The package manifest provides the producer’s identity, destination and dataset declarations. Registration compares those declarations with completed uploads, establishes dependency waves and saves the run with its steps. It does not load the business records. The same package can have later runs after an earlier attempt ends, so the package identity and the identity of one execution serve different purposes. A run preserves each dataset’s owning application, job reference and reported counts. Its summary adds the dataset counts into package totals. A separate readiness assessment identifies obstacles to customer use; completion alone does not establish a successful reconciliation. ## What It Is Not A run does not replace the individual data jobs that validate and save records. It also differs from provisioning, which creates the workspace and its initial configuration. The implemented package purpose is initial migration into a workspace being prepared for use, rather than a recurring integration delivery into an active workspace. Readiness is an assessment of the recorded load, rather than an enforced activation gate. The current activation action does not consult that assessment. Nor does a cancelled run prove that every child job stopped: some downstream cancellation failures are logged without preventing the run from becoming cancelled. ## How It Is Used Implementation staff upload and register the package, then start the run with their own authority. They continue reading it while jobs progress. Those reads also advance coordination by recording completed work and submitting eligible datasets; an unwatched run does not advance in the background. Staff inspect reported counts and read records back using producer keys. If a mapping problem requires a clean load, they reset the workspace and register a corrected package. An operator with activation authority opens the workspace after reviewing the migration. A migration tool can perform the same API sequence. ## Its Facets **Package and manifest.** The manifest describes what should arrive and where it belongs. Upload comparisons check sizes and available digests, but a declared digest on a large file can remain unverified. **Steps and waves.** Steps retain the jobs for individual datasets. Waves express dependencies and identify datasets eligible to proceed together. A blocked dataset receives one further attempt. **Run state.** State distinguishes registered work, active coordination, a pause and the final outcomes. The retained state explains whether staff can continue the same run. **Summary and readiness.** Counts and blockers answer different questions. Shortfalls are reported, while excess counts currently do not create a readiness blocker. **Loading state.** The workspace lifecycle separates migration from ordinary customer work. That lifecycle belongs to the workspace service, which the run consults. **Reset.** Reset clears the loaded records while retaining the agreed security, configuration, uploads and run history. **Activation.** Activation has separate authority from import management and admits the customer’s staff to the workspace. ## How It Relates Data jobs perform the record changes; uploads hold the input files; producer keys support reconciliation. Workspace lifecycle services provide loading, reset and activation. The run coordinates these concerns without owning the applications’ business records. ## What It Could Become The product should enforce readiness before activation and make unresolved child-job cancellation visible. Staff also need reliable handling of failed submissions, unexpected progress responses and inconsistent counts. Dedicated migration screens or a background driver are possible extensions; Membership’s declared operations need further implementation before a run can load them.