--- id: CAP-PFM-MTD-016 title: Load and Reconcile Record Files feature: FEAT-PFM-MTD-05 product_area: platform module: metadata revision: 1 status: draft product_intent: proposed implementation_evidence: observed supersedes: - CAP-CO-DAT-001 - CAP-CO-DAT-002 - CAP-CO-DAT-003 - CAP-CO-DAT-004 - CAP-CO-DAT-005 - CAP-CO-DAT-006 - CAP-CO-DAT-007 - CAP-CO-PRP-019 - CAP-CO-SEC-007 - CAP-CO-XAP-008 source_sha: services-platform: ad3f6a6582843008ed0f5a17b131878d15829af1 react-console: baf6dae382d6f41dce976c7f848667ccc37e076f intent_sources: - 'services-platform:docs/milestones/PLAT-M7-custom-record-loads-reports-events.md#Scope (Proposed; no plan)' - 'services-platform:docs/milestones/plans/PLAT-M6-plan.md#2. Scope (loading links from a file, built)' purpose: >- A repeated source file should reconcile records without duplicating the register. Data jobs validate the expected impact before applying normal record writes. outcomes: - A repeated source file should reconcile records without duplicating the register. - >- Relationship correction lines currently replace their complete attributes, which can erase omitted history. actors: - ACT-PFM-008 - ACT-PFM-012 surfaces: - job - api preconditions: - 'Object properties are configured before loading, and object and relationship types are active.' postconditions: [] permissions: - key: platform.associations.create where_checked: api scope: tenant - key: platform.associations.update where_checked: service scope: tenant - key: platform.custom-records.create where_checked: service scope: record - key: platform.custom-records.update where_checked: service scope: record rules: - id: BR-CAP-PFM-MTD-016-01 proved_by: [ATS-PFM-MTD-016-01, TST-PFM-MTD-135] statement: >- Producer keys identify load lines within their object and become external identifiers; new crosswalk keys can adopt matching records, known keys patch only differing supplied properties and identical reloads make no changes. kind: policy enforced_at: - service must_prove: true provenance: documented source: 'services-platform:docs/custom-objects.md#Import, export and reset' origin: none - id: BR-CAP-PFM-MTD-016-02 proved_by: [ATS-PFM-MTD-016-02, TST-PFM-MTD-134] statement: >- Preflight validates object state, property contracts, references, requirements, conditions, uniqueness within storage and the file, and caller reach, reporting rejected lines and properties without values. kind: validation enforced_at: - service must_prove: true provenance: test source: >- services-platform:tests/acceptance/custom-objects/journeys/data-operations/CO-ACC-0400.spec.js#step 3: the rejections name each bad line and its property, never a value or an error class origin: none - id: BR-CAP-PFM-MTD-016-03 proved_by: [ATS-PFM-MTD-016-03, TST-PFM-MTD-134] statement: >- Validation computes impact without writing records, and apply refuses a mismatched impact digest while keeping the job validated. kind: validation enforced_at: - service must_prove: true provenance: test source: >- services-platform:tests/acceptance/custom-objects/journeys/data-operations/CO-ACC-0400.spec.js#step 4: an apply with a stale digest is refused 409 impact_digest_mismatch and the job stays validated origin: none - id: BR-CAP-PFM-MTD-016-04 proved_by: [ATS-PFM-MTD-016-04, ATS-PFM-MTD-016-08, TST-PFM-MTD-134] statement: >- Loaded records use normal write behaviour for numbering, names, unique values, assets and audits and reject archived targets until restoration. kind: invariant enforced_at: - service must_prove: true provenance: test source: >- services-platform:tests/acceptance/custom-objects/journeys/data-operations/CO-ACC-0400.spec.js#step 6: every created boat reads back as the file said, and the id refused at step 2 named no record origin: none - id: BR-CAP-PFM-MTD-016-05 proved_by: [ATS-PFM-MTD-016-05, ATS-PFM-MTD-016-08, TST-PFM-MTD-136] statement: >- Link loads resolve type and endpoint keys or identities, validate type state, endpoint kind, dates, attributes and cardinality, and refuse changes to known link endpoints before applying. kind: validation enforced_at: - service must_prove: true provenance: test source: >- services-platform:tests/acceptance/custom-objects/journeys/data-operations/CO-ACC-0402.spec.js#step 9: OWN-2 naming another boat is refused because endpoints never change, and with its own endpoints its notes update origin: none - id: BR-CAP-PFM-MTD-016-06 proved_by: [ATS-PFM-MTD-016-06, TST-PFM-MTD-136] statement: >- A known-link correction replaces all link attributes, clears omitted dates, position and notes and reopens an ended link when its end date is omitted, without warning. kind: policy enforced_at: - service must_prove: true provenance: test source: >- services-platform:tests/acceptance/custom-objects/journeys/data-operations/CO-ACC-0402.spec.js#step 6: a CRW-1 line with only isPrimary replaces the association and clears its dates, position and notes, silently origin: none - id: BR-CAP-PFM-MTD-016-07 proved_by: [ATS-PFM-MTD-016-07, TST-PFM-MTD-137] statement: >- Link-load submission requires create permission and each line needs create or update according to whether it is new, while results remain limited to the submitter and workspace. kind: permission enforced_at: - api - service must_prove: true provenance: test source: >- services-platform:tests/acceptance/custom-objects/journeys/data-operations/CO-ACC-0404.spec.js#step 5: a clerk with the create key only has the known key refused as not found and the new key created origin: none - id: BR-CAP-PFM-MTD-016-08 proved_by: [] statement: >- Load controls support pause, resume, cancel and retry, and resubmission of an idempotency key returns its original job. kind: lifecycle enforced_at: - service must_prove: false provenance: test source: >- services-platform:tests/acceptance/custom-objects/journeys/data-operations/CO-ACC-0400.spec.js#step 9: the list is newest first with the three jobs, and a resent idempotent request answers the first job origin: none scenarios: - id: SC-CAP-PFM-MTD-016-01 given: A club has source files for builders and boats. when: Validate the files and apply their accepted impact. then: >- Validation reports rejected lines without writing records, and application creates the valid boats and their stated information. provenance: test source: >- services-platform:tests/acceptance/custom-objects/journeys/data-operations/CO-ACC-0400.spec.js#step 5: applies with the digest; the job completes with rejections, writing three lines and rejecting five edge_cases: [] constraints: [] depends_on: - CAP-PFM-MTD-012 - CAP-PFM-MTD-018 related: - CAP-PFM-MTD-009 journeys: [] open_questions: [] concepts: - CON-PFM-MTD-005 evidence: - >- services-platform at ad3f6a6582843008ed0f5a17b131878d15829af1: modules/associations/aws/api/association-data-jobs.routes.ts ; modules/custom-objects/services/persistence/custom-object-records.ts ; tests/acceptance/custom-objects/journeys/data-operations/**; docs/custom-objects.md#Import, export and reset. The record import operation and its routes are the Data Job Engine's , named as a dependency only. controls: [TC-AVL-001, TC-AVL-004, TC-AZ-001, TC-AZ-002, TC-AZ-003, TC-AZ-004, TC-LOG-002, TC-LOG-003, TC-LOG-004, TC-LOG-005, TC-MON-001, TC-PI-001, TC-PI-002, TC-PI-003, TC-PI-004, TC-PI-005, TC-TEN-001, TC-TEN-002, TC-TEN-003, TC-VEN-001, TC-VEN-002, TC-VEN-003] exercised_by: [FAS-PFM-MTD-05-05] suites: [TST-PFM-MTD-134, TST-PFM-MTD-135, TST-PFM-MTD-136, TST-PFM-MTD-137] --- # Load and Reconcile Record Files ## Purpose A repeated source file should reconcile records without duplicating the register. Data jobs validate the expected impact before applying normal record writes. ## How It Works An authorized submitter provides record or relationship lines identified by source keys. Validation resolves existing targets and checks the proposed writes. It reports line problems before applying records. The caller applies the job using the validated impact. Record lines change only differing supplied properties through normal storage rules. Relationship correction lines currently replace their complete attributes, which can erase omitted history. Job controls support recovery and repetition, while result access remains with the submitter. ## Relationships The Data Job Engine owns execution and job controls. Custom-record and relationship stores provide the writes and validation contracts.