--- id: CAP-PFM-DJE-003 title: Process valid records and report individual refusals feature: FEAT-PFM-DJE-01 product_area: platform module: data-job-engine revision: 1 status: draft product_intent: confirmed implementation_evidence: observed supersedes: - RUL-DJE-ENG-004 - RUL-DJE-ENG-023 source_sha: macroservice-libraries: 54536a441935c6e09fdb7f96f0bec71c52ba4e03 services-crm: a8d1364f1dd44b723bb5f9acba12f983e9ec8d69 services-platform: ad3f6a6582843008ed0f5a17b131878d15829af1 intent_sources: - 'services-platform:docs/milestones/plans/PLAT-M19-plan.md#S02' - 'services-platform:docs/milestones/plans/PLAT-M19-plan.md#S03' - 'services-platform:docs/milestones/plans/PLAT-M19-plan.md#S04' - 'services-platform:docs/milestones/plans/PLAT-M19-plan.md#S07' - 'services-platform:docs/milestones/plans/PLAT-M19-plan.md#A6' - 'services-platform:docs/milestones/plans/PLAT-M19-plan.md#A8' - 'services-platform:docs/milestones/plans/PLAT-M19-plan.md#A9' - 'services-crm:docs/milestones/CRM-M7-data-operations.md#E-11' purpose: >- Background processing preserves successful work while identifying records that need correction or a later retry. outcomes: - Changes and their receipts commit together. - Individual refusals identify the problem without exposing record values. - Poor input quality or revoked authority stops further processing. concepts: - CON-PFM-DJE-001 actors: - ACT-PFM-012 - ACT-PFM-035 surfaces: - job preconditions: [] postconditions: [] permissions: - key: crm.contacts.update where_checked: service scope: record - key: crm.contacts.bulk where_checked: service scope: tenant rules: - id: BR-CAP-PFM-DJE-003-01 proved_by: [ATS-PFM-DJE-001-12, ATS-PFM-DJE-003-01, ATS-PFM-DJE-003-11, ATS-PFM-DJE-003-12, ATS-PFM-DJE-004-08, TST-PFM-DJE-117, TST-PFM-DJE-139] statement: >- A record change commits with its receipt, allowing repeated or concurrent delivery of the same batch without applying that change twice. kind: invariant enforced_at: - service - database must_prove: true provenance: test source: >- services-crm:modules/data-jobs/tests/integration/services/queue/data-job-worker.integration.spec.js#changes each contact once between them, whichever delivery got there first origin: 'services-platform:docs/milestones/plans/PLAT-M19-plan.md#S02' - id: BR-CAP-PFM-DJE-003-02 proved_by: [ATS-PFM-DJE-003-02, ATS-PFM-DJE-003-13, ATS-PFM-DJE-003-14, ATS-PFM-DJE-005-09, ATS-PFM-DJE-012-11, TST-PFM-DJE-100, TST-PFM-DJE-117] statement: >- Invalid, absent, locked, stale or uniqueness-breaking records receive individual refusals while eligible records commit and the job completes with rejections. kind: policy enforced_at: - service - database must_prove: true provenance: test source: >- services-crm:modules/data-jobs/tests/integration/services/jobs/contact-data-jobs.integration.spec.js#refuses the other five, counting each as what refused it origin: 'services-platform:docs/milestones/plans/PLAT-M19-plan.md#S03' - id: BR-CAP-PFM-DJE-003-03 proved_by: [ATS-PFM-DJE-003-03, TST-PFM-DJE-117] statement: A record the submitter cannot access is reported as not found rather than access denied. kind: policy enforced_at: - service must_prove: true provenance: test source: >- services-crm:modules/data-jobs/tests/integration/services/jobs/contact-data-jobs.integration.spec.js#rejects that contact as not found, which tells the person nothing about a record they may not see origin: none - id: BR-CAP-PFM-DJE-003-04 proved_by: [ATS-PFM-DJE-003-04, ATS-PFM-DJE-003-13, ATS-PFM-DJE-005-09, TST-PFM-DJE-100, TST-PFM-DJE-117] statement: >- Refusal information includes a code, relevant field paths, any constraint name and a message safe for operators, without quoting record values. kind: invariant enforced_at: - service - database must_prove: true provenance: test source: >- services-crm:modules/data-jobs/tests/integration/services/jobs/contact-data-jobs.integration.spec.js#names the rule a refused value broke, and quotes no value of the record origin: 'services-platform:docs/milestones/plans/PLAT-M19-plan.md#A6' - id: BR-CAP-PFM-DJE-003-05 proved_by: [ATS-PFM-DJE-003-05, TST-PFM-DJE-011] statement: >- When a database refusal affects a commit unit, processing splits it until the failing record is isolated; committed units remain intact. kind: calculation enforced_at: - service must_prove: true provenance: test source: >- macroservice-libraries:packages/data-jobs/src/commit-split.spec.ts#halves the unit until the failing object stands alone, and isolates only it origin: 'services-platform:docs/milestones/plans/PLAT-M19-plan.md#S04' - id: BR-CAP-PFM-DJE-003-06 proved_by: [ATS-PFM-DJE-003-06, TST-PFM-DJE-027] statement: >- An atomic group’s members are written together or refused together, with required members ordered ahead of their dependants inside that group. kind: policy enforced_at: - service must_prove: true provenance: test source: >- macroservice-libraries:packages/data-jobs/src/tests/integration/group-refusals.integration.spec.ts#writes none of the group, rejects the other two as group_member_refused, and applies the items of no group origin: 'services-crm:docs/milestones/CRM-M7-data-operations.md#E-11' - id: BR-CAP-PFM-DJE-003-07 proved_by: [ATS-PFM-DJE-003-07, TST-PFM-DJE-019] statement: >- The default guard pauses processing when rejected, blocked and conflicted outcomes exceed 50% of 100–999 cumulative attempts or 20% from 1,000 attempts onward. kind: policy enforced_at: - service must_prove: true provenance: observed source: 'macroservice-libraries:packages/data-jobs/src/rejection-guard.ts:rejectionGuardVerdict' origin: 'services-platform:docs/milestones/plans/PLAT-M19-plan.md#A9' - id: BR-CAP-PFM-DJE-003-08 proved_by: [ATS-PFM-DJE-003-08] statement: >- Transient database errors are retried; exhausted transient failures pause the job for recovery, while systemic failures end it with an operator-safe summary. kind: policy enforced_at: - service must_prove: true provenance: documented source: 'services-crm:docs/data-jobs.md#What each contact''s outcome can be' origin: none - id: BR-CAP-PFM-DJE-003-09 proved_by: [ATS-PFM-DJE-003-09, TST-PFM-DJE-062, TST-PFM-DJE-139] statement: >- Each batch rechecks the submitter’s authority; lost permissions fail the job as access revoked, mark remaining work unattempted and preserve earlier commits. kind: permission enforced_at: - service must_prove: true provenance: test source: >- services-crm:modules/data-jobs/tests/integration/services/queue/data-job-worker.integration.spec.js#fails the whole job with the code, not just the batch origin: 'services-platform:docs/milestones/plans/PLAT-M19-plan.md#S07' - id: BR-CAP-PFM-DJE-003-10 proved_by: [] statement: >- Job processing uses the owning record’s save behavior, including its validation, history, events and derived values. kind: invariant enforced_at: - service must_prove: false provenance: documented source: 'services-crm:docs/data-jobs.md#Imports' origin: none - id: BR-CAP-PFM-DJE-003-11 proved_by: [ATS-PFM-DJE-003-10, TST-PFM-DJE-117] statement: >- A contact update that repeats its current values reports applied with unchanged detail and creates no new version, history entry or change event. kind: policy enforced_at: - service must_prove: true provenance: test source: >- services-crm:modules/data-jobs/tests/integration/services/jobs/contact-data-jobs.integration.spec.js#writes no revision, so the history names only the job that changed the contact origin: none scenarios: - id: SC-CAP-PFM-DJE-003-01 given: Five of 100 proposed contact updates contain first names longer than 200 characters. when: The worker processes the job. then: Ninety-five updates apply and five receive field-specific refusals that omit the invalid values. provenance: test source: >- react-console:acceptance/data-jobs/playwright/journeys/foundations/DJE-ACC-0004.spec.ts#DJE-ACC-0004 - id: SC-CAP-PFM-DJE-003-02 given: A worker exits after committing a batch. when: The queue redelivers that work. then: Receipts prevent duplicate changes and keep the job totals consistent. provenance: test source: >- services-crm:modules/data-jobs/tests/integration/services/jobs/initial-import-guarantees.integration.spec.js#applies nothing of the dead worker’s batch again when its message comes back, and finishes the batch edge_cases: [] constraints: [] depends_on: [] related: - CAP-PFM-DJE-004 - CAP-PFM-DJE-005 - CAP-PFM-DJE-012 journeys: - JNY-PFM-DJE-001 - JNY-PFM-DJE-002 open_questions: [] evidence: - >- Reviewed evidence: macroservice-libraries (54536a44): README.md (What the engine is; Groups succeed or fail together); src/run-data-job-batch.ts, commit-split.ts, rejection-guard.ts, data-job-config.ts; tests commit-split.spec.ts, rejection-guard.spec.ts, group-refusals.integration.spec.ts, batch-skips.integration.spec.ts, data-job-engine.integration.spec.ts. - >- Reviewed evidence: services-crm (a8d1364f): docs/data-jobs.md (What each contact's outcome can be; How a job runs); services/queue/data-job-worker.ts, sqs-redelivery.ts; services/operations/error-classification.ts; tests data-job-worker.integration.spec.js, contact-data-jobs.integration.spec.js. - 'Reviewed evidence: services-platform (ad3f6a65): docs/data-jobs/platform-data-operations.md §3.' controls: [TC-AVL-001, TC-AVL-004, TC-AZ-001, TC-AZ-002, TC-AZ-003, TC-AZ-004, TC-DAT-001, TC-DAT-003, TC-IAM-001, TC-IAM-006, TC-LOG-001, TC-LOG-002, TC-LOG-003, TC-LOG-004, TC-LOG-005, TC-MON-001, TC-PI-001, TC-PI-002, TC-PI-003, TC-PI-004, TC-PI-005, TC-PRV-001, TC-PRV-002, TC-TEN-001, TC-TEN-002, TC-TEN-003, TC-VEN-001, TC-VEN-002, TC-VEN-003] exercised_by: [FAS-PFM-DJE-01-01, FAS-PFM-DJE-01-02, FAS-PFM-DJE-01-03, FAS-PFM-DJE-01-04, FAS-PFM-DJE-01-05] suites: [TST-PFM-DJE-011, TST-PFM-DJE-019, TST-PFM-DJE-027, TST-PFM-DJE-062, TST-PFM-DJE-100, TST-PFM-DJE-117, TST-PFM-DJE-139] --- # Process valid records and report individual refusals ## Purpose Background processing preserves successful work while identifying records that need correction or a later retry. ## How It Works Workers process a job in batches and check its state before continuing. They recheck the submitter’s permission so withdrawn access stops later work. Each successful change commits with a receipt. Redelivery consults those receipts instead of applying the same change again. Individual validation failures become coded outcomes while other records proceed. Database refusals trigger smaller commit units that isolate the failing record. Atomic groups remain a single success or refusal unit. The rejection guard pauses a job with a high cumulative refusal rate. Temporary database problems can be retried, while a systemic problem produces a failure summary. Staff inspect the resulting outcomes to decide what needs correction. ## Relationships - Submission supplies the retained input. - Job controls govern interruption and recovery. - Progress inspection exposes the recorded outcomes.