---
id: CAP-PFM-DJE-014
title: Process file jobs within shared workload limits
feature: FEAT-PFM-DJE-04
product_area: platform
module: data-job-engine
revision: 1
status: draft
product_intent: confirmed
implementation_evidence: observed
supersedes:
  - RUL-DJE-ENG-019
  - RUL-DJE-ENG-026
source_sha:
  macroservice-libraries: 54536a441935c6e09fdb7f96f0bec71c52ba4e03
  services-crm: a8d1364f1dd44b723bb5f9acba12f983e9ec8d69
intent_sources:
  - 'services-platform:docs/milestones/plans/PLAT-M19-plan.md#A1'
  - 'services-platform:docs/milestones/plans/PLAT-M19-plan.md#S16'
  - 'services-platform:docs/milestones/plans/PLAT-M19-plan.md#S21'
purpose: >-
  Large imports progress through streamed batches while admission limits share processing capacity across
  competing workspaces.
outcomes:
  - Malformed input is traceable to file lines.
  - 'Admission observes job, workspace and shared ceilings.'
  - Stalled admission produces a visible failure reason.
concepts:
  - CON-PFM-DJE-001
  - CON-PFM-DJE-006
actors:
  - ACT-PFM-012
  - ACT-PFM-036
  - ACT-PFM-035
surfaces:
  - job
  - api
preconditions: []
postconditions: []
permissions:
  - key: crm.contacts.bulk
    where_checked: api
    scope: tenant
  - key: crm.organizations.bulk
    where_checked: api
    scope: tenant
rules:
  - id: BR-CAP-PFM-DJE-014-01
    proved_by: [ATS-PFM-DJE-011-10, ATS-PFM-DJE-014-01, ATS-PFM-DJE-014-10, TST-PFM-DJE-107]
    statement: >-
      A file job reads the exact version of a completed upload in its workspace; absent or foreign
      uploads receive the same refusal, and unfinished uploads report their state.
    kind: validation
    enforced_at:
      - api
      - service
    must_prove: true
    provenance: test
    source: >-
      services-crm:modules/data-jobs/tests/e2e/bulk-data-jobs.e2e.spec.js#refuses an upload id of no
      upload at all in the same words, so neither answer tells a caller whose it is
    origin: 'services-platform:docs/milestones/plans/PLAT-M19-plan.md#S16'
  - id: BR-CAP-PFM-DJE-014-02
    proved_by: [ATS-PFM-DJE-014-02, TST-PFM-DJE-024, TST-PFM-DJE-065]
    statement: >-
      File jobs accept JSON Lines and header-based CSV, including gzip variants identified by filename;
      single JSON documents and zip archives are refused.
    kind: validation
    enforced_at:
      - service
    must_prove: true
    provenance: documented
    source: 'services-crm:docs/data-jobs.md#The bulk lane'
    origin: none
  - id: BR-CAP-PFM-DJE-014-03
    proved_by: [ATS-PFM-DJE-012-15, ATS-PFM-DJE-014-03, ATS-PFM-DJE-014-11, TST-PFM-DJE-008]
    statement: >-
      Chunks target 100 records or two million bytes, allowing oversized records and existing groups to
      exceed those targets; a repeated group from the most recently closed chunk is rejected.
    kind: calculation
    enforced_at:
      - service
    must_prove: true
    provenance: observed
    source: >-
      macroservice-libraries:packages/data-jobs/src/chunk-plan.ts:shouldCloseChunk;
      macroservice-libraries:packages/data-jobs/src/chunk-source.ts:ChunkRun
    origin: none
  - id: BR-CAP-PFM-DJE-014-04
    proved_by: [ATS-PFM-DJE-014-04, TST-PFM-DJE-024]
    statement: >-
      Unreadable lines receive line-numbered refusals, while exceeding the configured malformed-input
      rate fails the job.
    kind: policy
    enforced_at:
      - service
    must_prove: true
    provenance: test
    source: >-
      macroservice-libraries:packages/data-jobs/src/tests/integration/bulk-data-jobs.integration.spec.ts#fails
      the job when most of the file cannot be read, rather than reading nine million lines of it
    origin: none
  - id: BR-CAP-PFM-DJE-014-05
    proved_by: [ATS-PFM-DJE-014-05, ATS-PFM-DJE-014-12, ATS-PFM-DJE-019-10, TST-PFM-DJE-024, TST-PFM-DJE-112]
    statement: >-
      Admission is bounded by the remaining job, workspace and service capacity; a workspace ceiling is
      below a divisible service share and the service ceiling is reduced by its capacity factor.
    kind: calculation
    enforced_at:
      - service
    must_prove: true
    provenance: observed
    source: >-
      macroservice-libraries:packages/data-jobs/src/tests/integration/bulk-data-jobs.integration.spec.ts#hands
      over no more batches at once than the budget allows;
      macroservice-libraries:packages/data-jobs/src/admission-budget.ts:tenantBulkCeiling
    origin: 'services-platform:docs/milestones/plans/PLAT-M19-plan.md#S21'
  - id: BR-CAP-PFM-DJE-014-06
    proved_by: [ATS-PFM-DJE-014-06, TST-PFM-DJE-002]
    statement: >-
      A job waiting longer than ten minutes receives priority; other grants rotate across workspaces with
      fewer batches in flight, selecting older jobs within each workspace.
    kind: policy
    enforced_at:
      - service
    must_prove: true
    provenance: test
    source: >-
      macroservice-libraries:packages/data-jobs/src/admission-planner.spec.ts#puts a starved job ahead of
      the round-robin
    origin: 'services-platform:docs/milestones/plans/PLAT-M19-plan.md#S21'
  - id: BR-CAP-PFM-DJE-014-07
    proved_by: [ATS-PFM-DJE-014-07, ATS-PFM-DJE-014-12, ATS-PFM-DJE-019-10, TST-PFM-DJE-024]
    statement: >-
      Admission fails after 15 minutes without room when the job has no work in flight and its blocking
      admissions are also stalled; a zero capacity factor holds work without that failure.
    kind: policy
    enforced_at:
      - service
    must_prove: true
    provenance: test
    source: >-
      macroservice-libraries:packages/data-jobs/src/tests/integration/bulk-data-jobs.integration.spec.ts#fails
      a job whose ceiling has been held by work that is not moving, rather than waiting for room for ever
    origin: none
  - id: BR-CAP-PFM-DJE-014-08
    proved_by: [ATS-PFM-DJE-014-08, TST-PFM-DJE-024]
    statement: 'Batches belonging to paused, cancelled, failed or completed jobs do not consume admission room.'
    kind: policy
    enforced_at:
      - service
    must_prove: true
    provenance: test
    source: >-
      macroservice-libraries:packages/data-jobs/src/tests/integration/bulk-data-jobs.integration.spec.ts#admits
      the next job when the one holding the tenant’s ceiling has been paused, because a paused batch is
      not in flight
    origin: none
  - id: BR-CAP-PFM-DJE-014-09
    proved_by: []
    statement: >-
      Default cluster bulk limits are twelve batches in production and four in development, divided
      between CRM and Accounting; workload profiles request workspace limits, with four as the fallback
      before share capping.
    kind: calculation
    enforced_at:
      - service
    must_prove: false
    provenance: documented
    source: 'macroservice-libraries:packages/data-jobs/README.md#The bulk lane'
    origin: none
  - id: BR-CAP-PFM-DJE-014-10
    proved_by: [ATS-PFM-DJE-012-15, ATS-PFM-DJE-014-11]
    statement: >-
      File-job progress distinguishes source reading, batch planning, application and completion, without
      persisting one input row per record.
    kind: policy
    enforced_at:
      - service
      - console
    must_prove: false
    provenance: documented
    source: 'services-crm:docs/data-jobs.md#The bulk lane'
    origin: none
  - id: BR-CAP-PFM-DJE-014-11
    proved_by: [ATS-PFM-DJE-014-09]
    statement: >-
      Within one source-reading invocation, repeated-group detection remembers only the most recently
      closed chunk; distant or resumed repetitions can escape it.
    kind: invariant
    enforced_at:
      - service
    must_prove: true
    provenance: observed
    source: 'macroservice-libraries:packages/data-jobs/src/chunk-source.ts:closedGroupKeys'
    origin: none
scenarios:
  - id: SC-CAP-PFM-DJE-014-01
    given: Three lines in a thousand-line file cannot be parsed.
    when: The engine processes the file.
    then: 'The job records 997 applications and refusals at lines 100, 500 and 900.'
    provenance: test
    source: >-
      macroservice-libraries:packages/data-jobs/src/tests/integration/bulk-data-jobs.integration.spec.ts#reads
      a thousand-line file into chunks, applies 997 records and rejects three lines by their line number
  - id: SC-CAP-PFM-DJE-014-02
    given: Two workspaces have imports competing for a four-batch budget.
    when: Admission distributes available batches.
    then: The cited fixture grants two batches to each workspace.
    provenance: test
    source: >-
      services-crm:modules/data-jobs/tests/integration/services/jobs/bulk-admission.integration.spec.js#leaves
      each tenant holding two batches of the ceiling’s four
edge_cases: []
constraints: []
depends_on: []
related:
  - CAP-PFM-DJE-011
  - CAP-PFM-DJE-012
  - CAP-PFM-DJE-019
journeys:
  - JNY-PFM-DJE-001
  - JNY-PFM-DJE-005
open_questions: []
evidence:
  - >-
    Reviewed evidence: macroservice-libraries (54536a44): README.md (The bulk lane); src/chunk-source.ts,
    chunk-plan.ts, bulk-data-job-tasks.ts, admission-budget.ts, admission-planner.ts,
    bulk-batch-budget.ts, bulk-data-job-workflow.ts; tests chunk-plan.spec.ts, admission-budget.spec.ts,
    admission-planner.spec.ts, admit-batches.spec.ts, bulk-data-jobs.integration.spec.ts.
  - >-
    Reviewed evidence: services-crm (a8d1364f): docs/data-jobs.md (The bulk lane);
    services/jobs/bulk-data-job-submission.ts, bulk-admission.ts, bulk-data-job.ts;
    services/persistence/tenant-workload-policies.ts; db/bulk-data-jobs.table.ts; tests
    bulk-admission.integration.spec.js, bulk-data-job-lane.integration.spec.js,
    bulk-data-jobs.integration.spec.js, bulk-data-jobs.e2e.spec.js.
controls: [TC-AVL-004, TC-AZ-001, TC-AZ-002, TC-AZ-003, TC-AZ-004, TC-DAT-001, TC-DAT-003, TC-DAT-006, TC-IAM-001, TC-IAM-002, TC-IAM-004, TC-IAM-005, TC-LOG-001, TC-LOG-004, TC-LOG-005, TC-MON-001, TC-PI-001, TC-PI-002, TC-PI-003, TC-PI-004, TC-PI-005, TC-PRV-001, TC-PRV-002, TC-TEN-001, TC-TEN-002, TC-TEN-003, TC-VEN-001, TC-VEN-002, TC-VEN-003]
exercised_by: [FAS-PFM-DJE-04-01, FAS-PFM-DJE-04-02, FAS-PFM-DJE-04-03, FAS-PFM-DJE-04-04, FAS-PFM-DJE-04-05]
suites: [TST-PFM-DJE-002, TST-PFM-DJE-008, TST-PFM-DJE-024, TST-PFM-DJE-065, TST-PFM-DJE-107, TST-PFM-DJE-112]
---

# Process file jobs within shared workload limits

## Purpose

Large imports progress through streamed batches while admission limits share processing capacity across competing workspaces.

## How It Works

A file job starts from a completed upload and its recorded storage version. Source reading streams input into chunk files rather than retaining the entire file. Chunk targets can be exceeded to preserve a group already in the open chunk. Malformed lines receive traceable outcomes, and a mostly unreadable file fails early. Admission controls how many planned batches reach workers. The tightest remaining ceiling determines available room. Capacity protection can reduce that room without undoing batches already running. Priority and rotation distribute it among waiting jobs. Stalled blocking work can fail admission with its ceiling identified. Whole-file group integrity remains an unresolved implementation weakness because repetition memory is bounded.

## Relationships

- Upload sessions supply the fixed input version.
- Producer imports provide record operations for file batches.
- Capacity protection adjusts shared admission.