Process file jobs within shared workload limits

Purpose

Large imports progress through streamed batches while admission limits share processing capacity across competing workspaces.

How It Works

A file job starts from a completed upload and its recorded storage version. Source reading streams input into chunk files rather than retaining the entire file. Chunk targets can be exceeded to preserve a group already in the open chunk. Malformed lines receive traceable outcomes, and a mostly unreadable file fails early. Admission controls how many planned batches reach workers. The tightest remaining ceiling determines available room. Capacity protection can reduce that room without undoing batches already running. Priority and rotation distribute it among waiting jobs. Stalled blocking work can fail admission with its ceiling identified. Whole-file group integrity remains an unresolved implementation weakness because repetition memory is bounded.

Relationships

Upload sessions supply the fixed input version.
Producer imports provide record operations for file batches.
Capacity protection adjusts shared admission.

Show evidence (sources, provenance, origins, pinned commits)

The Formal Contract

The narrative above explains; the sections below are authoritative — the rules, scenarios and tests this document binds.

Outcomes: Malformed input is traceable to file lines.Admission observes job, workspace and shared ceilings.Stalled admission produces a visible failure reason.
Surfaces: jobapi
Intent sources: services-platform:docs/milestones/plans/PLAT-M19-plan.md#A1services-platform:docs/milestones/plans/PLAT-M19-plan.md#S16services-platform:docs/milestones/plans/PLAT-M19-plan.md#S21

Controls

29 controls apply, from the corpus control catalogue.

TC-AVL-004 TC-AZ-001 TC-AZ-002 TC-AZ-003 TC-AZ-004 TC-DAT-001 TC-DAT-003 TC-DAT-006 TC-IAM-001 TC-IAM-002 TC-IAM-004 TC-IAM-005 TC-LOG-001 TC-LOG-004 TC-LOG-005 TC-MON-001 TC-PI-001 TC-PI-002 TC-PI-003 TC-PI-004 TC-PI-005 TC-PRV-001 TC-PRV-002 TC-TEN-001 TC-TEN-002 TC-TEN-003 TC-VEN-001 TC-VEN-002 TC-VEN-003

Permissions

Key	Checked at	Scope
`crm.contacts.bulk`	api	tenant
`crm.organizations.bulk`	api	tenant

Business Rules

11 rules, 9 to be proved before approval. A rule's product intent is derived from its origin: 11 confirmed by a product decision, the rest inheriting this capability's confirmed.

Id	Rule	Kind	Enforced at	Provenance
`01`must prove Confirmed	A file job reads the exact version of a completed upload in its workspace; absent or foreign uploads receive the same refusal, and unfinished uploads report their state. origin: services-platform:docs/milestones/plans/PLAT-M19-plan.md#S16 proved by `ATS-PFM-DJE-011-10`, `ATS-PFM-DJE-014-01`, `ATS-PFM-DJE-014-10`, `TST-PFM-DJE-107`	validation	api, service	test services-crm:modules/data-jobs/tests/e2e/bulk-data-jobs.e2e.spec.js#refuses an upload id of no upload at all in the same words, so neither answer tells a caller whose it is
`02`must prove Confirmed	File jobs accept JSON Lines and header-based CSV, including gzip variants identified by filename; single JSON documents and zip archives are refused. proved by `ATS-PFM-DJE-014-02`, `TST-PFM-DJE-024`, `TST-PFM-DJE-065`	validation	service	documented services-crm:docs/data-jobs.md#The bulk lane
`03`must prove Confirmed	Chunks target 100 records or two million bytes, allowing oversized records and existing groups to exceed those targets; a repeated group from the most recently closed chunk is rejected. proved by `ATS-PFM-DJE-012-15`, `ATS-PFM-DJE-014-03`, `ATS-PFM-DJE-014-11`, `TST-PFM-DJE-008`	calculation	service	observed macroservice-libraries:packages/data-jobs/src/chunk-plan.ts:shouldCloseChunk; macroservice-libraries:packages/data-jobs/src/chunk-source.ts:ChunkRun
`04`must prove Confirmed	Unreadable lines receive line-numbered refusals, while exceeding the configured malformed-input rate fails the job. proved by `ATS-PFM-DJE-014-04`, `TST-PFM-DJE-024`	policy	service	test macroservice-libraries:packages/data-jobs/src/tests/integration/bulk-data-jobs.integration.spec.ts#fails the job when most of the file cannot be read, rather than reading nine million lines of it
`05`must prove Confirmed	Admission is bounded by the remaining job, workspace and service capacity; a workspace ceiling is below a divisible service share and the service ceiling is reduced by its capacity factor. origin: services-platform:docs/milestones/plans/PLAT-M19-plan.md#S21 proved by `ATS-PFM-DJE-014-05`, `ATS-PFM-DJE-014-12`, `ATS-PFM-DJE-019-10`, `TST-PFM-DJE-024`, `TST-PFM-DJE-112`	calculation	service	observed macroservice-libraries:packages/data-jobs/src/tests/integration/bulk-data-jobs.integration.spec.ts#hands over no more batches at once than the budget allows; macroservice-libraries:packages/data-jobs/src/admission-budget.ts:tenantBulkCeiling
`06`must prove Confirmed	A job waiting longer than ten minutes receives priority; other grants rotate across workspaces with fewer batches in flight, selecting older jobs within each workspace. origin: services-platform:docs/milestones/plans/PLAT-M19-plan.md#S21 proved by `ATS-PFM-DJE-014-06`, `TST-PFM-DJE-002`	policy	service	test macroservice-libraries:packages/data-jobs/src/admission-planner.spec.ts#puts a starved job ahead of the round-robin
`07`must prove Confirmed	Admission fails after 15 minutes without room when the job has no work in flight and its blocking admissions are also stalled; a zero capacity factor holds work without that failure. proved by `ATS-PFM-DJE-014-07`, `ATS-PFM-DJE-014-12`, `ATS-PFM-DJE-019-10`, `TST-PFM-DJE-024`	policy	service	test macroservice-libraries:packages/data-jobs/src/tests/integration/bulk-data-jobs.integration.spec.ts#fails a job whose ceiling has been held by work that is not moving, rather than waiting for room for ever
`08`must prove Confirmed	Batches belonging to paused, cancelled, failed or completed jobs do not consume admission room. proved by `ATS-PFM-DJE-014-08`, `TST-PFM-DJE-024`	policy	service	test macroservice-libraries:packages/data-jobs/src/tests/integration/bulk-data-jobs.integration.spec.ts#admits the next job when the one holding the tenant’s ceiling has been paused, because a paused batch is not in flight
`09` Confirmed	Default cluster bulk limits are twelve batches in production and four in development, divided between CRM and Accounting; workload profiles request workspace limits, with four as the fallback before share capping.	calculation	service	documented macroservice-libraries:packages/data-jobs/README.md#The bulk lane
`10` Confirmed	File-job progress distinguishes source reading, batch planning, application and completion, without persisting one input row per record. proved by `ATS-PFM-DJE-012-15`, `ATS-PFM-DJE-014-11`	policy	service, console	documented services-crm:docs/data-jobs.md#The bulk lane
`11`must prove Confirmed	Within one source-reading invocation, repeated-group detection remembers only the most recently closed chunk; distant or resumed repetitions can escape it. proved by `ATS-PFM-DJE-014-09`	invariant	service	observed macroservice-libraries:packages/data-jobs/src/chunk-source.ts:closedGroupKeys

Scenarios

Id	Given	When	Then	Provenance
`01`	Three lines in a thousand-line file cannot be parsed.	The engine processes the file.	The job records 997 applications and refusals at lines 100, 500 and 900.	test macroservice-libraries:packages/data-jobs/src/tests/integration/bulk-data-jobs.integration.spec.ts#reads a thousand-line file into chunks, applies 997 records and rejects three lines by their line number
`02`	Two workspaces have imports competing for a four-batch budget.	Admission distributes available batches.	The cited fixture grants two batches to each workspace.	test services-crm:modules/data-jobs/tests/integration/services/jobs/bulk-admission.integration.spec.js#leaves each tenant holding two batches of the ceiling’s four

Planning

Implementation

Process file jobs within shared workload limits ​

Purpose ​

How It Works ​

Relationships ​