--- id: DOM-PFM-DJE-003 title: Upload realises: CON-PFM-DJE-003 product_area: platform module: data-job-engine revision: 1 status: draft design_intent: confirmed origin: >- services-platform:docs/proposals/rhythm-data-job-engine-plan.md §6.2 (inputs), §15.3 and §16 "Source files" (approved by Andrew 2026-10-12); PLAT-M19 Product Accepted 2026-10-22 implementation_evidence: observed source_sha: services-platform: ad3f6a6582843008ed0f5a17b131878d15829af1 services-crm: a8d1364f1dd44b723bb5f9acba12f983e9ec8d69 react-console: baf6dae382d6f41dce976c7f848667ccc37e076f owned_by: SVC-PFM-DJE-data-jobs readable_by: - SVC-PFM-DJE-crm-data-jobs lifecycle: STATE-PFM-DJE-003 custom_fields: none identity: >- Platform assigns a workspace-scoped upload identity and derives its storage key from the workspace, session and file name. A completed session retains the exact object version for later consumption; another workspace’s identity is concealed as missing. attributes: - name: file name kind: text required: true computed: false notes: >- The client supplies a slash-free name of no more than 255 characters; it contributes to the storage key. - name: content type kind: enum required: true computed: false notes: >- The upload service accepts a defined set of file types, including types for which no dataset reader exists. - name: declared size kind: number required: true computed: false notes: >- The client declares the whole-object byte size, between one byte and 5 TiB. - name: declared digest kind: text required: false computed: false notes: The client may provide a whole-file SHA-256 as an integrity claim. - name: part size and part count kind: number required: true computed: true notes: >- The server uses at least 16 MiB per part, or a larger whole-MiB size derived from a 9,000-part target. - name: state kind: enum required: true computed: true notes: >- The service records whether the transfer is open, completed, abandoned or unavailable. - name: 'stored size, version and entity tag' kind: text required: false computed: true notes: >- Completion records the object size, version and entity tag returned by storage. - name: verified digest kind: text required: false computed: true notes: >- A supplied full-object storage digest takes precedence; otherwise the service measures a declared digest only through 256 MiB. A file without either has no verified digest. - name: completion instant kind: date required: false computed: true notes: The server records the instant completion is saved. - name: storage coordinates kind: text required: true computed: true notes: >- The server keeps bucket, object key and multipart identity out of ordinary upload responses. relationships: - to: DOM-PFM-DJE-001 cardinality: >- many-to-many: one upload can supply several jobs and a job can name several uploads ownership: >- Jobs retain upload identities and versions; uploads do not retain job references. cross_service: true - to: DOM-PFM-DJE-005 cardinality: >- many-to-many: runs can reference several uploads and reuse existing uploads ownership: 'Runs hold upload references, and reset retains the upload records.' cross_service: false invariants: - id: INV-DOM-PFM-DJE-003-01 statement: >- Upload requests carry transfer metadata and signed addresses, while clients send file parts directly to storage. enforced_at: - api - service provenance: observed source: >- services-platform:modules/data-jobs/aws/api/data-jobs.routes.ts:dataJobUploadRoutes - id: INV-DOM-PFM-DJE-003-02 statement: >- Completion accepts the assembled object only when its measured size equals the client declaration; a mismatch marks the session aborted. enforced_at: - service provenance: observed source: >- services-platform:modules/data-jobs/services/uploads/upload-sessions.ts:completeUpload - id: INV-DOM-PFM-DJE-003-03 statement: >- A checkable declared digest must match; a large object without a supplied storage digest retains only the unverified declaration. enforced_at: - service provenance: observed source: >- services-platform:modules/data-jobs/services/uploads/upload-sessions.ts:verifiedDigestOf - id: INV-DOM-PFM-DJE-003-04 statement: >- Session services refuse to reopen settled transfers or abort a completed upload; another file requires another session. enforced_at: - service provenance: test source: >- services-platform:modules/data-jobs/tests/integration/services/uploads/upload-sessions.integration.spec.js#refuses to abort a completed session, whose object a job may already be reading - id: INV-DOM-PFM-DJE-003-05 statement: >- Upload access uses the caller’s workspace context and conceals both foreign sessions and private storage coordinates. enforced_at: - api - service - database provenance: test source: >- services-platform:modules/data-jobs/tests/integration/services/uploads/upload-sessions.integration.spec.js#refuses to complete another tenant’s session, and makes no S3 call for it - id: INV-DOM-PFM-DJE-003-06 statement: >- Storage effects occur before their database bookkeeping and outside the database transaction. enforced_at: - service provenance: documented source: >- services-platform:modules/data-jobs/services/uploads/upload-sessions.ts (module comment, "Which effect comes first") physical: - data_job_uploads - 'Dedicated data-job bucket with inbound, chunk and export storage' evidence: - >- services-platform at ad3f6a65: modules/data-jobs/db/data-job-uploads.table.ts; services/uploads/upload-sessions.ts, upload-storage.ts; services/persistence/uploads.ts; aws/api/data-jobs.routes.ts; aws/resources/data-jobs-storage.s3.ts; docs/data-jobs/uploads.md. - >- services-crm at a8d1364f: services/persistence/data-job-uploads.ts (how CRM reads a completed upload); docs/data-jobs.md#The bulk lane. - >- react-console at baf6dae3: lib/api/data-jobs/multipart.ts, uploads.ts, sha256.ts. --- # Upload ## What It Models An upload records a file transferred for later job input. It separates the client’s declaration from the facts returned by storage. Jobs and migration packages refer to the completed session rather than accepting a caller-selected storage address. ## Identity and Attributes The session identity belongs to Platform. File name, content type, size and optional digest come from the client; part sizing and completion facts come from the service. Retaining the stored version fixes the bytes a consumer reads even if another version later appears. ## Relationships An upload can support multiple jobs and migration runs. A run references its manifest and dataset uploads, but the upload keeps no reverse list of consumers. CRM resolves the shared session record and reads the stored object through its storage adapter. ## Invariants Completion validates size and available integrity evidence before recording success. Settled sessions do not reopen, and foreign workspace sessions appear missing. Storage and bookkeeping form separate steps rather than one atomic commit. ## How It Is Stored and Shared Platform owns the session table and dedicated bucket. Bucket rules expire ordinary input after 30 days unless retained, chunks after 14 days and exports after seven days; incomplete multipart work is abandoned after two days. Reset retains package uploads, while workspace deletion removes session records and stored versions. The upload model has no custom-field declaration. ## Open Design Questions Large files can depend on declared integrity. A crash after assembly can leave the session open over a completed object, and its next interaction may report expiry. Supported upload formats also exceed the implemented dataset-reader formats.