Rhythm 2.0 Specifications

Appearance

DOM-PFM-DJE-003Upload

Markdown

DOM-PFM-DJE-003

Upload

Markdown
  • DocumentDraft
  • ImplementationObserved

Planning

Specified by
CON-PFM-DJE-003

Design

Owned by
SVC-PFM-DJE-data-jobs

KindDomain objectProduct areaPlatformModuleData Job EngineRevision1

Read atservices-platform ad3f6a65services-crm a8d1364freact-console baf6dae3

Upload ​

What It Models ​

An upload records a file transferred for later job input. It separates the client’s declaration from the facts returned by storage. Jobs and migration packages refer to the completed session rather than accepting a caller-selected storage address.

Identity and Attributes ​

The session identity belongs to Platform. File name, content type, size and optional digest come from the client; part sizing and completion facts come from the service. Retaining the stored version fixes the bytes a consumer reads even if another version later appears.

Relationships ​

An upload can support multiple jobs and migration runs. A run references its manifest and dataset uploads, but the upload keeps no reverse list of consumers. CRM resolves the shared session record and reads the stored object through its storage adapter.

Invariants ​

Completion validates size and available integrity evidence before recording success. Settled sessions do not reopen, and foreign workspace sessions appear missing. Storage and bookkeeping form separate steps rather than one atomic commit.

How It Is Stored and Shared ​

Platform owns the session table and dedicated bucket. Bucket rules expire ordinary input after 30 days unless retained, chunks after 14 days and exports after seven days; incomplete multipart work is abandoned after two days. Reset retains package uploads, while workspace deletion removes session records and stored versions. The upload model has no custom-field declaration.

Open Design Questions ​

Large files can depend on declared integrity. A crash after assembly can leave the session open over a completed object, and its next interaction may report expiry. Supported upload formats also exceed the implemented dataset-reader formats.

Relationships

ToCardinalityOwnershipCrosses a serviceOn archive or delete
DOM-PFM-DJE-001many-to-many: one upload can supply several jobs and a job can name several uploadsJobs retain upload identities and versions; uploads do not retain job references.yes
DOM-PFM-DJE-005many-to-many: runs can reference several uploads and reuse existing uploadsRuns hold upload references, and reset retains the upload records.no

Invariants

InvariantProtectsEnforced atProvenance
INV-DOM-PFM-DJE-003-01
Upload requests carry transfer metadata and signed addresses, while clients send file parts directly to storage.
api, serviceobserved
services-platform:modules/data-jobs/aws/api/data-jobs.routes.ts:dataJobUploadRoutes
INV-DOM-PFM-DJE-003-02
Completion accepts the assembled object only when its measured size equals the client declaration; a mismatch marks the session aborted.
serviceobserved
services-platform:modules/data-jobs/services/uploads/upload-sessions.ts:completeUpload
INV-DOM-PFM-DJE-003-03
A checkable declared digest must match; a large object without a supplied storage digest retains only the unverified declaration.
serviceobserved
services-platform:modules/data-jobs/services/uploads/upload-sessions.ts:verifiedDigestOf
INV-DOM-PFM-DJE-003-04
Session services refuse to reopen settled transfers or abort a completed upload; another file requires another session.
servicetest
services-platform:modules/data-jobs/tests/integration/services/uploads/upload-sessions.integration.spec.js#refuses to abort a completed session, whose object a job may already be reading
INV-DOM-PFM-DJE-003-05
Upload access uses the caller’s workspace context and conceals both foreign sessions and private storage coordinates.
api, service, databasetest
services-platform:modules/data-jobs/tests/integration/services/uploads/upload-sessions.integration.spec.js#refuses to complete another tenant’s session, and makes no S3 call for it
INV-DOM-PFM-DJE-003-06
Storage effects occur before their database bookkeeping and outside the database transaction.
servicedocumented
services-platform:modules/data-jobs/services/uploads/upload-sessions.ts (module comment, "Which effect comes first")

The Markdown files under specs/ in rhythmsoftware/rhythm-v2-docs are the canonical source of truth. This reader is a generated, read-only projection of them.

Rhythm Software