Rhythm 2.0 Specifications

Appearance

CAP-PFM-DJE-011Upload a bounded file for reliable job input

Markdown

CAP-PFM-DJE-011

Upload a bounded file for reliable job input

Markdown
  • DocumentDraft
  • Product intentConfirmed
  • ImplementationObserved

Planning

Feature
FEAT-PFM-DJE-04
Exercised by
FAS-PFM-DJE-04-01, FAS-PFM-DJE-04-03, FAS-PFM-DJE-04-04
Key concepts
CON-PFM-DJE-003
Actors
ACT-PFM-035, ACT-PFM-036, ACT-PFM-037
Journeys
JNY-PFM-DJE-001, JNY-PFM-DJE-004, JNY-PFM-DJE-005
Related
CAP-PFM-DJE-014, CAP-PFM-DJE-016
Open questions
OQ-PFM-DJE-001
Proved by
ATS-PFM-DJE-011

Implementation

Executable suites
TST-PFM-DJE-164, TST-PFM-DJE-178, TST-PFM-DJE-190, TST-PFM-DJE-193

KindCapabilityProduct areaPlatformModuleData Job EngineRevision1

Read atservices-platform ad3f6a65react-console baf6dae3

Upload a bounded file for reliable job input ​

Purpose ​

Staff and connected systems upload a file directly to storage and retain a completed version for later job processing.

How It Works ​

A client declares the file name, type, size and any digest when opening a session. Rhythm chooses the part size and count. The client obtains signed destinations and sends parts directly to storage. It can inspect received parts and request new addresses when resuming. Completion supplies the receipts for all parts. Rhythm checks the assembled size and, where supported, the declared digest. A size or digest mismatch settles the session as aborted. Missing parts leave the session open so the client can repair its input. Completed uploads identify the stored version used by jobs. The product should decide how large-file verification becomes dependable before migration readiness relies on it.

Relationships ​

  • File processing reads the completed stored version.
  • Migration registration compares package declarations with upload evidence.

The Formal Contract

The narrative above explains; the sections below are authoritative — the rules, scenarios and tests this document binds.

Outcomes
File parts travel directly to storage.Clients can resume from parts already received.Completion checks declared size and supported digest verification.
Surfaces
consoleapi
Intent sources
services-platform:docs/milestones/plans/PLAT-M19-plan.md#A10services-platform:docs/milestones/plans/PLAT-M19-plan.md#S16services-platform:docs/milestones/plans/PLAT-M19-plan.md#S18services-crm:docs/milestones/CRM-M7-data-operations.md#E-16

Controls

39 controls apply, from the corpus control catalogue.

TC-AVL-004 TC-AZ-001 TC-AZ-002 TC-AZ-003 TC-AZ-004 TC-CHG-001 TC-CHG-002 TC-CHG-003 TC-CHG-004 TC-CHG-005 TC-CHG-006 TC-CHG-007 TC-CRY-001 TC-CRY-002 TC-CRY-003 TC-CRY-004 TC-DAT-001 TC-DAT-002 TC-DAT-003 TC-DAT-004 TC-DAT-005 TC-IAM-001 TC-IAM-002 TC-IAM-004 TC-IAM-005 TC-LOG-001 TC-LOG-004 TC-LOG-005 TC-PI-001 TC-PI-002 TC-PI-003 TC-PI-004 TC-PI-005 TC-TEN-001 TC-TEN-002 TC-TEN-003 TC-VEN-001 TC-VEN-002 TC-VEN-003

Permissions

KeyChecked atScope
platform.data-jobs.uploadapitenant

Business Rules

11 rules, 9 to be proved before approval. A rule's product intent is derived from its origin: 11 confirmed by a product decision, the rest inheriting this capability's confirmed.

IdRuleKindEnforced atProvenance
01must prove
Confirmed
Upload operations, including session reads and listings, require upload permission and an authorized workspace in the caller’s security context.
proved by ATS-PFM-DJE-011-01, TST-PFM-DJE-164, TST-PFM-DJE-193
permissionapitest
services-platform:modules/data-jobs/tests/e2e/data-jobs.e2e.spec.js#refuses reading a session, because which files a tenant loaded is not a read-only user’s business
02must prove
Confirmed
An upload declaration specifies a slash-free filename of at most 255 characters, a supported content type and an exact size from one byte to 5 TiB, with an optional SHA-256 digest; invalid declarations create no session.
proved by ATS-PFM-DJE-011-02, TST-PFM-DJE-164, TST-PFM-DJE-178
validationapi, servicetest
services-platform:modules/data-jobs/tests/integration/services/uploads/upload-sessions.integration.spec.js#refuses a declaration before it creates anything in S3, so a refused request leaves no upload behind
03must prove
Confirmed
Part size is at least 16 MiB, otherwise the declared size divided by 9,000 rounded upward to a whole MiB; clients use that size.
proved by ATS-PFM-DJE-011-03, TST-PFM-DJE-190
calculationservicetest
services-platform:modules/data-jobs/tests/unit/part-sizing.spec.ts#rounds a part size the division leaves between whole MiB up to the next whole MiB
04must prove
Confirmed
Clients can request at most 100 signed part addresses at once, each valid for one hour; requests can be repeated only while the session is open.
proved by ATS-PFM-DJE-011-04, TST-PFM-DJE-178
policyservicetest
services-platform:modules/data-jobs/tests/integration/services/uploads/upload-sessions.integration.spec.js#refuses more than a hundred parts at a time, which is the page the protocol presigns
05must prove
Confirmed
Completion supplies every part once with its storage receipt; Rhythm sorts them by part number and refuses a size mismatch by settling the session as aborted.
proved by ATS-PFM-DJE-011-05, ATS-PFM-DJE-011-11, ATS-PFM-DJE-016-14, TST-PFM-DJE-178
validationserviceobserved
services-platform:modules/data-jobs/services/uploads/upload-sessions.ts:requireCompletedParts
06must prove
Confirmed
Declared digests are measured for files up to 256 MiB, or checked against a digest supplied by storage; mismatches abort completion, while larger files without a storage digest retain an unverified declaration.
origin: services-crm:docs/milestones/CRM-M7-data-operations.md#E-16
proved by ATS-PFM-DJE-011-06, ATS-PFM-DJE-011-11, ATS-PFM-DJE-016-14, TST-PFM-DJE-178
validationserviceobserved
services-platform:modules/data-jobs/tests/integration/services/uploads/upload-sessions.integration.spec.js#refuses a completion whose object is not the file the declared SHA-256 describes, naming sha256; services-platform:modules/data-jobs/services/uploads/upload-sessions.ts:verifiedDigestOf
07must prove
Confirmed
Open sessions settle as completed, aborted or expired; completion cannot be aborted, repeated abort is harmless and reads settle storage-abandoned sessions as expired.
proved by ATS-PFM-DJE-011-07, ATS-PFM-DJE-011-10, ATS-PFM-DJE-014-10, TST-PFM-DJE-178
lifecycleservice, databasetest
services-platform:modules/data-jobs/tests/integration/services/uploads/upload-sessions.integration.spec.js#refuses to abort a completed session, whose object a job may already be reading
08must prove
Confirmed
Completion with missing storage parts returns a descriptive refusal and leaves the session open for repair.
origin: services-platform:docs/milestones/plans/PLAT-M19-plan.md#S18
proved by ATS-PFM-DJE-011-08, TST-PFM-DJE-178
policyservicetest
services-platform:modules/data-jobs/tests/integration/services/uploads/upload-sessions.integration.spec.js#refuses in words a completion naming a part S3 does not hold, rather than failing as a server fault
09must prove
Confirmed
Session metadata omits internal storage coordinates; another workspace’s session is indistinguishable from a missing session, while signed part addresses provide the upload destination.
origin: services-platform:docs/milestones/plans/PLAT-M19-plan.md#S11
proved by ATS-PFM-DJE-011-09, ATS-PFM-DJE-011-10, ATS-PFM-DJE-014-10, TST-PFM-DJE-164, TST-PFM-DJE-178
invariantserviceobserved
services-platform:modules/data-jobs/tests/integration/services/uploads/upload-sessions.integration.spec.js#refuses to complete another tenant’s session, and makes no S3 call for it; services-platform:modules/data-jobs/services/uploads/upload-sessions.ts:presignUploadParts
10
Confirmed
Storage lifecycle policies retain ordinary uploads for 30 days unless tagged to keep, chunks for 14 days and exports for seven days, including stored versions, and abandon incomplete parts after two days.
lifecycleservicedocumented
services-platform:docs/data-jobs/uploads.md#Retention
11
Confirmed
Workspace deletion removes upload records and stored file versions, while reset preserves uploaded input.
proved by ATS-PFM-DJE-011-12, ATS-PFM-DJE-017-10
lifecycleservicedocumented
services-platform:docs/data-jobs/uploads.md#Deletion

Scenarios

IdGivenWhenThenProvenance
01A producer has a file containing exactly 200 GiB.It requests an upload session.The part-sizing calculation gives 8,534 parts at 24 MiB, which the client can upload directly to storage.inferred
services-platform:modules/data-jobs/db/data-job-uploads.table.ts:partSizeOf

The Markdown files under specs/ in rhythmsoftware/rhythm-v2-docs are the canonical source of truth. This reader is a generated, read-only projection of them.

Rhythm Software