How data ingestion works
Direct-to-cloud uploads with end-to-end integrity.
Files go from your browser straight to your tenant's isolated S3
prefix — never through a shared application server. From there an
async worker validates, transforms, and merges the data with strong
guarantees against partial writes or silent corruption.
Step 1
Presigned upload
The browser receives a short-lived (1-hour) presigned policy with a
200 MB hard cap enforced at the S3 edge — bytes never traverse our
application servers.
Step 2
Tenant-scoped lock
A DynamoDB lock prevents two concurrent uploads of the same data
source from racing. Conflicts return immediately rather than
corrupting the merged dataset.
Step 3
Idempotent worker
Async processing with conditional state transitions — duplicate
invocations are no-ops. Schema validation, type coercion, and
per-column data-quality warnings run before any data is committed.
Step 4
Recovery built in
A scheduled sweeper detects stuck jobs and a dead-letter queue
captures permanent failures. Abandoned uploads expire automatically
via S3 lifecycle rules — no orphaned data, ever.