Fast parallel S3 downloads in Node.js with s3-multipartdownload
A single GetObject uses one connection, and one connection rarely fills your bandwidth. Here's how my open-source package splits large S3 objects into byte ranges, downloads them side by side, and streams them straight to disk.
Benchmark · 500 MB object
14.3 s 2.0 s
- faster
- 7.1×
- throughput
- 249 MB/s
- peak memory
- ~130 MB
- Single GetObject1 connection14.3 s
- Parallel ×44 byte ranges4.5 s
- Parallel ×88 byte ranges2.5 s
- Parallel ×1616 byte ranges2.0 s
TL;DR
- Problem: AWS SDK v3 for JavaScript has a managed uploader but no parallel downloader, so a large
GetObjectcrawls over one connection. - Fix: s3-multipartdownload fetches byte ranges concurrently and writes each one at its offset in the output file.
- Result: a 500 MB object went from 14.3 s to 2.0 s in my benchmark (7.1×), with memory flat at about 130 MB.
Why one GetObject is slow
When you download an object with the AWS SDK, the whole thing comes back over one HTTP response on one TCP connection. Each connection to S3 has a throughput ceiling well below what a modern machine or EC2 instance can pull, so most of your bandwidth sits idle.
AWS's own S3 performance guidelines describe the answer: request byte ranges of the object with the Range header and fetch them concurrently. The AWS CLI, and the transfer managers in the Java and Python SDKs, do this for you. In Node.js, @aws-sdk/lib-storage handles parallel uploads, but there is no equivalent for downloads.
I first wrote s3-multipartdownload in 2021 to close that gap. Version 2.0 is a full rewrite on AWS SDK v3.
How it works
The idea is simple: ask S3 how big the object is, carve it into fixed-size ranges, and run a small pool of workers that each fetch a range and write it into place.
- HeadObjectReads the object's size and ETag.
- Pre-size a temp fileCreating
file.downloadat full size lets every part write at its own offset. No part files, no merge step. - Fetch ranges with a worker poolAt most
parallel_countrequests run at once. Each sendsIf-Match: <ETag>, so if the object is overwritten mid-download, the download fails instead of mixing two versions. - Retry, or cancel cleanlyA part that fails, even halfway through its stream, is retried with backoff. A fatal error cancels in-flight parts and deletes the temp file.
- RenameOnly when every byte has landed does the temp file become the real one.
Install and quick start
You need Node.js 20.3 or later. The AWS SDK is a peer dependency, so you keep control of its version and your client configuration.
npm install s3-multipartdownload @aws-sdk/client-s3
const { S3Client } = require('@aws-sdk/client-s3')
const { download } = require('s3-multipartdownload')
// Credentials come from the default provider chain (env, ~/.aws, SSO, IAM role)
const s3 = new S3Client({ region: 'us-east-1' })
const file = await download(s3, { Bucket: 'my-bucket', Key: 'datasets/big-file.parquet' }, {
download_dir: './downloads',
parallel_count: 8
})
console.log('Saved to', file) // ./downloads/big-file.parquet
TypeScript types ship with the package, and ES module import works too.
Show progress
await download(s3, params, {
onProgress: ({ downloadedBytes, totalBytes }) => {
const pct = totalBytes ? (downloadedBytes / totalBytes) * 100 : 100
process.stdout.write(`\r${pct.toFixed(1)}%`)
}
})
Cancel or time out
// Give up after 60 seconds; the partial temp file is removed
await download(s3, params, { abortSignal: AbortSignal.timeout(60_000) })
Versions, encryption and requester pays
Any HeadObject or GetObject parameter passes straight through, including VersionId, SSECustomerKey and RequestPayer.
await download(s3, {
Bucket: 'my-bucket',
Key: 'secret.bin',
VersionId: '3HL4kqtJlcpXroDTDmJ',
SSECustomerAlgorithm: 'AES256',
SSECustomerKey: process.env.SSE_KEY
}, { download_dir: './out', file_name: 'secret-v3.bin' })
MinIO, Cloudflare R2 and other S3-compatible storage
const s3 = new S3Client({ endpoint: 'https://<account>.r2.cloudflarestorage.com', region: 'auto' })
await download(s3, { Bucket: 'assets', Key: 'build.tar.gz' })
Benchmarks: how much faster?
I compared the standard SDK v3 approach (one GetObject piped to a file) with s3-multipartdownload at 4, 8 and 16 parallel ranges. Each strategy ran three times in its own process, and every downloaded file's SHA-256 matched the original.
Throughput on a 500 MB object MB/s · higher is better
| Strategy | 100 MB | 500 MB | Peak memory |
|---|---|---|---|
Single GetObject | 2.91 s | 14.29 s | 134 MB |
| s3-multipartdownload ×4 | 1.01 s | 4.48 s | 133 MB |
| s3-multipartdownload ×8 | 0.55 s | 2.49 s | 130 MB |
| s3-multipartdownload ×16 | 0.41 s | 2.01 s | 127 MB |
An honest note on the setup. These numbers come from a local S3-compatible server, not live Amazon S3. It caps each connection at 40 MB/s, total bandwidth at 400 MB/s, and adds 20 ms of latency per request, which models why parallelism helps. It ran on an Apple M1 with Node 22. At 16 parts the run levels off near 250 MB/s because the server and the downloader share one machine's CPU.
Your gains will depend on your network, region and instance size. The benchmark script runs against a real bucket too:
node bench/benchmark.js --bucket my-bucket --key path/to/big.bin --region us-east-1
Speed went up 7×. Memory didn't move. Streaming each range to disk means a 50 GB object costs no more RAM than a 50 MB one.
Choosing parallel_count and part_size
- Laptops and small instances:
parallel_count4–8. - EC2 in the same region as the bucket: start at 16–32 and raise it until throughput stops improving.
- Part size: keep the default 8 MiB, or go up to 16 MiB. Smaller parts add request overhead, and each part is a billed GET. Larger parts make each retry slower.
Cost is rarely a concern. A 1 GB object at 8 MiB parts is 128 GET requests, a fraction of a cent at standard S3 pricing, and you transfer the same bytes either way.
What's new in 2.0
Version 1 was a quick wrapper on the now end-of-support AWS SDK v2. It buffered each part in memory, wrote part files, then merged them. The 2.0 rewrite fixes that design and several bugs:
- AWS SDK v3 with
S3Clientand commands, as a peer dependency. - Streaming writes at file offsets replace part files and the merge step.
- Bounded concurrency over fixed-size parts, instead of exactly
parallel_countgiant parts. - Reliability: per-part retries, ETag pinning, atomic rename and clean-up on failure.
- Bug fixes: a crash when the download folder didn't exist, an infinite loop on objects smaller than
parallel_countbytes, and the library no longer modifies your params object. - New options:
part_size,retries,file_name,abortSignalandonProgress, plus TypeScript types.
Upgrading from 1.x means passing an SDK v3 S3Client instead of an AWS.S3 instance. The promise now resolves to the downloaded file's path.
FAQ
Is this the same as S3 multipart upload?
No. Multipart upload is an S3 API. A multipart download is a client-side technique: several HTTP Range requests against an ordinary GetObject. It works on any object, however it was uploaded.
How is it different from aws s3 cp?
The AWS CLI also downloads in parallel, and it's the right tool for shell scripts. This package brings the same technique inside a Node.js app, Lambda function or worker, with your own client, progress callbacks and cancellation.
Can I download to a stream or a buffer?
Not yet. Parts arrive out of order, so writing to known file offsets is what keeps memory flat. If you need it, open an issue.
Try it
- npm: s3-multipartdownload
- GitHub: ranjithkumarmv/s3-multipartdownload (MIT licensed; issues and pull requests welcome)
- Background reading: AWS S3 performance guidelines and the GetObject API reference
If you work on platform and data infrastructure, you might also like how we migrated a 17-year-old WebLogic system to microservices with zero downtime.