All posts

Fast parallel S3 downloads in Node.js with s3-multipartdownload

A single GetObject uses one connection, and one connection rarely fills your bandwidth. Here's how my open-source package splits large S3 objects into byte ranges, downloads them side by side, and streams them straight to disk.

  • Open source
  • Node.js
  • AWS S3
  • Performance

Benchmark · 500 MB object

14.3 s 2.0 s

faster
7.1×
throughput
249 MB/s
peak memory
~130 MB
  1. Single GetObject1 connection14.3 s
  2. Parallel ×44 byte ranges4.5 s
  3. Parallel ×88 byte ranges2.5 s
  4. Parallel ×1616 byte ranges2.0 s
Replayed at 2× speed · median of 3 runs on a simulated S3 server · full results

TL;DR

  • Problem: AWS SDK v3 for JavaScript has a managed uploader but no parallel downloader, so a large GetObject crawls over one connection.
  • Fix: s3-multipartdownload fetches byte ranges concurrently and writes each one at its offset in the output file.
  • Result: a 500 MB object went from 14.3 s to 2.0 s in my benchmark (7.1×), with memory flat at about 130 MB.

Why one GetObject is slow

When you download an object with the AWS SDK, the whole thing comes back over one HTTP response on one TCP connection. Each connection to S3 has a throughput ceiling well below what a modern machine or EC2 instance can pull, so most of your bandwidth sits idle.

AWS's own S3 performance guidelines describe the answer: request byte ranges of the object with the Range header and fetch them concurrently. The AWS CLI, and the transfer managers in the Java and Python SDKs, do this for you. In Node.js, @aws-sdk/lib-storage handles parallel uploads, but there is no equivalent for downloads.

I first wrote s3-multipartdownload in 2021 to close that gap. Version 2.0 is a full rewrite on AWS SDK v3.

How it works

The idea is simple: ask S3 how big the object is, carve it into fixed-size ranges, and run a small pool of workers that each fetch a range and write it into place.

How s3-multipartdownload works The app calls download. A HeadObject request returns the size and ETag. The object is split into 8 MiB ranges, and up to parallel_count GetObject range requests run at once, each pinned to the ETag. Each range streams into its offset of a pre-sized temp file, which is renamed to the final file when every part is done. HeadObject size + ETag Pre-size temp file file.download Range 0 – 8 MiB Range 8 – 16 MiB Range 16 – 24 MiB … queued up to parallel_count Each range streams to its own offset If-Match: ETag on every part · retried on failure Rename → final file
One HeadObject, then a bounded pool of ranged GETs writing straight into a pre-sized file.
  1. HeadObjectReads the object's size and ETag.
  2. Pre-size a temp fileCreating file.download at full size lets every part write at its own offset. No part files, no merge step.
  3. Fetch ranges with a worker poolAt most parallel_count requests run at once. Each sends If-Match: <ETag>, so if the object is overwritten mid-download, the download fails instead of mixing two versions.
  4. Retry, or cancel cleanlyA part that fails, even halfway through its stream, is retried with backoff. A fatal error cancels in-flight parts and deletes the temp file.
  5. RenameOnly when every byte has landed does the temp file become the real one.

Install and quick start

You need Node.js 20.3 or later. The AWS SDK is a peer dependency, so you keep control of its version and your client configuration.

npm install s3-multipartdownload @aws-sdk/client-s3
const { S3Client } = require('@aws-sdk/client-s3')
const { download } = require('s3-multipartdownload')

// Credentials come from the default provider chain (env, ~/.aws, SSO, IAM role)
const s3 = new S3Client({ region: 'us-east-1' })

const file = await download(s3, { Bucket: 'my-bucket', Key: 'datasets/big-file.parquet' }, {
  download_dir: './downloads',
  parallel_count: 8
})

console.log('Saved to', file) // ./downloads/big-file.parquet

TypeScript types ship with the package, and ES module import works too.

Show progress

await download(s3, params, {
  onProgress: ({ downloadedBytes, totalBytes }) => {
    const pct = totalBytes ? (downloadedBytes / totalBytes) * 100 : 100
    process.stdout.write(`\r${pct.toFixed(1)}%`)
  }
})

Cancel or time out

// Give up after 60 seconds; the partial temp file is removed
await download(s3, params, { abortSignal: AbortSignal.timeout(60_000) })

Versions, encryption and requester pays

Any HeadObject or GetObject parameter passes straight through, including VersionId, SSECustomerKey and RequestPayer.

await download(s3, {
  Bucket: 'my-bucket',
  Key: 'secret.bin',
  VersionId: '3HL4kqtJlcpXroDTDmJ',
  SSECustomerAlgorithm: 'AES256',
  SSECustomerKey: process.env.SSE_KEY
}, { download_dir: './out', file_name: 'secret-v3.bin' })

MinIO, Cloudflare R2 and other S3-compatible storage

const s3 = new S3Client({ endpoint: 'https://<account>.r2.cloudflarestorage.com', region: 'auto' })
await download(s3, { Bucket: 'assets', Key: 'build.tar.gz' })

Benchmarks: how much faster?

I compared the standard SDK v3 approach (one GetObject piped to a file) with s3-multipartdownload at 4, 8 and 16 parallel ranges. Each strategy ran three times in its own process, and every downloaded file's SHA-256 matched the original.

Median of 3 runs on a simulated S3 server. Full results are in the repo.
Strategy100 MB500 MBPeak memory
Single GetObject2.91 s14.29 s134 MB
s3-multipartdownload ×41.01 s4.48 s133 MB
s3-multipartdownload ×80.55 s2.49 s130 MB
s3-multipartdownload ×160.41 s2.01 s127 MB

An honest note on the setup. These numbers come from a local S3-compatible server, not live Amazon S3. It caps each connection at 40 MB/s, total bandwidth at 400 MB/s, and adds 20 ms of latency per request, which models why parallelism helps. It ran on an Apple M1 with Node 22. At 16 parts the run levels off near 250 MB/s because the server and the downloader share one machine's CPU.

Your gains will depend on your network, region and instance size. The benchmark script runs against a real bucket too:

node bench/benchmark.js --bucket my-bucket --key path/to/big.bin --region us-east-1

Speed went up 7×. Memory didn't move. Streaming each range to disk means a 50 GB object costs no more RAM than a 50 MB one.

Choosing parallel_count and part_size

  • Laptops and small instances: parallel_count 4–8.
  • EC2 in the same region as the bucket: start at 16–32 and raise it until throughput stops improving.
  • Part size: keep the default 8 MiB, or go up to 16 MiB. Smaller parts add request overhead, and each part is a billed GET. Larger parts make each retry slower.

Cost is rarely a concern. A 1 GB object at 8 MiB parts is 128 GET requests, a fraction of a cent at standard S3 pricing, and you transfer the same bytes either way.

What's new in 2.0

Version 1 was a quick wrapper on the now end-of-support AWS SDK v2. It buffered each part in memory, wrote part files, then merged them. The 2.0 rewrite fixes that design and several bugs:

  • AWS SDK v3 with S3Client and commands, as a peer dependency.
  • Streaming writes at file offsets replace part files and the merge step.
  • Bounded concurrency over fixed-size parts, instead of exactly parallel_count giant parts.
  • Reliability: per-part retries, ETag pinning, atomic rename and clean-up on failure.
  • Bug fixes: a crash when the download folder didn't exist, an infinite loop on objects smaller than parallel_count bytes, and the library no longer modifies your params object.
  • New options: part_size, retries, file_name, abortSignal and onProgress, plus TypeScript types.

Upgrading from 1.x means passing an SDK v3 S3Client instead of an AWS.S3 instance. The promise now resolves to the downloaded file's path.

FAQ

Is this the same as S3 multipart upload?

No. Multipart upload is an S3 API. A multipart download is a client-side technique: several HTTP Range requests against an ordinary GetObject. It works on any object, however it was uploaded.

How is it different from aws s3 cp?

The AWS CLI also downloads in parallel, and it's the right tool for shell scripts. This package brings the same technique inside a Node.js app, Lambda function or worker, with your own client, progress callbacks and cancellation.

Can I download to a stream or a buffer?

Not yet. Parts arrive out of order, so writing to known file offsets is what keeps memory flat. If you need it, open an issue.

Try it

If you work on platform and data infrastructure, you might also like how we migrated a 17-year-old WebLogic system to microservices with zero downtime.