Bound concurrent filesystem reads during asset discovery and source ZIP generation

Open Beginner friendly
#278 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
82/100
Issue type
Refactor
Clarity
Clearly specified
Activity status
Quiet
Tech stack
typescript
Domain
cli, performance

Research direction

Read packages/cli/src/util/AssetUploader.ts and packages/cli/src/util/getAppSourceZip.ts, then inspect the existing mapAsyncWithMaxConcurrency() helper. Limit only the filesystem reads to a concurrency such as 16 while preserving traversal, ZIP contents, hashing, ignore rules, ordering, uploads, and output format.

Written by the indexing model from the issue text.

Description

Summary

The Devvit CLI currently performs some filesystem reads with unbounded concurrency. For projects containing large numbers of assets or source files, this can result in hundreds or thousands of simultaneous readFile() operations.

While this is functionally correct, it can unnecessarily increase memory usage, file descriptor pressure, and disk contention.

Current behavior

Two areas currently read files with effectively unbounded concurrency:

  • packages/cli/src/util/AssetUploader.ts

    • queryAssets() uses Promise.all() to read every asset simultaneously.
  • packages/cli/src/util/getAppSourceZip.ts

    • Source files are added to the ZIP by launching all file reads at once before awaiting them.

Proposed solution

Reuse the existing mapAsyncWithMaxConcurrency() helper to limit concurrent filesystem reads to a reasonable maximum (for example, 16).

This keeps the implementation consistent with the rest of the repository while:

  • reducing peak memory usage
  • reducing file descriptor pressure
  • improving behavior on slower disks and CI runners
  • preserving parallelism

Importantly, this proposal only limits filesystem reads.

It does not change:

  • upload concurrency
  • ZIP contents
  • hashing behavior
  • ignore rules
  • asset ordering
  • output format

Benefits

  • Lower peak resource usage
  • Better scalability for repositories with many files
  • More predictable performance across operating systems
  • Minimal implementation risk

Implementation notes

A small, self-contained change would:

  • replace the Promise.all() asset reads in AssetUploader.ts
  • replace the unbounded ZIP source file reads in getAppSourceZip.ts
  • reuse the existing mapAsyncWithMaxConcurrency() utility
  • keep directory traversal behavior unchanged

This should be a low-risk performance improvement with no observable behavioral changes.

Dominant language
TypeScript
Stars
210
Forks
88
PR merge metrics
No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from reddit/devvit

All issues in reddit/devvit

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.