adobe / adobe/aio-lib-files

Limitation of aio-files

Open
#182 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
13
Forks
10
PR merge metrics
No merged PRs in 30d

Description

**Hi Team,**
I have a requirement to generate a CSV file from stock data.
The processed stock data is stored in Adobe I/O Files in the following structure:
`inv/logs/stock///.json``

My goal is to read all files for a specific date and generate a single CSV.
Problem
When I list files for a given date folder using @adobe/aio-lib-files, I only receive 1000 files. There doesn’t appear to be any pagination mechanism available for listing beyond this limit.
Constraints
I cannot delete the original files, as they need to be retained.
Earlier, when the number of files was small, this approach worked fine.
Now that the file count exceeds 1000+, I’m unable to process all files.

Attempted Approach

I tried copying the date folder into a temporary folder, then reading and deleting files from the temp folder in batches. However, the copy() operation also handles only 1000 files at a time, so the same limitation applies.

**Question**
What is the recommended approach to:
Read and process more than 1000 files from Adobe I/O Files for a given directory/date?
Any guidance or best practices would be greatly appreciated.

Thank you so much for your attention and participation.

Contributor guide

Open the contributing guide

Research direction

Review the @adobe/aio-lib-files file-listing and copy operations described in the issue, focusing on their 1,000-file behavior. Determine whether the existing APIs expose a supported way to process all files for a date and document or define the expected handling for collections larger than 1,000 files.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
backend, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.