Limitation of aio-files
- Dominant language
- JavaScript
- Stars
- 13
- Forks
- 10
- PR merge metrics
- No merged PRs in 30d
Description
**Hi Team,**
I have a requirement to generate a CSV file from stock data.
The processed stock data is stored in Adobe I/O Files in the following structure:
`inv/logs/stock///.json``
My goal is to read all files for a specific date and generate a single CSV.
Problem
When I list files for a given date folder using @adobe/aio-lib-files, I only receive 1000 files. There doesn’t appear to be any pagination mechanism available for listing beyond this limit.
Constraints
I cannot delete the original files, as they need to be retained.
Earlier, when the number of files was small, this approach worked fine.
Now that the file count exceeds 1000+, I’m unable to process all files.
Attempted Approach
I tried copying the date folder into a temporary folder, then reading and deleting files from the temp folder in batches. However, the copy() operation also handles only 1000 files at a time, so the same limitation applies.
**Question**
What is the recommended approach to:
Read and process more than 1000 files from Adobe I/O Files for a given directory/date?
Any guidance or best practices would be greatly appreciated.
Thank you so much for your attention and participation.
Contributor guide
Research direction
Review the @adobe/aio-lib-files file-listing and copy operations described in the issue, focusing on their 1,000-file behavior. Determine whether the existing APIs expose a supported way to process all files for a date and document or define the expected handling for collections larger than 1,000 files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- backend, cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100