aws / aws/aws-toolkit-azure-devops
S3Download fails unless flattenFolders is used
- Dominant language
- TypeScript
- Stars
- 258
- Forks
- 114
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the bug**
I have a bucket that we use for cache build files. We are currently using ccache so the folder structure has a tree, which we need to keep in place in the agent.
If we use the default values, we get an issue and the process fails, unless we use flattenFolders, which messes up the tree structure we need for ccache.
**To reproduce**
```
- task: S3Download@1
displayName: Cache Download
inputs:
awsCredentials: 'AWS DevOps'
regionName: 'us-east-1'
sourceFolder: ccache
bucketName: 'build-cache'
globExpressions: '**'
#flattenFolders: true
targetFolder: $(Agent.TempDirectory)
```
Structure in the bucket is
/ccache/app/ios/0/2/38428937428947 (example cache file)
**Expected behavior**
Task downloads S3 bucket respecting tree structure.
**Screenshots**
This is the issue we get right after the download queue:
```
Queueing download of ccache/app/ios/f/f/eak4snpi3jd2838irr7etbr759b5rbm1W
Queueing download of ccache/app/ios/f/f/eak4snpi3jd2838irr7etbr759b5rbmR
Queueing download of ccache/app/ios/f/f/edija9200gj5mr373ijd1rusuqa7se4M
Queueing download of ccache/app/ios/f/f/f32dt44pm56o3hjujjtm3ato9vdsi920W
Queueing download of ccache/app/ios/f/f/f32dt44pm56o3hjujjtm3ato9vdsi92R
Queueing download of ccache/app/ios/f/stats
Queueing download of ccache/app/ios/tmp/
Queueing download of ccache/app/ios/tmp/inode-cache.v1
##[error]Error: ENOENT: no such file or directory, open '/Users/runner/work/_temp/ccache/'
```
**Your Environment**
- Cloud agent macos
- 1.13.0 toolkit
**Additional context**
We tried setting up different folders, assuming it could be a permission issue but the fact that the flattenFolders does something turns out it is not. At the same time, with flattenFolders we get that some files are being found and overridden in subsequent runs. Most likely the issue is that we are getting the same agent again and the temp folder is not clean. Not a problem for us, since it is overridden, but it is indeed saving files so it should not be a problem with the permissions.
Most likely issue is that it is not correctly creating all the intermediate folders for download.
Contributor guide
Research direction
Start at the S3Download@1 download queue and reproduce the configuration shown against a bucket containing nested ccache paths. Trace the download destination handling, especially intermediate directory creation; done means the task downloads every object while preserving the bucket tree under the target folder without ENOENT errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- cloud, devops
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100