aws-amplify / aws-amplify/amplify-hosting
AWS Amplify fails to clone projects with multiple private submodules from GitHub
- Dominant language
- Dockerfile
- Stars
- 481
- Forks
- 123
- PR merge metrics
- No merged PRs in 30d
Description
### Before opening, please confirm:
- [X] I have checked to see if my question is addressed in the [FAQ](https://github.com/aws-amplify/amplify-hosting/blob/master/FAQ.md).
- [X] I have [searched for duplicate or closed issues](https://github.com/aws-amplify/amplify-hosting/issues?q=is%3Aissue+).
- [X] I have read the guide for [submitting bug reports](https://github.com/aws-amplify/amplify-hosting/blob/master/CONTRIBUTING.md).
- [X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
- [X] I have removed any sensitive information from my code snippets and submission.
### App Id
private at this stage - happy to share in private message
### AWS Region
us-west-1
### Amplify Hosting feature
Build settings
### Describe the bug
My project consists of a core application which is stored in a private repository within GitHub. This repository is connected to Amplify.
The project is modular in that the core contains the generic framework and pulls the content in for various sub-projects from other repositories which are included as submodules. These repositories are also private and in the same GitHub account.
Until today I have had the core repository and one submodule. Using a deploy key in the submodule (as per many other bug reports) I have been able to have Amplify clone the repository including the submodule and it has worked as expected. Today, I added a second submodule, including deploy keys etc and this fails to clone. After much testing and troubleshooting, if i remove one of the submodules (it doesn't matter which) the project is able to be cloned and builds as expected. Adding a second submodule causes the project to fail.
The error is "
Failed to clone
ERROR: Repository not found.
fatal: Could not read from remote repository.
Please make sure you have the correct access right and the repository exists"
in my build script in the preBuild phase I grab the Deploy Keys from the env variables, use ssh-agent to add them (this works fine) i then use git submodule update --init --recursive to update the submodules; this succeeds for the first submodule, but fails on each subsequent module
as noted above, the repo exists and the access rights are correct (as I can remove the _other_ submodule without changing anything else and the project succeeds; i can also clone this project on my local PC)
### Expected behavior
It should not matter if there is one or multiple submodules - the project should clone
### Reproduction steps
1. create a project consisting of more than one sub-module
2. create an amplify app and follow the instructions such as https://stackoverflow.com/questions/56219052/fetching-private-submodules-in-aws-amplify-build to add deploy keys and ensure this is done for each submodule
3. attempt to build; it will fail
4. remove a submodule (so there is only one) the build will succeed
### Build Settings
_No response_
### Log output
```
# Put your logs below this line
```
### Additional information
_No response_
Contributor guide
Research direction
Start with the preBuild script and its ssh-agent setup, then reproduce `git submodule update --init --recursive` using two private GitHub submodules. Inspect the Amplify build logs to identify why the first submodule succeeds while later modules fail; done means a build clones and updates all private submodules.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, git, github
- Domain
- ci-cd, cloud, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100