aws-amplify / aws-amplify/amplify-hosting

AWS Amplify fails to clone projects with multiple private submodules from GitHub

Open
#3,252 0 comments 0 reactions 0 assignees View on GitHub
bug frontend-builds investigating
Dominant language
Dockerfile
Stars
481
Forks
123
PR merge metrics
No merged PRs in 30d

Description

### Before opening, please confirm:

- [X] I have checked to see if my question is addressed in the [FAQ](https://github.com/aws-amplify/amplify-hosting/blob/master/FAQ.md).
- [X] I have [searched for duplicate or closed issues](https://github.com/aws-amplify/amplify-hosting/issues?q=is%3Aissue+).
- [X] I have read the guide for [submitting bug reports](https://github.com/aws-amplify/amplify-hosting/blob/master/CONTRIBUTING.md).
- [X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
- [X] I have removed any sensitive information from my code snippets and submission.

### App Id

private at this stage - happy to share in private message

### AWS Region

us-west-1

### Amplify Hosting feature

Build settings

### Describe the bug

My project consists of a core application which is stored in a private repository within GitHub. This repository is connected to Amplify.

The project is modular in that the core contains the generic framework and pulls the content in for various sub-projects from other repositories which are included as submodules. These repositories are also private and in the same GitHub account.

Until today I have had the core repository and one submodule. Using a deploy key in the submodule (as per many other bug reports) I have been able to have Amplify clone the repository including the submodule and it has worked as expected. Today, I added a second submodule, including deploy keys etc and this fails to clone. After much testing and troubleshooting, if i remove one of the submodules (it doesn't matter which) the project is able to be cloned and builds as expected. Adding a second submodule causes the project to fail.

The error is "
Failed to clone
ERROR: Repository not found.
fatal: Could not read from remote repository.
Please make sure you have the correct access right and the repository exists"

in my build script in the preBuild phase I grab the Deploy Keys from the env variables, use ssh-agent to add them (this works fine) i then use git submodule update --init --recursive to update the submodules; this succeeds for the first submodule, but fails on each subsequent module

as noted above, the repo exists and the access rights are correct (as I can remove the _other_ submodule without changing anything else and the project succeeds; i can also clone this project on my local PC)

### Expected behavior

It should not matter if there is one or multiple submodules - the project should clone

### Reproduction steps

1. create a project consisting of more than one sub-module
2. create an amplify app and follow the instructions such as https://stackoverflow.com/questions/56219052/fetching-private-submodules-in-aws-amplify-build to add deploy keys and ensure this is done for each submodule
3. attempt to build; it will fail
4. remove a submodule (so there is only one) the build will succeed

### Build Settings

_No response_

### Log output

```
# Put your logs below this line

```

### Additional information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the preBuild script and its ssh-agent setup, then reproduce `git submodule update --init --recursive` using two private GitHub submodules. Inspect the Amplify build logs to identify why the first submodule succeeds while later modules fail; done means a build clones and updates all private submodules.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, git, github
Domain
ci-cd, cloud, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.