aws-amplify / aws-amplify/amplify-hosting
Amplify fails to recursively clone git submodules from private repositories
- Dominant language
- Dockerfile
- Stars
- 481
- Forks
- 123
- PR merge metrics
- No merged PRs in 30d
Description
### Before opening, please confirm:
- [X] I have checked to see if my question is addressed in the [FAQ](https://github.com/aws-amplify/amplify-hosting/blob/master/FAQ.md).
- [X] I have [searched for duplicate or closed issues](https://github.com/aws-amplify/amplify-hosting/issues?q=is%3Aissue+).
- [X] I have read the guide for [submitting bug reports](https://github.com/aws-amplify/amplify-hosting/blob/master/CONTRIBUTING.md).
- [X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
### App Id
arn:aws:amplify:us-east-1:312512371189:apps/dftk998dzh5qp
### Region
us-east-1
### Amplify Hosting feature
Backend builds
### Describe the bug
Related to #150 , in that I am also using a git submodule that refers to a private repository, **EXCEPT** in my case the credentials needed to access the submodule repo are the same as those used to access the parent repository (and Amplify has permissions to access both). This should mean that `git submodule update` is able to run, HOWEVER, there is a *bug* in the run order of the Amplify "Cloning repository" stage. The runner *removes* the git credentials immediately after cloning the parent repository and *before* calling `git submodule update` as can be verified in the build log:
```
2022-08-02T16:23:43.388Z [INFO]: Cloning into 'amplify-app'...
2022-08-02T16:23:43.388Z [INFO]: # Switching to commit: ff50e322dcbe6fe5b353069cef010b97bdea9a53
2022-08-02T16:23:43.398Z [INFO]: Note: switching to 'ff50e322dcbe6fe5b353069cef010b97bdea9a53'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:
git switch -c
Or undo this operation with:
git switch -
Turn off this advice by setting config variable advice.detachedHead to false
HEAD is now at ff50e32 use ssh for git submodule refs
2022-08-02T16:23:43.452Z [INFO]: Successfully cleaned up Git credentials
2022-08-02T16:23:43.452Z [INFO]: # Checking for Git submodules at: /codebuild/output/src964993672/src/amplify-app/.gitmodules
2022-08-02T16:23:43.453Z [INFO]: # Updating Git submodules...
2022-08-02T16:23:43.814Z [WARNING]: # Unable to update submodules: Error: Command failed: git submodule update
Cloning into '/codebuild/output/src964993672/src/amplify-app/src'...
Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
```
### Expected behavior
The Amplify CI/CD runner should not clear the git credentials before running `git submodule update`
### Reproduction steps
1. Create a private repository on Github and add a git submodule to it, which references another private Github repository, containing source code for the frontend.
2. Create a new app in amplify and set up CI/CD linked to the parent private Github repo; enable Amplify permissions to access Github repo(s)
3. Commit and allow CI/CD to perform build in the Amplify console.
### Build Settings
```yaml
version: 1
backend:
phases:
build:
commands:
- '# Execute Amplify CLI with the helper script'
- amplifyPush --simple
frontend:
phases:
preBuild:
commands:
- npm ci
build:
commands:
- npm run build
artifacts:
baseDirectory: /dist
files:
- '**/*'
cache:
paths:
- node_modules/**/*
```
### Additional information
_No response_
Contributor guide
Research direction
Start with the Amplify Hosting "Cloning repository" stage and inspect the ordering shown in the build log around credential cleanup and `git submodule update`. Reproduce the private nested-submodule setup from the listed steps; done means the runner retains usable Git credentials through recursive submodule cloning without the host key or repository-access failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- git, github
- Domain
- ci-cd
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100