aws-amplify / aws-amplify/amplify-hosting

Amplify fails to recursively clone git submodules from private repositories

Open
#2,904 39 comments 15 reactions 0 assignees View on GitHub
bug
Dominant language
Dockerfile
Stars
481
Forks
123
PR merge metrics
No merged PRs in 30d

Description

### Before opening, please confirm:

- [X] I have checked to see if my question is addressed in the [FAQ](https://github.com/aws-amplify/amplify-hosting/blob/master/FAQ.md).
- [X] I have [searched for duplicate or closed issues](https://github.com/aws-amplify/amplify-hosting/issues?q=is%3Aissue+).
- [X] I have read the guide for [submitting bug reports](https://github.com/aws-amplify/amplify-hosting/blob/master/CONTRIBUTING.md).
- [X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

### App Id

arn:aws:amplify:us-east-1:312512371189:apps/dftk998dzh5qp

### Region

us-east-1

### Amplify Hosting feature

Backend builds

### Describe the bug

Related to #150 , in that I am also using a git submodule that refers to a private repository, **EXCEPT** in my case the credentials needed to access the submodule repo are the same as those used to access the parent repository (and Amplify has permissions to access both). This should mean that `git submodule update` is able to run, HOWEVER, there is a *bug* in the run order of the Amplify "Cloning repository" stage. The runner *removes* the git credentials immediately after cloning the parent repository and *before* calling `git submodule update` as can be verified in the build log:

```
2022-08-02T16:23:43.388Z [INFO]: Cloning into 'amplify-app'...
2022-08-02T16:23:43.388Z [INFO]: # Switching to commit: ff50e322dcbe6fe5b353069cef010b97bdea9a53
2022-08-02T16:23:43.398Z [INFO]: Note: switching to 'ff50e322dcbe6fe5b353069cef010b97bdea9a53'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:
git switch -c
Or undo this operation with:
git switch -
Turn off this advice by setting config variable advice.detachedHead to false
HEAD is now at ff50e32 use ssh for git submodule refs
2022-08-02T16:23:43.452Z [INFO]: Successfully cleaned up Git credentials
2022-08-02T16:23:43.452Z [INFO]: # Checking for Git submodules at: /codebuild/output/src964993672/src/amplify-app/.gitmodules
2022-08-02T16:23:43.453Z [INFO]: # Updating Git submodules...
2022-08-02T16:23:43.814Z [WARNING]: # Unable to update submodules: Error: Command failed: git submodule update
Cloning into '/codebuild/output/src964993672/src/amplify-app/src'...
Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
```

### Expected behavior

The Amplify CI/CD runner should not clear the git credentials before running `git submodule update`

### Reproduction steps

1. Create a private repository on Github and add a git submodule to it, which references another private Github repository, containing source code for the frontend.
2. Create a new app in amplify and set up CI/CD linked to the parent private Github repo; enable Amplify permissions to access Github repo(s)
3. Commit and allow CI/CD to perform build in the Amplify console.

### Build Settings

```yaml
version: 1
backend:
phases:
build:
commands:
- '# Execute Amplify CLI with the helper script'
- amplifyPush --simple
frontend:
phases:
preBuild:
commands:
- npm ci
build:
commands:
- npm run build
artifacts:
baseDirectory: /dist
files:
- '**/*'
cache:
paths:
- node_modules/**/*
```

### Additional information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the Amplify Hosting "Cloning repository" stage and inspect the ordering shown in the build log around credential cleanup and `git submodule update`. Reproduce the private nested-submodule setup from the listed steps; done means the runner retains usable Git credentials through recursive submodule cloning without the host key or repository-access failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, github
Domain
ci-cd
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.