Difficulties with private git materials containing submodules

### Issue Type
- Bug Report

### Environment

GoCD Server: GoCD Version: 20.4.0 (11749-5553066d29d315951efa5ead0c087374df6338b9).
Server JVM version: | 13.0.2
Server OS Information: | Windows 10 10.0

Agent: Windows .zip deployed agent, version 2020.6.0
Git: git version 2.28.0.windows.1
Agent Java:
```
java --version
openjdk 11.0.8 2020-07-14
OpenJDK Runtime Environment 18.9 (build 11.0.8+10)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.8+10, mixed mode)
```

### Summary

We've had pipelines running fairly seamlessly lately, but recently added a submodule and I've been having a lot of difficulties, and inconsistent results trying to get it working.

Console output looks like:
```
[go] Start updating PipelineRepo at revision 7ace640aa345c84a97909faee44208060981ad26 from https://dev.azure.com/######/_git/######
STDERR: Cloning into 'D:\Agents\go-agent-20.6.0-unity2\pipelines\\PipelineRepo'...
STDERR: warning: current Git remote contains credentials
STDERR: Updating files: 88% (1857/2096)
STDERR: Updating files: 100% (2096/2096), done.
STDERR: warning: current Git remote contains credentials
STDERR: Filtering content: 2% (5/234)
STDERR: Filtering content: 100% (234/234), 47.59 MiB | 10.49 MiB/s, done.
[GIT] Fetching changes
[GIT] Performing git gc
[GIT] Reset working directory pipelines\\PipelineRepo
[GIT] Cleaning all unversioned files in working copy
[GIT] Cleaning submodule configurations in .git/config
[GIT] Updating working copy to revision 7ace640aa345c84a97909faee44208060981ad26
HEAD is now at 7ace640 Removing old unsued code
[GIT] Removing modified files in submodules
[GIT] Updating git sub-modules
Failed to update submodule
```

Log file output with *git* trace rule

```
2020-08-13 18:27:57,112 INFO [scheduler-1] HttpService:125 - Got back 200 from server
2020-08-13 18:27:57,113 INFO [scheduler-1] DefaultGoPublisher:88 - Agent [Sulaco, 192.168.20.93, 46f7c830-d275-4b67-be1b-db56c75392fc] is reporting status [Preparing] to Go Server for Build [/34/LOD1/3/LOD1/6122]
2020-08-13 18:27:57,119 DEBUG [scheduler-1] GitMaterial:218 - Invalid git working copy or repository changed. Delete folder: pipelines\\PipelineRepo
2020-08-13 18:28:05,901 DEBUG [scheduler-1] GitCommand:515 - Fetching changes
2020-08-13 18:28:06,338 DEBUG [scheduler-1] GitCommand:515 - Performing git gc
2020-08-13 18:28:06,397 DEBUG [scheduler-1] GitCommand:515 - Reset working directory pipelines\\PipelineRepo
2020-08-13 18:28:06,397 DEBUG [scheduler-1] GitCommand:515 - Cleaning all unversioned files in working copy
2020-08-13 18:28:06,822 DEBUG [scheduler-1] GitCommand:515 - Cleaning submodule configurations in .git/config
2020-08-13 18:28:06,850 DEBUG [scheduler-1] GitCommand:515 - Updating working copy to revision 7ace640aa345c84a97909faee44208060981ad26
2020-08-13 18:28:07,125 INFO [pool-18-thread-1] HttpService:125 - Got back 200 from server
2020-08-13 18:28:07,281 DEBUG [scheduler-1] GitCommand:515 - Removing modified files in submodules
2020-08-13 18:28:07,662 DEBUG [scheduler-1] GitCommand:515 - Updating git sub-modules
2020-08-13 18:28:10,657 WARN [scheduler-1] GitCommand:259 - Error performing command: --- Command ---
git submodule update --depth=1
--- Environment ---
{}
--- INPUT ----

--- EXIT CODE (1) ---
--- STANDARD OUT ---

--- STANDARD ERR ---
STDERR: Cloning into 'D:/Agents/go-agent-20.6.0-unity2/pipelines//PipelineRepo/PipelineConfiguration'...
STDERR: Logon failed, use ctrl+c to cancel basic credential prompt.
STDERR: bash: /dev/tty: No such device or address
STDERR: error: failed to execute prompt script (exit code 1)
STDERR: fatal: could not read Username for 'https://dev.azure.com': No such file or directory
STDERR: fatal: clone of 'https://dev.azure.com/#####/_git/TheSubmoduleRepo' into submodule path 'D:/Agents/go-agent-20.6.0-unity2/pipelines//PipelineRepo/PipelineConfiguration' failed
STDERR: Failed to clone 'PipelineConfiguration'. Retry scheduled
STDERR: Cloning into 'D:/Agents/go-agent-20.6.0-unity2/pipelines//PipelineRepo/PipelineConfiguration'...
STDERR: Logon failed, use ctrl+c to cancel basic credential prompt.
STDERR: bash: /dev/tty: No such device or address
STDERR: error: failed to execute prompt script (exit code 1)
STDERR: fatal: could not read Username for 'https://dev.azure.com': No such file or directory
STDERR: fatal: clone of 'https://dev.azure.com/#####/_git/TheSubmoduleRepo' into submodule path 'D:/Agents/go-agent-20.6.0-unity2/pipelines//PipelineRepo/PipelineConfiguration' failed
STDERR: Failed to clone 'PipelineConfiguration' a second time, aborting
---

com.thoughtworks.go.util.command.CommandLineException: Error performing command: --- Command ---
git submodule update --depth=1
--- Environment ---
{}
--- INPUT ----

--- EXIT CODE (1) ---
--- STANDARD OUT ---

--- STANDARD ERR ---
STDERR: Cloning into 'D:/Agents/go-agent-20.6.0-unity2/pipelines//PipelineRepo/PipelineConfiguration'...
STDERR: Logon failed, use ctrl+c to cancel basic credential prompt.
STDERR: bash: /dev/tty: No such device or address
STDERR: error: failed to execute prompt script (exit code 1)
STDERR: fatal: could not read Username for 'https://dev.azure.com': No such file or directory
STDERR: fatal: clone of 'https://dev.azure.com/#####/_git/TheSubmoduleRepo' into submodule path 'D:/Agents/go-agent-20.6.0-unity2/pipelines//PipelineRepo/PipelineConfiguration' failed
STDERR: Failed to clone 'PipelineConfiguration'. Retry scheduled
STDERR: Cloning into 'D:/Agents/go-agent-20.6.0-unity2/pipelines//PipelineRepo/PipelineConfiguration'...
STDERR: Logon failed, use ctrl+c to cancel basic credential prompt.
STDERR: bash: /dev/tty: No such device or address
STDERR: error: failed to execute prompt script (exit code 1)
STDERR: fatal: could not read Username for 'https://dev.azure.com': No such file or directory
STDERR: fatal: clone of 'https://dev.azure.com/#####/_git/TheSubmoduleRepo' into submodule path 'D:/Agents/go-agent-20.6.0-unity2/pipelines//PipelineRepo/PipelineConfiguration' failed
STDERR: Failed to clone 'PipelineConfiguration' a second time, aborting
---

at com.thoughtworks.go.util.command.CommandLine.runOrBomb(CommandLine.java:413)
at com.thoughtworks.go.util.command.CommandLine.runOrBomb(CommandLine.java:439)
at com.thoughtworks.go.domain.materials.SCMCommand.runOrBomb(SCMCommand.java:42)
at com.thoughtworks.go.domain.materials.git.GitCommand.updateSubmoduleWithDepth(GitCommand.java:257)
at com.thoughtworks.go.domain.materials.git.GitCommand.tryToShallowUpdateSubmodules(GitCommand.java:239)
at com.thoughtworks.go.domain.materials.git.GitCommand.updateSubmoduleWithInit(GitCommand.java:224)
at com.thoughtworks.go.domain.materials.git.GitCommand.resetWorkingDir(GitCommand.java:156)
at com.thoughtworks.go.config.materials.git.GitMaterial.updateTo(GitMaterial.java:168)
at com.thoughtworks.go.domain.MaterialRevision.updateTo(MaterialRevision.java:147)
at com.thoughtworks.go.domain.materials.AbstractMaterialAgent.prepare(AbstractMaterialAgent.java:43)
at com.thoughtworks.go.remote.work.BuildWork.prepareJob(BuildWork.java:165)
at com.thoughtworks.go.remote.work.BuildWork.build(BuildWork.java:118)
at com.thoughtworks.go.remote.work.BuildWork.doWork(BuildWork.java:79)
at com.thoughtworks.go.agent.JobRunner.run(JobRunner.java:53)

2020-08-13 18:28:10,657 WARN [scheduler-1] GitCommand:242 - git submodule update with --depth=1 failed. Attempting again with --depth=100
2020-08-13 18:28:12,507 WARN [scheduler-1] GitCommand:259 - Error performing command: --- Command ---
git submodule update --depth=100
--- Environment ---
{}
--- INPUT ----

--- EXIT CODE (1) ---
--- STANDARD OUT ---

--- STANDARD ERR ---
STDERR: Cloning into 'D:/Agents/go-agent-20.6.0-unity2/pipelines//PipelineRepo/PipelineConfiguration'...
STDERR: Logon failed, use ctrl+c to cancel basic credential prompt.
STDERR: bash: /dev/tty: No such device or address
STDERR: error: failed to execute prompt script (exit code 1)
STDERR: fatal: could not read Username for 'https://dev.azure.com': No such file or directory
STDERR: fatal: clone of 'https://dev.azure.com/#####/_git/TheSubmoduleRepo' into submodule path 'D:/Agents/go-agent-20.6.0-unity2/pipelines//PipelineRepo/PipelineConfiguration' failed
STDERR: Failed to clone 'PipelineConfiguration'. Retry scheduled
STDERR: Cloning into 'D:/Agents/go-agent-20.6.0-unity2/pipelines//PipelineRepo/PipelineConfiguration'...
STDERR: Logon failed, use ctrl+c to cancel basic credential prompt.
STDERR: bash: /dev/tty: No such device or address
STDERR: error: failed to execute prompt script (exit code 1)
STDERR: fatal: could not read Username for 'https://dev.azure.com': No such file or directory
STDERR: fatal: clone of 'https://dev.azure.com/#####/_git/TheSubmoduleRepo' into submodule path 'D:/Agents/go-agent-20.6.0-unity2/pipelines//PipelineRepo/PipelineConfiguration' failed
STDERR: Failed to clone 'PipelineConfiguration' a second time, aborting
---

com.thoughtworks.go.util.command.CommandLineException: Error performing command: --- Command ---
git submodule update --depth=100
--- Environment ---
{}
--- INPUT ----

--- EXIT CODE (1) ---
--- STANDARD OUT ---

--- STANDARD ERR ---
STDERR: Cloning into 'D:/Agents/go-agent-20.6.0-unity2/pipelines//PipelineRepo/PipelineConfiguration'...
STDERR: Logon failed, use ctrl+c to cancel basic credential prompt.
STDERR: bash: /dev/tty: No such device or address
STDERR: error: failed to execute prompt script (exit code 1)
STDERR: fatal: could not read Username for 'https://dev.azure.com': No such file or directory
STDERR: fatal: clone of 'https://dev.azure.com/#####/_git/TheSubmoduleRepo' into submodule path 'D:/Agents/go-agent-20.6.0-unity2/pipelines//PipelineRepo/PipelineConfiguration' failed
STDERR: Failed to clone 'PipelineConfiguration'. Retry scheduled
STDERR: Cloning into 'D:/Agents/go-agent-20.6.0-unity2/pipelines//PipelineRepo/PipelineConfiguration'...
STDERR: Logon failed, use ctrl+c to cancel basic credential prompt.
STDERR: bash: /dev/tty: No such device or address
STDERR: error: failed to execute prompt script (exit code 1)
STDERR: fatal: could not read Username for 'https://dev.azure.com': No such file or directory
STDERR: fatal: clone of 'https://dev.azure.com/#####/_git/TheSubmoduleRepo' into submodule path 'D:/Agents/go-agent-20.6.0-unity2/pipelines//PipelineRepo/PipelineConfiguration' failed
STDERR: Failed to clone 'PipelineConfiguration' a second time, aborting
---

at com.thoughtworks.go.util.command.CommandLine.runOrBomb(CommandLine.java:413)
at com.thoughtworks.go.util.command.CommandLine.runOrBomb(CommandLine.java:439)
at com.thoughtworks.go.domain.materials.SCMCommand.runOrBomb(SCMCommand.java:42)
at com.thoughtworks.go.domain.materials.git.GitCommand.updateSubmoduleWithDepth(GitCommand.java:257)
at com.thoughtworks.go.domain.materials.git.GitCommand.tryToShallowUpdateSubmodules(GitCommand.java:243)
at com.thoughtworks.go.domain.materials.git.GitCommand.updateSubmoduleWithInit(GitCommand.java:224)
at com.thoughtworks.go.domain.materials.git.GitCommand.resetWorkingDir(GitCommand.java:156)
at com.thoughtworks.go.config.materials.git.GitMaterial.updateTo(GitMaterial.java:168)
at com.thoughtworks.go.domain.MaterialRevision.updateTo(MaterialRevision.java:147)

2020-08-13 18:28:12,507 WARN [scheduler-1] GitCommand:246 - git submodule update with depth=100 failed. Attempting again with --depth=Integer.MAX
2020-08-13 18:28:14,915 WARN [scheduler-1] GitCommand:259 - Error performing command: --- Command ---
git submodule update --depth=2147483647
--- Environment ---
{}
--- INPUT ----

--- EXIT CODE (1) ---
--- STANDARD OUT ---

--- STANDARD ERR ---
STDERR: Cloning into 'D:/Agents/go-agent-20.6.0-unity2/pipelines//PipelineRepo/PipelineConfiguration'...
STDERR: Logon failed, use ctrl+c to cancel basic credential prompt.
STDERR: bash: /dev/tty: No such device or address
STDERR: error: failed to execute prompt script (exit code 1)
STDERR: fatal: could not read Username for 'https://dev.azure.com': No such file or directory
STDERR: fatal: clone of 'https://dev.azure.com/#####/_git/TheSubmoduleRepo' into submodule path 'D:/Agents/go-agent-20.6.0-unity2/pipelines//PipelineRepo/PipelineConfiguration' failed
STDERR: Failed to clone 'PipelineConfiguration'. Retry scheduled
STDERR: Cloning into 'D:/Agents/go-agent-20.6.0-unity2/pipelines//PipelineRepo/PipelineConfiguration'...
STDERR: Logon failed, use ctrl+c to cancel basic credential prompt.
STDERR: bash: /dev/tty: No such device or address
STDERR: error: failed to execute prompt script (exit code 1)
STDERR: fatal: could not read Username for 'https://dev.azure.com': No such file or directory
STDERR: fatal: clone of 'https://dev.azure.com/#####/_git/TheSubmoduleRepo' into submodule path 'D:/Agents/go-agent-20.6.0-unity2/pipelines//PipelineRepo/PipelineConfiguration' failed
STDERR: Failed to clone 'PipelineConfiguration' a second time, aborting
---

com.thoughtworks.go.util.command.CommandLineException: Error performing command: --- Command ---
git submodule update --depth=2147483647
--- Environment ---
{}
--- INPUT ----

--- EXIT CODE (1) ---
--- STANDARD OUT ---

--- STANDARD ERR ---
STDERR: Cloning into 'D:/Agents/go-agent-20.6.0-unity2/pipelines//PipelineRepo/PipelineConfiguration'...
STDERR: Logon failed, use ctrl+c to cancel basic credential prompt.
STDERR: bash: /dev/tty: No such device or address
STDERR: error: failed to execute prompt script (exit code 1)
STDERR: fatal: could not read Username for 'https://dev.azure.com': No such file or directory
STDERR: fatal: clone of 'https://dev.azure.com/#####/_git/TheSubmoduleRepo' into submodule path 'D:/Agents/go-agent-20.6.0-unity2/pipelines//PipelineRepo/PipelineConfiguration' failed
STDERR: Failed to clone 'PipelineConfiguration'. Retry scheduled
STDERR: Cloning into 'D:/Agents/go-agent-20.6.0-unity2/pipelines//PipelineRepo/PipelineConfiguration'...
STDERR: Logon failed, use ctrl+c to cancel basic credential prompt.
STDERR: bash: /dev/tty: No such device or address
STDERR: error: failed to execute prompt script (exit code 1)
STDERR: fatal: could not read Username for 'https://dev.azure.com': No such file or directory
STDERR: fatal: clone of 'https://dev.azure.com/#####/_git/TheSubmoduleRepo' into submodule path 'D:/Agents/go-agent-20.6.0-unity2/pipelines//PipelineRepo/PipelineConfiguration' failed
STDERR: Failed to clone 'PipelineConfiguration' a second time, aborting
---

at com.thoughtworks.go.util.command.CommandLine.runOrBomb(CommandLine.java:413)
at com.thoughtworks.go.util.command.CommandLine.runOrBomb(CommandLine.java:439)
at com.thoughtworks.go.domain.materials.SCMCommand.runOrBomb(SCMCommand.java:42)
at com.thoughtworks.go.domain.materials.git.GitCommand.updateSubmoduleWithDepth(GitCommand.java:257)
at com.thoughtworks.go.domain.materials.git.GitCommand.tryToShallowUpdateSubmodules(GitCommand.java:247)
at com.thoughtworks.go.domain.materials.git.GitCommand.updateSubmoduleWithInit(GitCommand.java:224)
at com.thoughtworks.go.domain.materials.git.GitCommand.resetWorkingDir(GitCommand.java:156)
at com.thoughtworks.go.config.materials.git.GitMaterial.updateTo(GitMaterial.java:168)
at com.thoughtworks.go.domain.MaterialRevision.updateTo(MaterialRevision.java:147)

```

Things I've tried:
- Emptying agent pipelines directory and re-running
- Removing user credentials from submodule url, i.e .was `https://Company@dev.azure.com/###` and is now `https://dev.azure.com/####`
- Confirmed same credentials can be used for both pipelines
- Installing fresh new gocd server and agent, added simple pipeline with my git repo as a material. Same issue

New server cfg

```

artifacts

```

- ~~Tried unsuccessfully to reproduce issue with fresh private azure repos. It seemed to work without issue~~.

### Github reproduce attempt

To attempt to reproduce the issue I started with a couple github repos (have since made them private)
- https://github.com/badbort/gocd-bug1.git
- https://github.com/badbort/gocd-bug1-submodule.git

Top one has the bottom one added as a submodule:
```
[submodule "Submodules/gocd-bug1-submodule"]
path = Submodules/gocd-bug1-submodule
url = https://github.com/badbort/gocd-bug1-submodule.git
```
When they were public I had no issues.

Public repo console material output

```

[go] Start to prepare Test-Submodules/4/Test/2/Test on Sulaco [D:\Agents\go-agent-20.6.0-unity2]
[go] Start to update materials.

[go] Start updating files at revision 8c4174d34aa002086ed24e83f2ad8e1c61a99a63 from https://github.com/badbort/gocd-bug1.git
STDERR: Cloning into 'D:\Agents\go-agent-20.6.0-unity2\pipelines\Test-Submodules'...
[GIT] Fetching changes
[GIT] Performing git gc
[GIT] Reset working directory pipelines\Test-Submodules
[GIT] Cleaning all unversioned files in working copy
[GIT] Cleaning submodule configurations in .git/config
[GIT] Updating working copy to revision 8c4174d34aa002086ed24e83f2ad8e1c61a99a63
HEAD is now at 8c4174d Added submodule
[GIT] Removing modified files in submodules
[GIT] Updating git sub-modules
[GIT] Cleaning unversioned files and sub-modules
[GIT] Git sub-module status
a13c9effce4ed3a3bed97c521d8da3c2f35fe5b5 Submodules/gocd-bug1-submodule (heads/master)
[GIT] Cleaning all unversioned files in working copy
[go] Done.

[go] setting environment variable 'GO_SERVER_URL' to value '#######'
[go] setting environment variable 'GO_PIPELINE_GROUP_NAME' to value 'Testing'
[go] setting environment variable 'GO_AGENT_RESOURCES' to value 'pipeline-unity'
[go] setting environment variable 'GO_TRIGGER_USER' to value 'anonymous'
[go] setting environment variable 'GO_PIPELINE_NAME' to value 'Test-Submodules'
[go] setting environment variable 'GO_PIPELINE_COUNTER' to value '4'
[go] setting environment variable 'GO_PIPELINE_LABEL' to value '4'
[go] setting environment variable 'GO_STAGE_NAME' to value 'Test'
[go] setting environment variable 'GO_STAGE_COUNTER' to value '2'
[go] setting environment variable 'GO_JOB_NAME' to value 'Test'
[go] setting environment variable 'GO_REVISION' to value '8c4174d34aa002086ed24e83f2ad8e1c61a99a63'
[go] setting environment variable 'GO_TO_REVISION' to value '8c4174d34aa002086ed24e83f2ad8e1c61a99a63'
[go] setting environment variable 'GO_FROM_REVISION' to value '8c4174d34aa002086ed24e83f2ad8e1c61a99a63'
[go] setting environment variable 'GO_MATERIAL_URL' to value 'https://github.com/badbort/gocd-bug1.git'
[go] setting environment variable 'GO_MATERIAL_BRANCH' to value 'master'
[go] setting environment variable 'GO_MATERIAL_HAS_CHANGED' to value 'true'
```

I then update both github repos so they are private, and enter the credentials in my pipeline. At this point I've experience multiple behaviours.

#### Behaviour 1
Immediate fail:
```
[go] Start to prepare Test-Submodules/6/Test/3/Test on Sulaco [D:\Agents\go-agent-20.6.0-unity2]
[go] Start to update materials.

[go] Start updating files at revision 8c4174d34aa002086ed24e83f2ad8e1c61a99a63 from https://github.com/badbort/gocd-bug1.git
STDERR: Cloning into 'D:\Agents\go-agent-20.6.0-unity2\pipelines\Test-Submodules'...
STDERR: remote: Repository not found.
STDERR: fatal: Authentication failed for 'https://github.com/badbort/gocd-bug1.git/'
Failed to run git clone command
```

Can confirm credentials are good:

#### Behaviour 2 - hangs forever

Console:
```
[go] Start to prepare Test-Submodules/6/Test/2/Test on Sulaco [D:\Agents\go-agent-20.6.0-unity2]
[go] Start to update materials.

[go] Start updating files at revision 8c4174d34aa002086ed24e83f2ad8e1c61a99a63 from https://github.com/badbort/gocd-bug1.git
[GIT] Fetching changes
```

Killing git-credential-manager.exe causes it to end and the pipeline to finally fail

Start with the GitCommand.java methods updateSubmoduleWithDepth, tryToShallowUpdateSubmodules, updateSubmoduleWithInit, and resetWorkingDir, then compare their behavior with the logged `git submodule update --depth=1` command. Reproduce the private Azure DevOps submodule failure on the reported Windows environment; done means submodule checkout completes non-interactively with the required credentials and the existing fallback behavior remains covered.

Written by the indexing model from the issue text.