The invocations to `git grep` fails if the total length of filepaths causes the command line be over 32767
- Dominant language
- Shell
- Stars
- 13.4k
- Forks
- 1.3k
- PR merge metrics
- No merged PRs in 30d
Description
So in the git-secrets script below there is an invocation to `git grep` where it supplies the list of source controlled files to the command
https://github.com/awslabs/git-secrets/blob/80230afa8c8bdeac766a0fece36f95ffaa0be778/git-secrets#L116
If the aggregate list of the file paths supplied to the git grep command causes the comamnd line length to be larger than 32767 characters on Windows which is the [max command line length](https://devblogs.microsoft.com/oldnewthing/20031210-00/?p=41553)
The invocation fails with the following error
```
C:\Users\\git-secrets\git-secrets: line 116: /mingw64/libexec/git-core/git: Argument list too long
```
This can occur with a merge from a large code base where the number of files and paths to those files causes the file list that would be supplied to `git grep` to be large.
Having several hundred files that are part of the merge that has a relative path similar to filepaths as `Code/CoreLibrary/EditorViewport/ViewportFile.ext` will eventually cause the single the `git grep` command to be over the 32767 line length
Now using "echo" and "xargs" allows the git-grep command to be invoked multiple times without issue
`GREP_OPTIONS= LC_ALL=C printf "%s\0" "${files[@]}" | xargs -0 git grep -nwHEI ${options} "${combined_patterns}"`
Now the caveat with using xargs, is that when any of invocations that it spawns off returns a return code between 1-125, xargs returns the return code of 123
This is explained in the xargs man page [here](https://man7.org/linux/man-pages/man1/xargs.1.html#EXIT_STATUS)
Now the `process_output()` function is checking for a return code of 1 to determine if the pattern wasn't matched.
https://github.com/awslabs/git-secrets/blob/80230afa8c8bdeac766a0fece36f95ffaa0be778/git-secrets#L131-L145
Because of that the return code of 123 from xargs needs to be mapped to 1, to make sure that `process_output`
still function successfully
Therefore the full set of changes to the `git_grep` functions need to be the following
```
if (( ${#files[@]} )); then
GREP_OPTIONS= LC_ALL=C printf "%s\0" "${files[@]}" | xargs -0 git grep -nwHEI ${options} "${combined_patterns}"
else
GREP_OPTIONS= LC_ALL=C git grep -nwHEI ${options} "${combined_patterns}"
fi
rc=$?
[ $rc -eq 123 ] && return 1 # xargs returns 123 when the invocations return a value between 1-125
[ $rc -eq 0 ] && return 0
return $rc
```
Contributor guide
Research direction
Start with the git_grep functions in git-secrets around line 116, then read process_output around lines 131-145 to understand how grep statuses are handled. Reproduce the failure with a sufficiently large file list on Windows, and verify that large lists complete without the command-line limit while no-match and grep-error statuses remain distinguishable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- git, shell
- Domain
- cli, security
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100