Regexes with spaces and \s cause issues with different versions of git and grep

# Background

* We need a way to prevent the addition of secrets into git repos with users of every flavor of OS.
* We need to minimize false positives and inconsistency across OSes.
* Every OS vendor has a different version of git and a different version of grep
* grep -E is different across versions
* `git-secrets` is too quiet on what it is doing, you do not know what it is doing (no VERBOSE mode or logging of any type) and you often do not know your regex is wrong.
* Requiring uniformity across development environments would be ideal, but it is an unreasonable/unenforceable requirement

# Test environments
* Windows 10
* with Cygwin
* git version 2.17.0
* grep (GNU grep) 3.0 - Packaged by Cygwin (3.0-2)
* Native (using PowerShell)
* git version 2.21.0.windows.1

* RedHat Linux: Linux 3.10.0-957.1.3.el7.x86_64
* git version 1.8.3.1
* grep (GNU grep) 2.20

# Test file

```
password=helloNOW
passwd = "adsjhjksadfhksahdfsadf"
secret: "sjkdhfjksahdfksaf"
PassWord =ddaffafdasfd
"my_password": "strong_one!"
```

# The regex

```
"[^ ]*([pP][aA][sS]{2}[wWoOrRdD]{0,4}|[sS][eE][cC][rR][eE][tT])[\"\' ]*[=:>] *[^ ]+"
```

# Testing with `grep`

I have tested (on all test servers above) my regex with the same flags as the `git grep` command in the script:

```
grep -nwHEI "[^ ]*([pP][aA][sS]{2}[wWoOrRdD]{0,4}|[sS][eE][cC][rR][eE][tT])[\"\' ]*[=:>] *[^ ]+" test.txt
```

Which gives me:

```
test.txt:1:password=helloNOW
test.txt:2:passwd = "adsjhjksadfhksahdfsadf"
test.txt:3:secret: "sjkdhfjksahdfksaf"
test.txt:4:PassWord =ddaffafdasfd
test.txt:5:"my_password": "strong_one!"
```

# Testing with `git grep`

```
git grep -nwHEI "[^ ]*([pP][aA][sS]{2}[wWoOrRdD]{0,4}|[sS][eE][cC][rR][eE][tT])[\"\' ]*[=:>] *[^ ]+" test.txt
```

And get:

```
test.txt:1:password=helloNOW
test.txt:2:passwd = "adsjhjksadfhksahdfsadf"
test.txt:3:secret: "sjkdhfjksahdfksaf"
test.txt:4:PassWord =ddaffafdasfd
test.txt:5:"my_password": "strong_one!"
```

# Testing with `git-secrets`

I add the regex to git secrets:

```
git secrets --add "[^ ]*([pP][aA][sS]{2}[wWoOrRdD]{0,4}|[sS][eE][cC][rR][eE][tT])[\"\' ]*[=:>] *[^ ]+"
```

And `git secrets --scan` fails, because all the spaces become pipes:

```
$ git secrets --scan
fatal: command line, '[^|]*([pP][aA][sS]{2}[wWoOrRdD]{0,4}|[sS][eE][cC][rR][eE][tT])["\'|]*[=:>]|*[^|]+': repetition-operator operand invalid
```

If I scan just the file, this makes `get secrets` just do a regular grep and it works (I am removing the possible mitigations output to save space):

```
$ git secrets --scan test.txt
test.txt:1:password=helloNOW
test.txt:2:passwd = "adsjhjksadfhksahdfsadf"
test.txt:3:secret: "sjkdhfjksahdfksaf"
test.txt:4:PassWord =ddaffafdasfd
test.txt:5:"my_password": "strong_one!"

[ERROR] Matched one or more prohibited patterns
```

If I replace the spaces with `\\s` in the `.git/config`:

```
[secrets]
patterns = [^\\s]*([pP][aA][sS]{2}[wWoOrRdD]{0,4}|[sS][eE][cC][rR][eE][tT])[\"\\'\\s]*[=:>]\\s*[^\\s]+
```

Then the scans return fewer and inconsistent matches across environments:

*Windows*, both PowerShell and Cygwin (less matches with git grep than even Linux below):

```
$ git secrets --scan
test.txt:1:password=helloNOW

[ERROR] Matched one or more prohibited patterns
ANDREWCG@INFECTED10-01 /cygdrive/c/git/misc/git-tools
$ git secrets --scan test.txt
test.txt:1:password=helloNOW
test.txt:3:secret: "sjkdhfjksahdfksaf"
test.txt:5:"my_password": "strong_one!"

[ERROR] Matched one or more prohibited patterns
```

*Linux*

```
$ git secrets --scan
test.txt:1:test.txt:1:password=helloNOW
test.txt:3:test.txt:3:secret: "sjkdhfjksahdfksaf"
test.txt:5:test.txt:5:"my_password": "strong_one!"

[ERROR] Matched one or more prohibited patterns

$ git secrets --scan test.txt
test.txt:1:test.txt:1:password=helloNOW
test.txt:3:test.txt:3:secret: "sjkdhfjksahdfksaf"
test.txt:5:test.txt:5:"my_password": "strong_one!"

[ERROR] Matched one or more prohibited patterns
```

# The issue is `\s`

It seems that `\s` is a problematic replacement in some versions of grep. The fix I found is to replace every instance of `\s` with a space:

# A possible fix in the `load_combined_patterns` function:

Change:

```
# load patterns and combine them with |
load_combined_patterns() {
local patterns=$(load_patterns)
local combined_patterns=''
for pattern in $patterns; do
combined_patterns=${combined_patterns}${pattern}"|"
done
combined_patterns=${combined_patterns%?}
echo $combined_patterns
}
```

To:

```
# load patterns and combine them with |
load_combined_patterns() {
local patterns=$(load_patterns)
local combined_patterns=''
for pattern in $patterns; do
combined_patterns=${combined_patterns}${pattern}"|"
done
echo ${combined_patterns%?} | sed 's/\\s/ /g'
}
```

Now it works:

```
$ git secrets --scan
test.txt:1:password=helloNOW
test.txt:2:passwd = "adsjhjksadfhksahdfsadf"
test.txt:3:secret: "sjkdhfjksahdfksaf"
test.txt:4:PassWord =ddaffafdasfd
test.txt:5:"my_password": "strong_one!"

[ERROR] Matched one or more prohibited patterns
```

This is not an ideal solution as `\s` means much more than just the space character. So what can be done?

## Better fix: line break only at `\n` in `load_combined_patterns`

```
load_combined_patterns() {
local patterns=$(load_patterns)
local combined_patterns=''
local IFS=$'\n'
for pattern in $patterns; do
combined_patterns=${combined_patterns}${pattern}"|"
done
combined_patterns=${combined_patterns%?}
echo $combined_patterns
}
```

Now the regex I created with space characters in it works on all versions of git/OS I have (Linux, Cygwin and Powershell):

```
PS C:\git\misc\git-tools> git secrets --scan
test.txt:1:password=helloNOW
test.txt:2:passwd = "adsjhjksadfhksahdfsadf"
test.txt:3:secret: "sjkdhfjksahdfksaf"
test.txt:4:PassWord =ddaffafdasfd
test.txt:5:"my_password": "strong_one!"

[ERROR] Matched one or more prohibited patterns
```

Start in the script's load_combined_patterns function and inspect how load_patterns is split before patterns are passed to git grep; reproduce the issue with test.txt, the shown regex, and git secrets --scan on the listed environments. Done means patterns containing spaces or \s are handled consistently without invalid-regex errors and the scan reports the expected matches.

Written by the indexing model from the issue text.