Investigate changing lockfile format to reduce needless git conflicts

The Problem

We have repos internally with a very large number of dependencies. When a decent number of dependencies are updated at the same time by Excavator in different PRs, there are a very large number of lockfiles conflicts. This lead to the discussion about whether the lockfile is too conservative with trying to be safe by causing git conflicts in the presence of not-up-to date PRs (which we rely on to not explode the number of CI builds). If we could reduce the number of conflicts, it would have a big effect on the mergeability of automated upgrade PRs (and also save humans time)

Proposed changes

Remove the (1 constraints: 50295fc1)

Currently, there are (1 constraints: 50295fc1) at the end of versions.locks lines, and these often cause merge conflicts. These do prevent some issues in the presence of not up-to-date PR builds, basically this case where Branch A and B both fork off develop:

• develop has a single dep on foo:bar, versions.locks looks like foo:bar:1.0.0 (1 constraints: aaaaaaa)
• Branch A adds another dep on foo:bar, versions.locks becomes foo:bar:1.0.0 (2 constraints: bbbbbbb)
• Branch B deletes the original dep on foo:bar, versions.locks becomes empty

If Branch A in merged first, because there is a diff on that line from develop to Branch A, the delete from Branch B conflicts with it, so you must pull develop and regenerate the correct the versions.locks

In the world without the (1 constraints: 50295fc1), this conflict never appears and it breaks develop with an incorrect lock file:

• develop has a single dep on foo:bar, versions.locks looks like foo:bar:1.0.0
• Branch A adds another dep on foo:bar, versions.locks stays foo:bar:1.0.0 - no diff
• Branch B deletes the original dep on foo:bar, versions.locks becomes empty

If Branch A in merged first, because there is a no diff for versions.locks, Branch B has no conflict. Branch B merges in with no changes, the lockfile becomes empty on develop, despite there still being a dependency on foo:bar introduced by Branch A. Semantic merge conflict.

Change from (1 constraints: 50295fc1) to just 1 dependencies

I also looked at whether we could simplify (1 constraints: 50295fc1) to just (1 dependencies). That keeps protection for the case above (count changes whenever a requester is added or removed), but loses protection for a narrower case: when one branch swaps a requester for a different one at the same version. Consider this case where Branch A and B both fork off develop:

• develop has a dep baz which transitively pulls in foo:bar 1.0.0. versions.lock has foo:bar:1.0.0 (1 constraints: aaaaaaa)
• Branch A replaces baz with quux, which also transitively pulls in foo:bar 1.0.0. versions.lock becomes foo:bar:1.0.0 (1 constraints: bbbbbbb) — same count, different requester, different hash
• Branch B deletes baz. versions.lock becomes empty

If Branch A is merged first, because there is a diff on that line from develop to Branch A (the hash changed), the delete from Branch B conflicts with it, so you must pull develop and regenerate the correct the versions.locks

In the world without the hash, just (1 dependencies), this conflict never appears and it breaks develop with an incorrect lockfile:

• develop has foo:bar:1.0.0 (1 dependencies)
• Branch A replaces baz with quux. Still 1 requester of foo:bar, so versions.lock stays foo:bar:1.0.0 (1 dependencies) — no diff
• Branch B deletes baz. versions.lock becomes empty

If Branch A is merged first, because there is no diff for versions.lock on Branch A, Branch B has no conflict. Branch B merges in with no changes, the lockfile becomes empty on develop, despite quux still transitively requiring foo:bar. Semantic merge conflict.

Change from (1 constraints: <hash_of_constraints_with_versions>) to just (1 dependencies: <hash_of_dependencies_without_versions>)

We currently both:

1. Count the number of constraints (not just dependencies)
2. Form the hash from constraints as well as dependencies, including version numbers, by converting them to string form:

   com.google.guava:guava       -> [1.0.0,2.0)
   com.fasterxml.jackson:core   -> 1.0.0
   projects                     -> 1.0.0   (from some sub-project)
   

I think both these are overkill.

The lines in versions.lock achieve two different goals:

1. Record and enforce the version of each dep that is used
   1. This is achieved by the actual version in the line.
   2. However, the hash also contains the version (and other range constraints).
2. Stop semantic merge conflicts in the presence of non-up-to-date PR merges which causes issues on the main branches. These occur when dependencies are removed or added (either directly or transitively) in separate non-up-to-date PRs.
   1. The hash currently stops this happening because it encodes all the dependencies (either direct or transitive) that bring in a certain line's dep - it forces a git conflict when the "dependency graph structure" changes.
   2. However, we have this extraneous information that doesn't matter:
      1. Constraints just contribute to the version selection. They don't force versions to be used. We already record the version.
      2. Version numbers in both constraints and dependencies also contribute to version selection. We don't need this information.

So basically we want to separate the concerns:

• Version information is only saved by version number in the line (not in the hash)
• The hash encodes the "structure" of the dependencies that bring in the line's dep (no version numbers or constraints)

This should result in much fewer needless git conflicts.

Start by locating the code that generates versions.lock entries and computes the constraints hash, then find tests covering lockfile regeneration and merge-conflict protection. Compare the current constraint-based behavior with the proposed dependency-structure behavior, including added, removed, and replaced requesters. Done means the format and tests demonstrate fewer needless conflicts without allowing semantic lockfile merges.