Debugging nuget packages through sourcelink displays wrong file

We have a .net framework 4.8 project with references to other .net projects and some of our own nuget packages.
Generally debugging into the nuget packages works fine.
When there are identical file names in the nuget package and one of the referenced projects, and we try to debug into the file in the nuget package, the file from the referenced project is displayed indstead of the one from the repo of the package. If we rename the file in the referenced project, the correct file from the nuget package is fetched and displayed when debugging.

_Wrong file loaded, the debugger highlights "random" code_

We expect the debugger fetch and display the correct file, even if a local file is identically named.