[SCM] Enabling scm.workingSets.enabled causes unwanted workspace file switching when checking out branches created with git worktree

Does this issue occur when all extensions are disabled?: Yes

- VS Code Version: 1.134.0
- OS Version: Windows 11

Steps to Reproduce:

1. Clone a repository that uses multiple Git worktrees.

2. Open the main worktree folder in VS Code.

3. Ensure scm.workingSets.enabled is set to true in settings.

4. Open the Source Control view (SCM).

5. Switch between branches that are linked to different worktrees (e.g., check out feature-branch from the main worktree, or open another worktree folder).