Project References fails when `baseUrl` is used

Expected Behaviour

I forked the project reference example with a change that uses baseUrl. It should build with no error.

Actual Behaviour

Error is seen (with logLevel: 'info'):

ts-loader: Using typescript@4.7.4 and /Users/me/repos/ts-loader/examples/project-references-example/tsconfig.json
Using SolutionBuilder api
message TS6355: Projects in this build: 
    * ../packages/core/tsconfig.json
    * ../packages/animals/tsconfig.json
    * ../packages/zoo/tsconfig.json

message TS6351: Project '../packages/core/tsconfig.json' is up to date because newest input '../packages/core/utilities.ts' is older than oldest output '../packages/core/lib/utilities.js'

message TS6352: Project '../packages/animals/tsconfig.json' is out of date because output file '../packages/animals/lib/animal.js' does not exist

message TS6358: Building project '/Users/me/repos/ts-loader/examples/project-references-example/packages/animals/tsconfig.json'...

packages/animals/dog.tsx(1,30): error TS2307: Cannot find module 'animal' or its corresponding type declarations.

packages/animals/dog.tsx(14,9): error TS2322: Type '{ size: Size; woof: (this: Dog) => string; name: string; }' is not assignable to type 'Dog'.
  Object literal may only specify known properties, and 'size' does not exist in type 'Dog'.

message TS6363: Project '../packages/zoo/tsconfig.json' can't be built because its dependency '../packages/animals' has errors

message TS6362: Skipping build of project '/Users/me/repos/ts-loader/examples/project-references-example/packages/zoo/tsconfig.json' because its dependency '/Users/me/repos/ts-loader/examples/project-references-example/packages/animals' has errors

Found 2 errors. Watching for file changes.

Steps to Reproduce the Problem

By using baseUrl (see changes at: https://github.com/govizlora/ts-loader/commit/0a66ce17b15f7056aa29af7ad108c2be1971adc5), the build will be broken.

Note that using paths will break the build as well.

Location of a Minimal Repository that Demonstrates the Issue.

https://github.com/govizlora/ts-loader/tree/main/examples/project-references-example

Start by reproducing the failure in examples/project-references-example, then inspect its tsconfig files and the project-reference build path used by ts-loader. Compare behavior with and without baseUrl, and with paths as noted in the issue. Done means the example builds without the reported module-resolution and dependent-project errors.