GPU Performance non functional

**System Information (please complete the following information):**
- OS & Version: Windows 11
- ML.NET Version: 1.7.0
- .NET Version: 6.0

**Describe the bug**
Following all available documentation here (https://learn.microsoft.com/en-us/dotnet/machine-learning/tutorials/image-classification-api-transfer-learning) and resources available using SciSharp.Tensorflow.Redist-Windows-GPU V2.3.1 to utilize a 2060 TI using the images within the documentation of concrete as a base for training I am getting absolutely useless and terrible predictions and data coming back.

**To Reproduce**
Steps to reproduce the behavior:
1. Open the solution folder
2. Run the API
3. Send a post request in postman to (http://localhost:5267/api/Trainer/buildModels with the path to the top level folder to build the model
4. Send a Postman request to http://localhost:5267/api/Trainer/predict with a form-data
Key: file, Value: image
Key: modelPath, Value: path to model folder, (sans the models.zip)

**Expected behavior**
Training results in the following:

Phase: Training, Dataset used: Validation, Batch Processed Count: 1124, Epoch: 0, Accuracy: 0.071183994, Cross-Entropy: 18.97249
Phase: Training, Dataset used: Validation, Batch Processed Count: 1124, Epoch: 1, Accuracy: 0.071183994, Cross-Entropy: 15.170444
Phase: Training, Dataset used: Validation, Batch Processed Count: 1124, Epoch: 2, Accuracy: 0.071183994, Cross-Entropy: 30.240627
Phase: Training, Dataset used: Validation, Batch Processed Count: 1124, Epoch: 3, Accuracy: 0.38297707, Cross-Entropy: 26.751282
Phase: Training, Dataset used: Validation, Batch Processed Count: 1124, Epoch: 4, Accuracy: 0.20089968, Cross-Entropy: 34.458427
Phase: Training, Dataset used: Validation, Batch Processed Count: 1124, Epoch: 5, Accuracy: 0.38297707, Cross-Entropy: 26.768661
Phase: Training, Dataset used: Validation, Batch Processed Count: 1124, Epoch: 6, Accuracy: 0.071183994, Cross-Entropy: 31.264683
Phase: Training, Dataset used: Validation, Batch Processed Count: 1124, Epoch: 7, Accuracy: 0.38297707, Cross-Entropy: 16.774061
Phase: Training, Dataset used: Validation, Batch Processed Count: 1124, Epoch: 8, Accuracy: 0.20089968, Cross-Entropy: 23.46656

And predictions lead to a 0 or 1 in one of 6 categories and always comes up the same regardless of what image is sent.

**Screenshots, Code, Sample Projects**
The entire program will be zipped and attached

**Additional context**
Using NVIDIA 10.1 and CUDNN 7.6.4 as required

[New Compressed (zipped) Folder.zip](https://github.com/dotnet/machinelearning/files/11409001/New.Compressed.zipped.Folder.zip)