GitHub Actions to push to Azure Artifacts: Unable to load the service index for source ' 'Response status code does not indicate success: 401 (Unauthorized)

I wanted to create a Github workflow that push feed to Azure Artifacts. I carefully followed the article [here](https://learn.microsoft.com/en-us/azure/devops/artifacts/quickstarts/github-actions?view=azure-devops) and the tutorial [here](https://github.com/Azure/actions-workflow-samples/blob/master/PackageManagement/nuget.yml). However, I keep getting this error

`error: Unable to load the service index for source https://pkgs.dev.azure.com///_packaging//nuget/v3/index.json.
error: Response status code does not indicate success: 401 (Unauthorized).`

I created a PAT for all accessible organizations, granted it full access and added it in my Github Action secrets just the docs says but the persists. Does anyone have idea what could be wrong here?