AddDaprSecretStore always times out

I have services deployed in Azure Container Apps with Dapr enabled. I'm trying to add a secrret store to the configuration like this:
```cs
builder.Services.AddDaprClient();

if (!string.IsNullOrEmpty(builder.Configuration["SecretStore"]))
{
var timeout = builder.Configuration.GetValue("DaprSidecarTimeout", 5);
using var sp = builder.Services.BuildServiceProvider();
builder.Configuration.AddDaprSecretStore(builder.Configuration["SecretStore"], sp.GetRequiredService(), TimeSpan.FromSeconds(timeout));
}
```

This works locally running with docker compose, but in the container app this always creates the following exception:

```
Unhandled exception. System.Threading.Tasks.TaskCanceledException: A task was canceled.
at Dapr.Client.DaprClientGrpc.WaitForSidecarAsync(CancellationToken cancellationToken)
at Dapr.Extensions.Configuration.DaprSecretStore.DaprSecretStoreConfigurationProvider.LoadAsync()
at Dapr.Extensions.Configuration.DaprSecretStore.DaprSecretStoreConfigurationProvider.Load()
at Microsoft.Extensions.Configuration.ConfigurationManager.AddSource(IConfigurationSource source)
at Microsoft.Extensions.Configuration.ConfigurationManager.Microsoft.Extensions.Configuration.IConfigurationBuilder.Add(IConfigurationSource source)
...
```
I have tried with timeouts of up to one minute.

Is there some configuration I am missing that I should have set?