Azure-Samples / Azure-Samples/Iot-Telemetry-Simulator

[CSS] Throttling during provisioning

Open
#39 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
73
Forks
49
Avg merge
4d 6h
Merged PRs (30d)
1

Description

### This issue is for a: (mark with an `x`)
```
- [x] bug report -> please search issues before submitting
- [ ] feature request
- [X] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)
```

### Minimal steps to reproduce
> Run the following command
> docker run -it -e "IotHubConnectionString=[redacted]" -e DeviceCount=1000 mcr.microsoft.com/oss/azure-samples/azureiot-simulatordeviceprovisioning

Provisioning crashes with a ThrottlingBacklogTimeout that causes the script to stop running.

### Any log messages given by the failure
>
PS C:\Users\kyichii>
Starting device provisioning
2021-10-12T07:17:21.6511837Z: 100 devices have been created @ 63.20/sec
2021-10-12T07:17:21.8319529Z: 200 devices have been created @ 553.18/sec
Microsoft.Azure.Devices.Common.Exceptions.ThrottlingException: {"Message":"ErrorCode:ThrottlingBacklogTimeout;The request has been throttled. Wait 10 seconds and try again. Operation type: CRUD","ExceptionMessage":"Tracking ID:55f0784980964ecf83ec9bf4cce2bc7c-G:0-TimeStamp:10/12/2021 07:18:02"}
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpClient httpClient, HttpMethod httpMethod, Uri requestUri, Func`3 modifyRequestMessageAsync, Func`2 isMappedToException, Func`3 processResponseMessageAsync, IDictionary`2 errorMappingOverrides, CancellationToken cancellationToken)
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpMethod httpMethod, Uri requestUri, Func`3 modifyRequestMessageAsync, Func`3 processResponseMessageAsync, IDictionary`2 errorMappingOverrides, CancellationToken cancellationToken)
at Microsoft.Azure.Devices.HttpClientHelper.PostAsync[T1,T2](Uri requestUri, T1 entity, IDictionary`2 errorMappingOverrides, IDictionary`2 customHeaders, CancellationToken cancellationToken)
at IotSimulatorDeviceProvisioning.Program.BulkCreateDevicesAsync(List`1 devices, RegistryManager registryManager, DeviceProvisionStats stats) in /src/src/IotSimulatorDeviceProvisioning/Program.cs:line 163
Microsoft.Azure.Devices.Common.Exceptions.ThrottlingException: {"Message":"ErrorCode:ThrottlingBacklogTimeout;The request has been throttled. Wait 10 seconds and try again. Operation type: CRUD","ExceptionMessage":"Tracking ID:3ea3022e253e4d7b95ae629f77c7c403-G:0-TimeStamp:10/12/2021 07:18:02"}
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpClient httpClient, HttpMethod httpMethod, Uri requestUri, Func`3 modifyRequestMessageAsync, Func`2 isMappedToException, Func`3 processResponseMessageAsync, IDictionary`2 errorMappingOverrides, CancellationToken cancellationToken)
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpMethod httpMethod, Uri requestUri, Func`3 modifyRequestMessageAsync, Func`3 processResponseMessageAsync, IDictionary`2 errorMappingOverrides, CancellationToken cancellationToken)
at Microsoft.Azure.Devices.HttpClientHelper.PostAsync[T1,T2](Uri requestUri, T1 entity, IDictionary`2 errorMappingOverrides, IDictionary`2 customHeaders, CancellationToken cancellationToken)
at IotSimulatorDeviceProvisioning.Program.BulkCreateDevicesAsync(List`1 devices, RegistryManager registryManager, DeviceProvisionStats stats) in /src/src/IotSimulatorDeviceProvisioning/Program.cs:line 163
Microsoft.Azure.Devices.Common.Exceptions.ThrottlingException: {"Message":"ErrorCode:ThrottlingBacklogTimeout;The request has been throttled. Wait 10 seconds and try again. Operation type: CRUD","ExceptionMessage":"Tracking ID:474cf77e1c214b409f7429e4ed439825-G:0-TimeStamp:10/12/2021 07:18:02"}
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpClient httpClient, HttpMethod httpMethod, Uri requestUri, Func`3 modifyRequestMessageAsync, Func`2 isMappedToException, Func`3 processResponseMessageAsync, IDictionary`2 errorMappingOverrides, CancellationToken cancellationToken)
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpMethod httpMethod, Uri requestUri, Func`3 modifyRequestMessageAsync, Func`3 processResponseMessageAsync, IDictionary`2 errorMappingOverrides, CancellationToken cancellationToken)
at Microsoft.Azure.Devices.HttpClientHelper.PostAsync[T1,T2](Uri requestUri, T1 entity, IDictionary`2 errorMappingOverrides, IDictionary`2 customHeaders, CancellationToken cancellationToken)
at IotSimulatorDeviceProvisioning.Program.BulkCreateDevicesAsync(List`1 devices, RegistryManager registryManager, DeviceProvisionStats stats) in /src/src/IotSimulatorDeviceProvisioning/Program.cs:line 163
Microsoft.Azure.Devices.Common.Exceptions.ThrottlingException: {"Message":"ErrorCode:ThrottlingBacklogTimeout;The request has been throttled. Wait 10 seconds and try again. Operation type: CRUD","ExceptionMessage":"Tracking ID:a6b0b0fd21ab40c0a341fbe65a366d54-G:0-TimeStamp:10/12/2021 07:18:02"}
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpClient httpClient, HttpMethod httpMethod, Uri requestUri, Func`3 modifyRequestMessageAsync, Func`2 isMappedToException, Func`3 processResponseMessageAsync, IDictionary`2 errorMappingOverrides, CancellationToken cancellationToken)
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpMethod httpMethod, Uri requestUri, Func`3 modifyRequestMessageAsync, Func`3 processResponseMessageAsync, IDictionary`2 errorMappingOverrides, CancellationToken cancellationToken)
at Microsoft.Azure.Devices.HttpClientHelper.PostAsync[T1,T2](Uri requestUri, T1 entity, IDictionary`2 errorMappingOverrides, IDictionary`2 customHeaders, CancellationToken cancellationToken)
at IotSimulatorDeviceProvisioning.Program.BulkCreateDevicesAsync(List`1 devices, RegistryManager registryManager, DeviceProvisionStats stats) in /src/src/IotSimulatorDeviceProvisioning/Program.cs:line 163

### Expected/desired behavior
> This should not crash when throttling. Ideally it would allow us to set the SKU rate for the IoT Hub to avoid the throttling all together and test provisioning at the maximum limits of a specific iot hub

### OS and Version?
> Windows 10

### Versions
> Latest

### Mention any other details that might be useful

This is behavior that inconsistently crashes but consistently throttles on lower end IoT Hub due to the registry operations. I am also attempting to get some additional logs from customers that are encountering the same issuje.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.