Azure / Azure/data-api-builder

[Feature Request]: Configurable Telemetry Throttling for High Throughput Scenarios

Open
#2,287 3 comments 0 reactions 1 assignee Claimed by @abhishekkumams View on GitHub
cri enhancement telemetry triage
Dominant language
C#
Stars
1.5k
Forks
370
Avg merge
3d 17h
Merged PRs (30d)
8

Description

### What happened?

**Issue Description**: While conducting load testing on an API utilizing the DataApiBuilder (DAB), it was observed that the telemetry data available in Live Metrics and other Application Insights tools only reflects a fraction of the actual traffic. Specifically, during a test simulating 1000 requests per second, Live Metrics reported processing only around 4 requests per second. This discrepancy is believed to be due to the current telemetry throttling mechanism, which significantly samples or throttles the telemetry data, thus not providing a true picture of the system's performance under load.

**Impact**: This issue makes it challenging to accurately monitor and assess the application's performance and health during high-load scenarios, which is critical for capacity planning, performance tuning, and ensuring the reliability of the service.

**Steps to Reproduce**:

1. Set up a basic API using DataApiBuilder.
2. Configure Application Insights for telemetry collection.
3. Conduct a load test simulating 1000 requests per second to the API.
4. Observe the telemetry data in Application Insights Live Metrics.

**Expected Behavior**: The telemetry data in Application Insights should accurately reflect the load generated by the test, allowing for real-time monitoring and analysis of the application's performance under stress.

**Actual Behavior**: The telemetry data significantly underreports the actual traffic, indicating only about 4 requests per second in Live Metrics, due to aggressive telemetry sampling or throttling.

**Feature Request**: I propose the introduction of a feature or configuration option within DAB that allows users to adjust the level of telemetry throttling or sampling, especially for scenarios requiring precise monitoring and diagnostics, such as load testing or performance benchmarking. This feature would enable developers and system administrators to get a more accurate picture of the application's behavior under various load conditions, improving the observability and manageability of applications built with DAB.

**Potential Benefits**:

1. Enhanced monitoring and diagnostics capabilities for high-load scenarios.
2. Improved accuracy of performance metrics, aiding in more effective capacity planning and performance tuning.
3. Increased flexibility in telemetry management, allowing for tailored configurations based on the specific needs of the application and environment.

Thank you for considering this feature request. I believe it would significantly enhance the utility and flexibility of monitoring, performance testing and benchmarking for DAB, especially for applications with high throughput demands.

### Version

1.1.7

### What database are you using?

Azure SQL

### What hosting model are you using?

Container Apps

### Which API approach are you accessing DAB through?

REST

### Relevant log output

_No response_

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.