Azure / Azure/data-api-builder
[Enh]: Telemetry in CLI
- Dominant language
- C#
- Stars
- 1.5k
- Forks
- 370
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 8
Description
## What?
Add usage telemetry to the CLI.
## Why?
Use telemetry to answer product questions, not surveillance.
## Opt out
Support environment variable `DAB_TELEMETRY_CLI_OPT_OUT=1` or `true`.
## Disclosure
We will detect the first time a DAB subcommand is used after installation and emit:
```text
Telemetry
---------
The DAB tools collect usage data to help us improve your experience. The data is collected by Microsoft. You can opt out of telemetry by setting the DAB_TELEMETRY_CLI_OPT_OUT environment variable to '1' or 'true' using your favorite shell.
Read more about DAB CLI Tools telemetry: https://aka.ms/dab/docs/cli
```
To disable this message and the DAB welcome message, set the `DAB_NOLOGO` environment variable to `1` or `true`. This variable has no effect on telemetry opt out. After the first time a DAB subcommand is used, the DAB CLI will somehow set this variable to `true`.
> [!NOTE]
> This will almost certainly require a sentinel file, but that is an ENG decision.
> [!IMPORTANT]
> We need a plan for upgrade, reinstall, and uninstall behavior.
## Allow-listed data only
Collect only these fields:
| Good | Bad |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| command name, subcommand, known switches used, DAB version, OS, install mode, CI yes/no, duration bucket, success/failure, exception type, `application_name` | connection strings, file paths, entity names, table names, view names, stored procedure names, descriptions, database policies, hostnames, tenant IDs, Key Vault references, environment variable values |
## Exception data
If the DAB CLI crashes, it collects the exception name and the stack trace of the CLI code. The DAB CLI collects this information to assess problems and improve quality.
The DAB CLI collects information for CLI exceptions only, not exceptions in your application. The collected data contains the exception name and the stack trace. This stack trace is from CLI code.
We will follow this approach: [https://learn.microsoft.com/en-us/dotnet/core/tools/telemetry?tabs=dotnet10#crash-exception-telemetry](https://learn.microsoft.com/en-us/dotnet/core/tools/telemetry?tabs=dotnet10#crash-exception-telemetry)
> [!IMPORTANT]
> We need to be sure CLI stack traces cannot accidentally include user paths, config values, or other sensitive strings in exception messages.
## Instrumentation point
Instrument once at the command dispatcher.
Add CLI middleware around every command to emit:
* `CommandStarted`
* `CommandCompleted`
* `CommandFailed`
## Transport
Keep transport simple and non-blocking.
Use Application Insights. Embed the default connection string in the CLI.
> [!TIP]
> Microsoft recommends the Azure Monitor OpenTelemetry Distro instead of the classic Application Insights SDK. We will use this only if it makes sense to us.
Support a hidden override for testing:
`DAB_TELEMETRY_AI_CONN_STRING=XYZ`
When set, this overrides the internal Application Insights connection string.
## Commands to instrument
Instrument every DAB CLI subcommand:
* `init`
* `add`
* `update`
* `configure`
* `auto-config`
* `start`
* `validate`
* `add-telemetry`
* `export`
## Flag capture rules
Capture every known flag used, but never its value.
Normalize flags to their canonical name.
Examples:
* `--connection-string` → capture `--connection-string`
* `--config dab-config.json` → capture `--config`
* `--permissions "anonymous:*"` → capture `--permissions`
* `-c` → capture `--config`
* `--runtime.mcp.enabled` → capture `--runtime.mcp.enabled`
> [!IMPORTANT]
> Make sure this means allow-listed switches only, resolved after alias normalization, with no fallback that logs unknown arguments.
Drop unknown flags.
Capture repeated flags once only.
## Existing context
Include the current DAB-specific telemetry `application_name`: `dab_oss_1.2.3+xxx|123|123|123+`
> [!NOTE]
> There will be no context group in the connection string. It is shown here as `xxx`.
## Additional context
Detect additional context values for the payload.
| Context | Value |
| ----------------- | --------------- |
| `Multi-config` | N = quantity |
| `CI Environment` | 1 = yes, 0 = no |
| `LLM Environment` | 1 = yes, 0 = no |
We might need to extend this as we continue.
## Example payload
```json
{
"dab_version": "1.2.3",
"command": "add",
"flags": ["--source", "--permissions", "--description"],
"app_name": "dab_oss_1.2.3+xxx|123|123|123+",
"multi_config": 1,
"ci_env": 1,
"llm_env": 1,
"os": "windows",
"duration_ms_bucket": "100-500",
"success": true
}
```
### Duration buckets:
* `<10`
* `10-100`
* `100-500`
* `500-1000`
* `>1000`
Contributor guide
Assessment
This issue has not been assessed yet.