Azure / Azure/data-api-builder

[Enh]: Telemetry in CLI

Open
#3,214 1 comment 0 reactions 1 assignee Assigned to @Aniruddh25 View on GitHub
2.1 telemetry
Dominant language
C#
Stars
1.5k
Forks
370
Avg merge
3d 17h
Merged PRs (30d)
8

Description

## What?

Add usage telemetry to the CLI.

## Why?

Use telemetry to answer product questions, not surveillance.

## Opt out

Support environment variable `DAB_TELEMETRY_CLI_OPT_OUT=1` or `true`.

## Disclosure

We will detect the first time a DAB subcommand is used after installation and emit:

```text
Telemetry
---------
The DAB tools collect usage data to help us improve your experience. The data is collected by Microsoft. You can opt out of telemetry by setting the DAB_TELEMETRY_CLI_OPT_OUT environment variable to '1' or 'true' using your favorite shell.

Read more about DAB CLI Tools telemetry: https://aka.ms/dab/docs/cli
```

To disable this message and the DAB welcome message, set the `DAB_NOLOGO` environment variable to `1` or `true`. This variable has no effect on telemetry opt out. After the first time a DAB subcommand is used, the DAB CLI will somehow set this variable to `true`.

> [!NOTE]
> This will almost certainly require a sentinel file, but that is an ENG decision.

> [!IMPORTANT]
> We need a plan for upgrade, reinstall, and uninstall behavior.

## Allow-listed data only

Collect only these fields:

| Good | Bad |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| command name, subcommand, known switches used, DAB version, OS, install mode, CI yes/no, duration bucket, success/failure, exception type, `application_name` | connection strings, file paths, entity names, table names, view names, stored procedure names, descriptions, database policies, hostnames, tenant IDs, Key Vault references, environment variable values |

## Exception data

If the DAB CLI crashes, it collects the exception name and the stack trace of the CLI code. The DAB CLI collects this information to assess problems and improve quality.

The DAB CLI collects information for CLI exceptions only, not exceptions in your application. The collected data contains the exception name and the stack trace. This stack trace is from CLI code.

We will follow this approach: [https://learn.microsoft.com/en-us/dotnet/core/tools/telemetry?tabs=dotnet10#crash-exception-telemetry](https://learn.microsoft.com/en-us/dotnet/core/tools/telemetry?tabs=dotnet10#crash-exception-telemetry)

> [!IMPORTANT]
> We need to be sure CLI stack traces cannot accidentally include user paths, config values, or other sensitive strings in exception messages.

## Instrumentation point

Instrument once at the command dispatcher.

Add CLI middleware around every command to emit:

* `CommandStarted`
* `CommandCompleted`
* `CommandFailed`

## Transport

Keep transport simple and non-blocking.

Use Application Insights. Embed the default connection string in the CLI.

> [!TIP]
> Microsoft recommends the Azure Monitor OpenTelemetry Distro instead of the classic Application Insights SDK. We will use this only if it makes sense to us.

Support a hidden override for testing:

`DAB_TELEMETRY_AI_CONN_STRING=XYZ`

When set, this overrides the internal Application Insights connection string.

## Commands to instrument

Instrument every DAB CLI subcommand:

* `init`
* `add`
* `update`
* `configure`
* `auto-config`
* `start`
* `validate`
* `add-telemetry`
* `export`

## Flag capture rules

Capture every known flag used, but never its value.

Normalize flags to their canonical name.

Examples:

* `--connection-string` → capture `--connection-string`
* `--config dab-config.json` → capture `--config`
* `--permissions "anonymous:*"` → capture `--permissions`
* `-c` → capture `--config`
* `--runtime.mcp.enabled` → capture `--runtime.mcp.enabled`

> [!IMPORTANT]
> Make sure this means allow-listed switches only, resolved after alias normalization, with no fallback that logs unknown arguments.

Drop unknown flags.

Capture repeated flags once only.

## Existing context

Include the current DAB-specific telemetry `application_name`: `dab_oss_1.2.3+xxx|123|123|123+`

> [!NOTE]
> There will be no context group in the connection string. It is shown here as `xxx`.

## Additional context

Detect additional context values for the payload.

| Context | Value |
| ----------------- | --------------- |
| `Multi-config` | N = quantity |
| `CI Environment` | 1 = yes, 0 = no |
| `LLM Environment` | 1 = yes, 0 = no |

We might need to extend this as we continue.

## Example payload

```json
{
"dab_version": "1.2.3",
"command": "add",
"flags": ["--source", "--permissions", "--description"],
"app_name": "dab_oss_1.2.3+xxx|123|123|123+",
"multi_config": 1,
"ci_env": 1,
"llm_env": 1,
"os": "windows",
"duration_ms_bucket": "100-500",
"success": true
}
```

### Duration buckets:

* `<10`
* `10-100`
* `100-500`
* `500-1000`
* `>1000`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.