Azure / Azure/data-api-builder

[Enh]: Telemetry in CLI

Aperta
#3,214 1 commento 0 reazioni 1 assegnatario Assegnata a @Aniruddh25 Vedi su GitHub
2.1 telemetry
Lingua principale
C#
Stelle
1.5k
Fork
370
Merge medio
3g 22h
PR unite (30g)
9

Descrizione

## What?

Add usage telemetry to the CLI.

## Why?

Use telemetry to answer product questions, not surveillance.

## Opt out

Support environment variable `DAB_TELEMETRY_CLI_OPT_OUT=1` or `true`.

## Disclosure

We will detect the first time a DAB subcommand is used after installation and emit:

```text
Telemetry
---------
The DAB tools collect usage data to help us improve your experience. The data is collected by Microsoft. You can opt out of telemetry by setting the DAB_TELEMETRY_CLI_OPT_OUT environment variable to '1' or 'true' using your favorite shell.

Read more about DAB CLI Tools telemetry: https://aka.ms/dab/docs/cli
```

To disable this message and the DAB welcome message, set the `DAB_NOLOGO` environment variable to `1` or `true`. This variable has no effect on telemetry opt out. After the first time a DAB subcommand is used, the DAB CLI will somehow set this variable to `true`.

> [!NOTE]
> This will almost certainly require a sentinel file, but that is an ENG decision.

> [!IMPORTANT]
> We need a plan for upgrade, reinstall, and uninstall behavior.

## Allow-listed data only

Collect only these fields:

| Good | Bad |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| command name, subcommand, known switches used, DAB version, OS, install mode, CI yes/no, duration bucket, success/failure, exception type, `application_name` | connection strings, file paths, entity names, table names, view names, stored procedure names, descriptions, database policies, hostnames, tenant IDs, Key Vault references, environment variable values |

## Exception data

If the DAB CLI crashes, it collects the exception name and the stack trace of the CLI code. The DAB CLI collects this information to assess problems and improve quality.

The DAB CLI collects information for CLI exceptions only, not exceptions in your application. The collected data contains the exception name and the stack trace. This stack trace is from CLI code.

We will follow this approach: [https://learn.microsoft.com/en-us/dotnet/core/tools/telemetry?tabs=dotnet10#crash-exception-telemetry](https://learn.microsoft.com/en-us/dotnet/core/tools/telemetry?tabs=dotnet10#crash-exception-telemetry)

> [!IMPORTANT]
> We need to be sure CLI stack traces cannot accidentally include user paths, config values, or other sensitive strings in exception messages.

## Instrumentation point

Instrument once at the command dispatcher.

Add CLI middleware around every command to emit:

* `CommandStarted`
* `CommandCompleted`
* `CommandFailed`

## Transport

Keep transport simple and non-blocking.

Use Application Insights. Embed the default connection string in the CLI.

> [!TIP]
> Microsoft recommends the Azure Monitor OpenTelemetry Distro instead of the classic Application Insights SDK. We will use this only if it makes sense to us.

Support a hidden override for testing:

`DAB_TELEMETRY_AI_CONN_STRING=XYZ`

When set, this overrides the internal Application Insights connection string.

## Commands to instrument

Instrument every DAB CLI subcommand:

* `init`
* `add`
* `update`
* `configure`
* `auto-config`
* `start`
* `validate`
* `add-telemetry`
* `export`

## Flag capture rules

Capture every known flag used, but never its value.

Normalize flags to their canonical name.

Examples:

* `--connection-string` → capture `--connection-string`
* `--config dab-config.json` → capture `--config`
* `--permissions "anonymous:*"` → capture `--permissions`
* `-c` → capture `--config`
* `--runtime.mcp.enabled` → capture `--runtime.mcp.enabled`

> [!IMPORTANT]
> Make sure this means allow-listed switches only, resolved after alias normalization, with no fallback that logs unknown arguments.

Drop unknown flags.

Capture repeated flags once only.

## Existing context

Include the current DAB-specific telemetry `application_name`: `dab_oss_1.2.3+xxx|123|123|123+`

> [!NOTE]
> There will be no context group in the connection string. It is shown here as `xxx`.

## Additional context

Detect additional context values for the payload.

| Context | Value |
| ----------------- | --------------- |
| `Multi-config` | N = quantity |
| `CI Environment` | 1 = yes, 0 = no |
| `LLM Environment` | 1 = yes, 0 = no |

We might need to extend this as we continue.

## Example payload

```json
{
"dab_version": "1.2.3",
"command": "add",
"flags": ["--source", "--permissions", "--description"],
"app_name": "dab_oss_1.2.3+xxx|123|123|123+",
"multi_config": 1,
"ci_env": 1,
"llm_env": 1,
"os": "windows",
"duration_ms_bucket": "100-500",
"success": true
}
```

### Duration buckets:

* `<10`
* `10-100`
* `100-500`
* `500-1000`
* `>1000`

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.