hashicorp / hashicorp/vault

`database.{NAME}.*` telemetry is keyed by plugin type, not by mount or connection name (docs inaccuracy + request for per-mount labels)

Open
#32,077 0 comments 1 reaction 0 assignees View on GitHub
core/metric secret/database
Dominant language
Go
Stars
36.3k
Forks
4.8k
PR merge metrics
PR metrics pending

Description

## Summary

Two related things:

1. **Docs bug** — [Database telemetry metrics](https://developer.hashicorp.com/vault/docs/internals/telemetry/metrics/database) describes the `{NAME}` in `database.{NAME}.*` as the named secrets engine. It is actually the **database plugin type string**, and for PostgreSQL it isn't even `postgresql` — it's `pgx`.
2. **Feature request** — there is currently no way to break database secrets engine telemetry down per mount or per configured connection. I'd like that.

## The documentation is inaccurate

The page currently states that enabling a PostgreSQL secrets engine called `postgresql-prod` produces `database.postgresql-prod.CreateUser.error`, and describes `database.{NAME}.Close` as the time to close "the database secrets engine {NAME}".

Both are wrong. `{NAME}` is neither the mount path nor the connection name, and the PostgreSQL plugin's type string is `pgx`. The real metric is:

```
database.pgx.CreateUser.error
```

### Where it comes from

`{NAME}` is `typeStr`, obtained from the plugin itself in `sdk/database/dbplugin/v5/plugin_factory.go` (~L87):

```go
typeStr, err := db.Type()
...
// Wrap with metrics middleware
db = &databaseMetricsMiddleware{
next: db,
typeStr: typeStr,
}
```

`sdk/database/dbplugin/v5/middleware.go` emits it directly, with no labels:

```go
metrics.MeasureSince([]string{"database", mw.typeStr, "Close"}, now)
```

The connection name never reaches the factory. In `builtin/logical/database/backend_ce.go` the connection name (`name`) is in scope and used for the connection cache, but only `config.PluginName` is forwarded:

```go
dbw, err := newDatabaseWrapper(ctx, config.PluginName, pluginVersion, b.System(), b.logger)
```

### Reproduction

I confirmed this by running the engine with an in-memory telemetry sink: mounted the database secrets engine, registered the `postgresql-database-plugin`, and created a connection named `zzz-my-distinctive-connection-name`. Every metric emitted:

```
counter database.Close
counter database.Initialize
counter database.pgx.Close
counter database.pgx.Initialize
sample database.Close
sample database.Initialize
sample database.pgx.Close
sample database.pgx.Initialize

key contains connection name "zzz-my-distinctive-connection-name" : false
key contains "pgx" : true
key contains "postgresql-database-plugin" : false
```

Vault's own debug log in that run:

```
[DEBUG] got database plugin instance: type=pgx
[DEBUG] created database object: name=zzz-my-distinctive-connection-name plugin_name=postgresql-database-plugin
```

The connection name is known to Vault at that exact moment and still does not appear.

### Actual `{NAME}` values

Verified by executing each plugin's `Type()`:

| Plugin | `{NAME}` |
| --- | --- |
| `postgresql-database-plugin` | `pgx` |
| `mysql-database-plugin` | `mysql` |
| `mssql-database-plugin` | `mssql` |
| `mongodb-database-plugin` | `mongodb` |
| `cassandra-database-plugin` | `cassandra` |

Read from source constants but not executed: `hana` → `hdb`, `influxdb` → `influxdb`, `redshift` → `redshift`. The redshift plugin's constant carries a comment saying it exists specifically for how the plugin appears in metrics middleware.

### Practical consequence

Every PostgreSQL connection, across every mount and every namespace, aggregates into a single `database.pgx.*` series. Operators reading the current docs will build dashboards expecting per-mount series that will never exist.

## Feature request — per-mount and per-connection labels

### Problem

I want to answer questions like "which mount is generating credential churn" and "is `orders-primary` slower than `reporting-replica`". Today that's impossible: the only dimension available is plugin type.

### Proposed solution

Attach `mount_point` and `connection_name` as **labels**, gated behind a new telemetry option, off by default:

```hcl
telemetry {
add_mount_point_database_metrics = true
}
```

Resulting emission:

```
# default (option absent or false) — unchanged from today
database.Initialize
database.pgx.Initialize

# with the option enabled
database.Initialize;mount_point=db-prod/;connection_name=orders-primary
database.pgx.Initialize;mount_point=db-prod/;connection_name=orders-primary
```

### Why labels rather than name segments

`vault.rollback.attempt.{MOUNT_POINT}` and `vault.route.rollback.{MOUNT_POINT}` were deliberately replaced with unsuffixed names because mount points in metric *names* explode Prometheus series counts, with `add_mount_point_rollback_metrics` added as the opt-in escape hatch. Repeating that pattern here would recreate the same problem and break existing queries. Labels leave metric names untouched, so current dashboards keep working and aggregate exactly as before.

### Why opt-in

Series count scales with mounts × connections × operations. Both closest precedents (`add_mount_point_rollback_metrics`, `add_lease_metrics_namespace_labels`) are booleans defaulting to false, so this follows the established convention.

Contributor guide

Open the contributing guide

Research direction

Start with the database telemetry metrics documentation, then read sdk/database/dbplugin/v5/plugin_factory.go and middleware.go to trace typeStr into emitted metrics. Compare that flow with builtin/logical/database/backend_ce.go, where the connection name is in scope, and verify that the documentation matches the observed plugin-type metrics while default telemetry remains unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, documentation, observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.