deco-cx / deco-cx/deco

loader_cache uses the resolver chain as a metric label (3,684 series on one site, 21,849 fleet-wide)

Open
#1,219 0 comments 0 reactions 0 assignees View on GitHub
triage
Dominant language
TypeScript
Stars
847
Forks
54
Avg merge
5h 13m
Merged PRs (30d)
8

Description

## Measured

`blocks/loader.ts` labels the `loader_cache` counter with `ctx.resolverId`:

```ts
const loader = ctx.resolverId || "unknown";
```

`resolverId` carries the full resolver **chain**, so label values look like:

```
Categories@sections.variants.1.value.5.sections.0.section.page
SearchResults Global@sections.0.section.page
Categories@sections.variants.1.value.2.jsonLD
```

Every section position, in every page variant, on every page becomes a distinct series.

Production ClickHouse, distinct `loader` values in 3h:

| site | distinct values |
|---|---|
| fila-store | **3,684** |
| todolivrooficial | 3,478 |
| montecarlo | 2,209 |
| farmrio | 1,469 |
| happybooksoficial | 1,028 |
| **fleet total** | **21,849** |

Against the budget documented in our own o11y guide — >1,000 per site is forbidden as an alert dimension, >100 across the fleet as a cross-fleet aggregation — five sites are over, and the fleet figure is 218× over.

Cost, same window:

- `loader_cache` = **11,485,954 of 14,998,196** rows in `otel_metrics_sum` — **76.6%** of the whole table
- `otel_metrics_sum` on disk: **48.4 GB**

For contrast, `cache_hit` covers 194 tenants in **9 series**, and the `@decocms/start` runtime does the equivalent job with **37 series** and 13 distinct `deco.cache.profile` values (`vtex/productDetailsPage`, `vtex/relatedProducts`, `site/categoryTree`).

## Why this is an issue and not a PR

The obvious fix — label with the resolver name instead of the chain — **loses information**. The chain tells you *which section of which variant of which page* a loader instance sits in, and that is genuinely useful when answering "which section is slow". Truncating trades debuggability for cost, and that is a call for whoever owns the metric, not a mechanical patch.

Options worth weighing:

1. Label with the resolver name only (`vtex/loaders/legacy/productListingPage.ts`), matching the `@decocms/start` design. Cheapest, loses section attribution.
2. Keep the chain but only on a sampled/opt-in basis, with the name as the always-on label.
3. Keep the chain as a **span** attribute (where high cardinality is acceptable for point lookups) and use the name for the metric.

Option 3 preserves both properties and is what the o11y guide's own rule implies — high-cardinality attributes are fine for filtering and point lookups, forbidden as aggregation keys.

Happy to send the PR once there is a direction.

## Context

Surfaced while building unsampled loader-cache panels on the `general-overview-otel` dashboard. Those panels are usable today only with a tenant selected — a fleet-wide `GROUP BY` on this label is precisely the forbidden case.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Contributor guide

Open the contributing guide

Research direction

Start in blocks/loader.ts and inspect how ctx.resolverId becomes the loader_cache label. Read the o11y guide and compare the @decocms/start runtime metric design, then review the general-overview-otel dashboard context. Done requires an agreed direction that controls series cardinality while documenting what happens to resolver-chain attribution.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
observability
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.