Decathlon / Decathlon/internal-developer-platform
[Perf][P3] EAV I/O tax — re-benchmark at 10x scale before considering hybrid typed-columns model
- Dominant language
- Java
- Stars
- 9
- Forks
- 0
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 5
Description
## Context
From `idp-v2-vs-app-referential-analysis.md`, §4/§5/§6, and consistent with ADR-013's "measurement-gated" framing for a denormalised read view.
IDP v2 stores every entity attribute as a row in a generic EAV (Entity-Attribute-Value) `property` table (VARCHAR values regardless of logical type), joined via `entity_properties`. app-referential instead uses typed columns for frequently-filtered attributes (`kind`, `type`, `tiering`, `business_score`) plus JSONB for the long tail — a hybrid model.
## Evidence
At ~4,000 entities / ~6,500 relationships (same dataset injected into both systems):
| Query | IDP v2 (EAV) | app-referential (typed columns) |
|---|---|---|
| Filtered list, low selectivity (18/953 match) | ~3.1 ms, **966 buffer page hits** | ~3.6 ms, 670 buffer hits (seq scan, no JSONB index) |
| Filtered list, high selectivity (1,042/3,972 match) | ~4.0 ms, **3,652 buffer page hits** | ~3.5 ms, 670 buffer hits |
**Wall-clock SQL time is nearly identical, but IDP v2 needs ~5.4x more buffer/page reads** to answer the same question. At this scale everything fits in Postgres's shared buffers so the extra I/O is invisible in latency — but it will not stay invisible as data grows past cache-resident size, or under concurrent load competing for buffer cache/CPU.
Flyway migrations V3.3 and V3.6 (GIN-trigram + `lower()` functional indexes) are evidence this is already a recognized problem being patched at the margins rather than redesigned.
## Recommended fix (do NOT implement yet — see gating condition below)
Promote the handful of consistently-present, frequently-filtered properties (per the benchmark dataset: `type`, `tiering`, `businessScore`, `domain`/`subdomain`) to real typed columns on `entity` (nullable, template-specific), keeping the EAV tables for genuinely dynamic/rare attributes. This is a hybrid model — exactly what app-referential already does — without losing the "flexible schema" value proposition for the long tail of attributes. Also add targeted expression/functional indexes for filters actually used in production, continuing the V3.3/V3.6 direction, rather than generic trigram indexes for every property.
## Priority / gating condition
**P3 — explicitly deferred.** The raw-SQL evidence does **not** justify this at current scale; the buffer-cost multiplier (4–5x) is a leading indicator worth monitoring, not an active bottleneck. Per the analysis's recommendation: re-run this benchmark at 10x–100x the current entity count (and under concurrent load) after P1 (#131, ORM/REST overhead) is addressed, to decide whether this hybrid-columns change is still needed. Feed the buffer-touch numbers into ADR-013 as the measurement gating this decision.
## Related
Do not start this until P1 (#131) is done and re-benchmarked — most of today's latency is ORM/REST overhead, not this schema cost.
Contributor guide
Research direction
Start with idp-v2-vs-app-referential-analysis.md §§4–6 and ADR-013, then check the status of P1 issue #131 before rerunning the benchmark. Measure the current workload at 10x–100x entity scale and under concurrent load, recording buffer touches alongside latency. Done means the results are added to ADR-013 and used to decide whether the hybrid typed-column change is still warranted.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, sql
- Domain
- databases, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100