Decathlon / Decathlon/internal-developer-platform

[Perf][P3] EAV I/O tax — re-benchmark at 10x scale before considering hybrid typed-columns model

Open
#133 0 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
Java
Stars
9
Forks
0
Avg merge
3d 1h
Merged PRs (30d)
5

Description

## Context

From `idp-v2-vs-app-referential-analysis.md`, §4/§5/§6, and consistent with ADR-013's "measurement-gated" framing for a denormalised read view.

IDP v2 stores every entity attribute as a row in a generic EAV (Entity-Attribute-Value) `property` table (VARCHAR values regardless of logical type), joined via `entity_properties`. app-referential instead uses typed columns for frequently-filtered attributes (`kind`, `type`, `tiering`, `business_score`) plus JSONB for the long tail — a hybrid model.

## Evidence

At ~4,000 entities / ~6,500 relationships (same dataset injected into both systems):

| Query | IDP v2 (EAV) | app-referential (typed columns) |
|---|---|---|
| Filtered list, low selectivity (18/953 match) | ~3.1 ms, **966 buffer page hits** | ~3.6 ms, 670 buffer hits (seq scan, no JSONB index) |
| Filtered list, high selectivity (1,042/3,972 match) | ~4.0 ms, **3,652 buffer page hits** | ~3.5 ms, 670 buffer hits |

**Wall-clock SQL time is nearly identical, but IDP v2 needs ~5.4x more buffer/page reads** to answer the same question. At this scale everything fits in Postgres's shared buffers so the extra I/O is invisible in latency — but it will not stay invisible as data grows past cache-resident size, or under concurrent load competing for buffer cache/CPU.

Flyway migrations V3.3 and V3.6 (GIN-trigram + `lower()` functional indexes) are evidence this is already a recognized problem being patched at the margins rather than redesigned.

## Recommended fix (do NOT implement yet — see gating condition below)

Promote the handful of consistently-present, frequently-filtered properties (per the benchmark dataset: `type`, `tiering`, `businessScore`, `domain`/`subdomain`) to real typed columns on `entity` (nullable, template-specific), keeping the EAV tables for genuinely dynamic/rare attributes. This is a hybrid model — exactly what app-referential already does — without losing the "flexible schema" value proposition for the long tail of attributes. Also add targeted expression/functional indexes for filters actually used in production, continuing the V3.3/V3.6 direction, rather than generic trigram indexes for every property.

## Priority / gating condition

**P3 — explicitly deferred.** The raw-SQL evidence does **not** justify this at current scale; the buffer-cost multiplier (4–5x) is a leading indicator worth monitoring, not an active bottleneck. Per the analysis's recommendation: re-run this benchmark at 10x–100x the current entity count (and under concurrent load) after P1 (#131, ORM/REST overhead) is addressed, to decide whether this hybrid-columns change is still needed. Feed the buffer-touch numbers into ADR-013 as the measurement gating this decision.

## Related

Do not start this until P1 (#131) is done and re-benchmarked — most of today's latency is ORM/REST overhead, not this schema cost.

Contributor guide

Open the contributing guide

Research direction

Start with idp-v2-vs-app-referential-analysis.md §§4–6 and ADR-013, then check the status of P1 issue #131 before rerunning the benchmark. Measure the current workload at 10x–100x entity scale and under concurrent load, recording buffer touches alongside latency. Done means the results are added to ADR-013 and used to decide whether the hybrid typed-column change is still warranted.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, sql
Domain
databases, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.