matrixorigin / matrixorigin/matrixone

[Subtask]: Establish an evidence-based JSON/VARIANT baseline for agent observability workloads

Open
#27,374 0 comments 0 reactions 1 assignee Claimed by @daviszhen View on GitHub
area/observability attention/tech-doc-required kind/subtask needs-more-tests phase/testing priority/p0
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

### Parent Issue

Parent issue: #23360

### Detail of Subtask

MatrixOne already exposes a `JSON` SQL type backed by a binary `ByteJson` representation, JSON path functions/operators, `UNNEST`, and JSON-capable full-text indexing. However, issue #23360 proposes JSON shredding and accelerated querying without a reproducible workload baseline that shows which semantic or physical gaps require that design.

This subtask establishes an evidence-based JSON/VARIANT capability baseline for semi-structured analytics and agent-observability workloads. It is a validation and decision task. It must not assume that adding a `VARIANT` alias, implementing shredding, or changing the type system is the correct solution before the evidence exists.

#### Claims and invariants

1. **Ingestion integrity**: every accepted NDJSON document is queryable with its required values and types preserved; every rejected document has an explicit error. Silent row loss, truncation, or type corruption negates the claim.
2. **Semi-structured semantics**: missing paths, JSON `null`, SQL `NULL`, mixed scalar types, nested objects, arrays, and schema evolution behave according to a frozen SQL contract. Conflating those states or changing results because object keys are reordered negates the claim.
3. **Query correctness**: filtering, projection, aggregation, unnesting, trace lookup, and text retrieval match an independent expected-result corpus. A query merely completing without an exact/differential oracle does not satisfy the claim.
4. **Evidence completeness**: unsupported queries, timeouts, OOMs, rejected rows, and invalid runs remain visible. Skipping them while reporting a green aggregate result negates the claim.
5. **Mixed workload recoverability**: under bounded concurrent ingestion and query load, acknowledged source records are neither lost nor duplicated, and a restart/retry converges to the same result as the fault-free reference run.

#### Public reachability

All externally visible claims must be exercised through the MySQL protocol and supported public SQL paths:

```text
NDJSON / INSERT / LOAD DATA
-> MatrixOne JSON column
-> JSON path extraction / UNNEST / aggregation / full-text retrieval
-> typed result rows and persisted side effects
```

Package-level `ByteJson` tests are supporting white-box evidence only; they cannot replace the public-path witness.

#### Required work

- [ ] Freeze the MatrixOne commit/image, SQL surface, dataset manifest, query corpus, expected results, topology, and resource fingerprint.
- [ ] Produce a capability matrix for JSON ingestion, path semantics, type fidelity, schema evolution, aggregation, unnesting, and full-text retrieval.
- [ ] Add a small MatrixOne-owned deterministic fixture covering missing/NULL states, mixed types, deep nesting, arrays, large values, key reordering, and path evolution.
- [ ] Map every scored JSONBench and agent-observability query to MatrixOne SQL, after code and dataset licenses are approved.
- [ ] Validate every mapped query against an independent exact, differential, or metamorphic oracle.
- [ ] Record unsupported syntax/capabilities explicitly; do not rewrite dynamic payload fields into predeclared top-level columns merely to make the workload pass.
- [ ] Collect cold/hot runtimes, load acceptance counts, storage footprint, peak RSS, CPU, scan volume, spill, timeout, and OOM classifications.
- [ ] Run a bounded ingestion-plus-query recovery test with retry and service restart.
- [ ] Use plans/profiles to identify whether slow cases are dominated by parsing, full scans, JSON path evaluation, text retrieval, aggregation, spill, or result materialization.
- [ ] Feed the resulting gap analysis back to #23360.

#### Decision output

The final report must distinguish these possible conclusions instead of collapsing them into `VARIANT supported=true`:

- Existing `JSON` is sufficient for the target workload; documentation and best practices are the remaining gaps.
- Existing `JSON` is semantically sufficient, but explicit generated/materialized paths or path indexes are needed.
- Automatic shredding/subcolumnization from #23360 is justified by measured hot-path and scale evidence.
- A genuinely distinct VARIANT logical type is required because the needed value/type semantics cannot be represented by the existing JSON contract.

### Describe implementation you've considered

Start with the smallest black-box correctness fixture and a benchmark adapter. Do not modify production type/storage code in this subtask. If a gap is confirmed, open or update a separate implementation issue with the smallest counterexample, owning boundary, expected hot-path cost, and dedicated regression matrix.

Suggested execution stages:

1. 10K MatrixOne-owned fixture for adapter and semantic debugging.
2. 1M public benchmark tier for complete query correctness.
3. 10M same-host, host-exclusive runs for scale and profile evidence.
4. Distributed 6-24 hour ingestion/query recovery validation.

### Acceptance Criteria

- [ ] Every declared query is `passed`, `unsupported`, `timeout`, `oom`, or `infra_invalid`; no query disappears from the denominator.
- [ ] Passed queries have independent result evidence, not only exit code or runtime.
- [ ] Missing path, JSON null, SQL NULL, type drift, array/object, malformed input, large value, and schema-evolution counterexamples are covered.
- [ ] Benchmark/config/data/MO revisions and environment fingerprints are immutable and recorded.
- [ ] At least three valid same-host runs are retained for performance claims; diagnostic/shared-host results are not promoted to formal evidence.
- [ ] Mixed ingestion/query restart and replay produce the same business result as the no-fault reference.
- [ ] The report states what current JSON proves, what remains unknown, and which measured gaps (if any) justify #23360.
- [ ] User-facing documentation gaps and support-version boundaries are listed.

### Additional information

Related public references:

- #23360
- Apache Doris VARIANT: https://doris.apache.org/docs/dev/key-features/variant-data-type/
- Snowflake semi-structured types: https://docs.snowflake.com/en/sql-reference/data-types-semistructured
- JSONBench: https://github.com/ClickHouse/JSONBench
- AgentLogsBench: https://github.com/velodb/agentlogsbench

License review is a prerequisite for copying benchmark code/data or publishing derived adapters/results. A public repository without an explicit license must not be treated as implicitly reusable.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.