matrixorigin / matrixorigin/matrixone

[Bug]: ODKU with synchronous FULLTEXT takes 12 minutes for 1,000 rows due to unconditional posting rebuild

Open
#27,933 11 comments 0 reactions 1 assignee Claimed by @Ariznawlll View on GitHub
area/performance kind/bug needs-more-tests phase/testing severity/s0
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Summary

A reported production-reachable `INSERT INTO ... ON DUPLICATE KEY UPDATE ...` statement takes about **12 minutes for only 1,000 input rows** when the target table has a synchronous FULLTEXT index. The table also contains a vector column, but no vector index is defined.

The exact deployed commit, table cardinality, fulltext token cardinality, document length, and statement evidence still need to be attached. However, current `main` (`33af6ae3c0c23e4f277137460577ea64678f7ecf`) contains a public-path mechanism that can explain pathological scaling:

1. every FULLTEXT index is unconditionally captured as an irregular index for INSERT/ODKU maintenance;
2. ODKU materializes the complete final row image;
3. synchronous FULLTEXT maintenance scans the hidden inverted-index table and joins it on `doc_id` to delete old postings;
4. the hidden table is clustered by `word`, not by `doc_id`;
5. the final rows are then re-tokenized and all postings are inserted again;
6. for ODKU, the ordinary UPDATE optimization that skips FULLTEXT maintenance when no indexed column changed is bypassed.

This means an ODKU that only changes a non-FULLTEXT column can still scan the existing inverted index, delete the conflicting documents' postings, tokenize their unchanged text again, and rewrite all postings.

The raw vector column is not an ANN-maintenance suspect when no vector index exists. It is still carried through the materialized final-row image and can increase row-copy/memory cost, but no vector hidden-table build/update should run.

## Severity rationale

Maintainer-requested project emergency: a common customer upsert path is effectively unusable at 1,000 rows / 12 minutes. This issue is intentionally marked `severity/s-1` pending immediate reproduction and owner assignment.

## Code evidence on current main

Pinned revision: `33af6ae3c0c23e4f277137460577ea64678f7ecf`.

- [`getIrregularIndexes`](https://github.com/matrixorigin/matrixone/blob/33af6ae3c0c23e4f277137460577ea64678f7ecf/pkg/sql/plan/bind_insert.go#L259-L274) includes every existing synchronous or asynchronous FULLTEXT index before the resolved ODKU action/update columns are considered.
- The ODKU path [materializes the final merged image and registers irregular-index maintenance](https://github.com/matrixorigin/matrixone/blob/33af6ae3c0c23e4f277137460577ea64678f7ecf/pkg/sql/plan/bind_insert.go#L2917-L2974).
- Maintenance [always schedules stale-entry deletion before re-insertion](https://github.com/matrixorigin/matrixone/blob/33af6ae3c0c23e4f277137460577ea64678f7ecf/pkg/sql/plan/bind_insert.go#L474-L497).
- The ODKU FULLTEXT delete branch [creates an unfiltered hidden-table scan and joins `doc_id` to the ODKU image](https://github.com/matrixorigin/matrixone/blob/33af6ae3c0c23e4f277137460577ea64678f7ecf/pkg/sql/plan/bind_insert.go#L732-L814). Unlike the generic FULLTEXT delete planner, this branch does not attach an exact runtime filter to the scan.
- The FULLTEXT hidden table is [`(doc_id, position, word, fake_pk)` and clustered by `word`](https://github.com/matrixorigin/matrixone/blob/33af6ae3c0c23e4f277137460577ea64678f7ecf/pkg/fulltext/plugin/plan/schema.go#L155-L230), so lookup/deletion by `doc_id` is not aligned with its clustering key.
- Re-insertion calls [`buildPreInsertFullTextIndex` with `updateColLength=0` and `updateColPosMap=nil`](https://github.com/matrixorigin/matrixone/blob/33af6ae3c0c23e4f277137460577ea64678f7ecf/pkg/sql/plan/bind_insert.go#L511-L543). That prevents the normal ["indexed columns unchanged" skip](https://github.com/matrixorigin/matrixone/blob/33af6ae3c0c23e4f277137460577ea64678f7ecf/pkg/sql/plan/build_dml_util.go#L6753-L6791) from firing and leads to [`fulltext_index_tokenize` plus hidden-table insertion](https://github.com/matrixorigin/matrixone/blob/33af6ae3c0c23e4f277137460577ea64678f7ecf/pkg/sql/plan/build_dml_util.go#L6813-L6978).

The synchronous maintenance design is related to #21769 (efficient update/delete through a docid map). #26173 is a separate large-LOAD manifestation of unbounded synchronous FULLTEXT maintenance. #26172 is a separate ODKU cold-read/S3 amplification incident; current `main` already contains cost-aware FileService protection, so it must not be assumed to be this incident's root cause without statement-level evidence.

## Actual behavior

- SQL shape: `INSERT INTO ... VALUES (...) ... ON DUPLICATE KEY UPDATE ...`
- Batch size: about 1,000 input rows
- Wall time: about 12 minutes
- Target has a FULLTEXT index
- Target has a vector column but no vector index
- Exact runtime SHA/version, total base rows, hidden FULLTEXT rows, average document bytes/tokens, conflict ratio, concurrency, topology, and object-store backend: **TBD**

## Expected behavior

- An ODKU that does not change any FULLTEXT-indexed column (and has no relevant `ON UPDATE` expression) must not delete and rebuild FULLTEXT postings.
- If FULLTEXT-indexed content changes, maintenance should scale with the number/size of changed documents and their tokens, not require a broad scan proportional to the whole hidden inverted index.
- A raw vector column without a vector index must not trigger ANN index maintenance.
- Correctness must remain exact: after commit, `MATCH ... AGAINST` returns the new document and never returns stale postings.

## Reproduction and counterexample matrix

Use the same frozen 1,000-row input, same conflict ratio, same seed, same host/CN, and same base-table snapshot for every run.

| Case | FULLTEXT | Vector column | ODKU assignment | Purpose |
|---|---|---|---|---|
| A | none | present, no vector index | non-indexed scalar column | ODKU/base-row control |
| B | synchronous | present, no vector index | same non-indexed scalar column | proves/disproves unconditional FULLTEXT maintenance |
| C | synchronous | absent | same non-indexed scalar column | isolates row-width/vector-copy cost |
| D | synchronous | present | FULLTEXT-indexed text column | required maintenance control |
| E | asynchronous | present | same text and scalar variants | separates foreground latency from CDC lag/catch-up |
| F | synchronous | present | no conflicts vs 100% conflicts | separates insert-only from stale-posting delete cost |

Repeat B and D against at least three hidden-index sizes while keeping the 1,000 changed document IDs and their text fixed. If latency/bytes scanned grow with total hidden-index size, the delete lookup is not bounded by changed documents.

Before each timed run, restore the same snapshot or recreate the database. Do not run cases sequentially on a mutating hidden table and compare them as if they were equivalent.

## Evidence to collect

### Statement evidence

Capture the exact `statement_id` and `transaction_id` from `system.statement_info`:

```sql
SELECT
statement_id,
transaction_id,
request_at,
response_at,
ROUND(duration / 1000000000, 3) AS duration_s,
status,
error,
rows_read,
bytes_scan,
stats,
exec_plan
FROM system.statement_info
WHERE request_at >= ''
AND statement_type = 'Insert'
AND statement LIKE '%ON DUPLICATE KEY UPDATE%'
ORDER BY request_at DESC;
```

In `exec_plan` / `EXPLAIN ANALYZE`, identify and record time, input/output rows, memory, and scanned bytes for:

- `DedupJoin` and the base-table `TableScan`;
- the `Table Scan` on `__mo_index_secondary_*`;
- the join on hidden `doc_id`;
- hidden-table `Delete`;
- `fulltext_index_tokenize` / `CROSS APPLY`;
- hidden-table `PreInsert` / `Insert`;
- base-table `MultiUpdate`;
- transaction commit.

Do not run `EXPLAIN ANALYZE` on the production statement unless executing the DML again is safe. Prefer the already exported `exec_plan`; use plain `EXPLAIN PHYPLAN` for a non-mutating shape check.

### CN raw logs

Using the statement/transaction ID and exact time window, search for:

- `BIG-TXN`, `workspace-size`, `statistical-size`, `actual-size` — token/posting write amplification and commit payload;
- `S3FS.Read`, `ioMerger.Merge wait expensive range`, `getReader`, `io.ReadAll` — hidden/base table cold reads or object-store tail latency;
- `lock wait on local`, `lock wait on local result`, `lock wait timeout`, `deadlock` — row/range lock contention;
- `wait-active`, `context canceled`, `commit`, `rollback`, `timeout` — transaction admission or commit delay;
- CN restart/OOM/panic evidence — distinguish slow completion from process failure.

Search by IDs first; keywords alone can include unrelated concurrent workloads.

### Metrics over the exact statement window

- `mo_txn_queue_size{type="active|wait-active|commit|lock-rpc"}`;
- `mo_txn_create_duration_seconds{type="wait-active"}` and lock wait histograms;
- `mo_txn_commit_duration_seconds{type="cn|cn-send|cn-resp|cn-wait-logtail|tn"}`;
- `mo_fs_read_write_duration{type="s3fs-read-total|s3fs-read-s3|get-reader|io-read-all|write"}`;
- `mo_fs_read_total{type="s3|hit-mem|hit-disk|hit-remote"}`;
- `mo_fs_s3_io_bytes{type="read|write"}`;
- CN CPU, RSS, Go heap, network, object-store GET/PUT, checkpoint/merge backlog.

Record deltas/rates for the statement window, not lifetime counters.

## Acceptance criteria

1. Add a regression in which ODKU changes only a non-FULLTEXT column; the generated plan contains no FULLTEXT hidden-table scan/delete/tokenize/insert, and `MATCH` results remain unchanged.
2. Keep a correctness regression for ODKU that changes indexed text: old terms disappear, new terms are searchable, replay does not duplicate postings, and rollback leaves both base and hidden tables unchanged.
3. Make stale-posting deletion bounded by the changed doc IDs. A fixed 1,000-document update must not scan work proportional to unrelated hidden-index size.
4. Add a deterministic benchmark over the A-F matrix with exact statement/operator evidence, at least three valid repeats, and no skipped/timeout/OOM results counted as passes.
5. Verify that a vector column without a vector index adds no vector-maintenance branch.
6. Verify synchronous and asynchronous FULLTEXT semantics separately; moving work to CDC is not by itself a correctness/performance fix unless lag, replay, failure recovery, and foreground contract are measured.

## Initial fix direction

The smallest safe first fix is to propagate the resolved ODKU update-column set into irregular-index maintenance and skip FULLTEXT delete/reinsert when none of that index's parts (including relevant generated/`ON UPDATE` dependencies) can change.

That does not solve changed-text deletion cost. For changed indexed text, evaluate a bounded `doc_id` lookup structure / docid map (#21769) or another physical design that does not scan a `word`-clustered posting table by `doc_id`. Do not mask the problem by only increasing timeouts, lowering batch size, or moving all maintenance to an unbounded asynchronous backlog.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.