matrixorigin / matrixorigin/matrixone

[Bug]: ISCP index consumer deadlocks after downstream SQL execution fails

Open
#25,632 4 comments 0 reactions 1 assignee Assigned to @jiangxinmeng1 View on GitHub
ai-easy kind/bug needs-triage severity/s0
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Summary

`IndexConsumer.Consume` can deadlock permanently after the downstream index SQL executor returns an error. The execution goroutine reports the error and exits, but the producer continues flushing to an unbuffered `sqlBufSendCh` with no remaining receiver.

This affects asynchronous index maintenance, including FULLTEXT and IVF_FLAT; HNSW uses the same producer/channel pattern with a different consumer loop.

## Version

Commit: `7f7de555c`

## Code path

`pkg/iscp/index_consumer.go`:

```go
func (c *IndexConsumer) sendSql(writer IndexSqlWriter) error {
// ...
c.sqlBufSendCh <- sql
writer.Reset()
return nil
}
```

The send does not select on `ctx.Done()` or the error channel. In `runIndex`, an `ExecWithResult` error is sent to `errch` and the goroutine returns. The producer does not read `errch` while generating data, so its next `sendSql` blocks forever.

There is a second error-propagation bug at the end of `sinkTail`:

```go
c.flushCdc()
return nil
```

Any final flush error is discarded.

## White-box reproduction

A focused test used a writer that flushes after every row and an `ExecWithResult` stub that fails the first SQL with `injected index SQL failure`:

1. The producer sends row 1.
2. `runIndex` receives it, the executor returns the injected error, `runIndex` writes `errch` and exits.
3. The producer processes row 2 and blocks sending to `sqlBufSendCh` because no receiver exists.

The test runs `Consume` in a goroutine and requires the injected error within two seconds.

## Actual result

```text
Consume deadlocked after the SQL executor returned an error
```

The original executor error is never returned to the caller.

## Expected result

- A downstream execution error must cancel/stop the producer and be returned by `Consume` promptly.
- Channel sends must select on context cancellation and consumer failure.
- `sinkTail` must return `flushCdc()` errors.
- Closing or abandoning the channel must not leave either goroutine blocked.

## Suggested verification

Cover execution failure on the first and middle batch for snapshot and tail consumers, final-flush failure, context cancellation during a blocked send, and the HNSW consumer variant.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.