apache / apache/texera

Self join with cache will cause repeated results

Open
#2,258 3 comments 0 reactions 1 assignee Assigned to @Yicong-Huang View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

For a workflow with a self-join (see example below), it will have incorrect output reading from cache.

Say normally the join operator would take 1K input tuples, because the 500 output tuples from distinct is joining themselves (500 + 500 = 1K).
![CleanShot 2023-12-10 at 19 14 15](https://github.com/Texera/texera/assets/17627829/23ce2427-1d43-449c-927b-3e2e006e9fe2)

However, if we cache the result of the distinct operator, the next run from the cache will result in more input data (more than 1K) for the join:
![CleanShot 2023-12-10 at 19 17 23](https://github.com/Texera/texera/assets/17627829/2c0d7078-bdd7-4df6-b021-0720fdcfcf6a)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.