[BUG] Loading from large dataframe/large numpy requires holding all chunks in coordinator
Open
bug
component:coordinator
good first issue
- Dominant language
- C++
- Stars
- 3.6k
- Forks
- 468
- Avg merge
- 29m
- Merged PRs (30d)
- 1
Description
**Describe the bug**
It looks strange that we need to accumulate all chunks in the request stream into a list in _coordinator_ before sending to analytical engine, that would requires large available memory for the coordinator pod.
https://github.com/alibaba/GraphScope/blob/b80a35599424580325a750e734f8a3b2dead2a5b/coordinator/gscoordinator/dag_manager.py#L77-L107
Contributor guide
Assessment
This issue has not been assessed yet.