apache / apache/datafusion

[DISCUSSION] Reduce the DataFusion WASM size

Open
#16,554 7 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

I have heard annecedotally a few times from different people that the size of the compiled WASM file for DataFusion is large (10MB?). This large size slows down page load times for webpages that use the DataFusion WASM.

I have heard this concern from @emilk @timsaucer @XiangpengHao at least Maybe @jonmmease or @waynexia too (but I can't remember for sure)

### Describe the solution you'd like

I would like some way to reduce the size of the final WASM package (perhaps optionally)

### Describe alternatives you've considered

Today when you build DataFusion it comes with many features (like support for all compression libraries for Parquet and functions specialized for multiple different types of Arrow arrays -- `StringArray`, and `LargeStringArray` and `StringViewArray` for example

One thing we could consider is feature flagging out many of the extra copies of code for different data formats somehow.

I think the challenge will be managing the complexity that this adds to build/ runtime.

### Additional context

- Epic WASM: #13815

Contributor guide

Open the contributing guide

Research direction

Start with Epic WASM #13815 and the current DataFusion WASM build, then measure the compiled package and its compression-library and Arrow-array feature contributions. Done means a concrete, optionally smaller build is defined and its size and runtime/build complexity are evaluated.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, wasm
Domain
build-system, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.