apache / apache/texera

Dataset Upload on Hub Website Is Significantly Slower Than Local Deployment

Open
#3,832 12 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### What happened?

The dataset upload on the Hub website is extremely slow compared to the local environment. When testing with a 5 GB file:

- Local (single-node, latest main branch): Upload completes in approximately 15 seconds.
- Hub website: Upload takes over 10 minutes (estimated).

The Hub deployment is currently 18 commits behind the latest main branch:

- Hub commit: b0075f6350e86ed363209350f2a31ca6a4b48b9a
- Local commit: 1c812a5e2d72a42a8b6698d286f41f618095a331

No code changes related to dataset upload were made between these commits. I suspect that the slowdown may be caused by environmental factors (e.g., network, deployment configuration, infrastructure).

### How to reproduce?

1. Upload a 5 GB dataset using the Hub website.
2. Upload the same 5 GB dataset using the local (single-node) environment on the latest main branch.
3. Compare the upload durations.

**Local Upload (Fast)**

https://github.com/user-attachments/assets/318786e6-1ec0-49ae-8d36-ecbaaf9e17b6

**Hub Upload (Slow)**

https://github.com/user-attachments/assets/937e5c7b-200e-49a6-8023-91f26547a1f7

### Version

1.1.0-incubating (Pre-release/Master)

### Commit Hash (Optional)

_No response_

### What browsers are you seeing the problem on?

_No response_

### Relevant log output

```shell

```

### Code of Conduct

- [x] I agree to follow Apache Code of Conduct

Contributor guide

Open the contributing guide

Research direction

No source file or test is identified. Start by comparing the Hub deployment with the local single-node upload path and deployment or network configuration at the two reported commits, then measure a 5 GB upload and inspect relevant logs. Done means identifying and resolving the bottleneck and confirming comparable upload performance.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
cloud, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.