Dataset Upload on Hub Website Is Significantly Slower Than Local Deployment
- Dominant language
- Scala
- Stars
- 314
- Forks
- 187
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 214
Description
### What happened?
The dataset upload on the Hub website is extremely slow compared to the local environment. When testing with a 5 GB file:
- Local (single-node, latest main branch): Upload completes in approximately 15 seconds.
- Hub website: Upload takes over 10 minutes (estimated).
The Hub deployment is currently 18 commits behind the latest main branch:
- Hub commit: b0075f6350e86ed363209350f2a31ca6a4b48b9a
- Local commit: 1c812a5e2d72a42a8b6698d286f41f618095a331
No code changes related to dataset upload were made between these commits. I suspect that the slowdown may be caused by environmental factors (e.g., network, deployment configuration, infrastructure).
### How to reproduce?
1. Upload a 5 GB dataset using the Hub website.
2. Upload the same 5 GB dataset using the local (single-node) environment on the latest main branch.
3. Compare the upload durations.
**Local Upload (Fast)**
https://github.com/user-attachments/assets/318786e6-1ec0-49ae-8d36-ecbaaf9e17b6
**Hub Upload (Slow)**
https://github.com/user-attachments/assets/937e5c7b-200e-49a6-8023-91f26547a1f7
### Version
1.1.0-incubating (Pre-release/Master)
### Commit Hash (Optional)
_No response_
### What browsers are you seeing the problem on?
_No response_
### Relevant log output
```shell
```
### Code of Conduct
- [x] I agree to follow Apache Code of Conduct
Contributor guide
Research direction
No source file or test is identified. Start by comparing the Hub deployment with the local single-node upload path and deployment or network configuration at the two reported commits, then measure a 5 GB upload and inspect relevant logs. Done means identifying and resolving the bottleneck and confirming comparable upload performance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- cloud, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100