meta-pytorch / meta-pytorch/data
[Stateful DataLoader] CI takes too long for Mac and WIndows
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 179
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 2
Description
🐛 Describe the bug
CI for Ubuntu / Linux in stateful dataloader takes about 20 seconds, but for mac and windows takes about 30+ minutes. Not only burning a huge amount of CI compute budget but slowing down iteration.
Linux probably runs so quickly because it's able to fork a process instead of spawning. The reason not to drop them is if the CI would have caught bugs that would not appear on Linux tests, and since we're not building native code in Stateful DataLoader, I'd assume it's unlikely.
Versions
Nightly
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the CI jobs that exercise Stateful DataLoader on Linux, macOS, and Windows, then compare their runtimes and process behavior. Done means identifying why the non-Linux jobs take 30+ minutes and reducing that cost without losing necessary platform coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ci-cd, data, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100