kohya-ss / kohya-ss/sd-scripts
WebDataset support
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 1.2k
- Avg merge
- 11m
- Merged PRs (30d)
- 2
Description
[WebDataset](https://github.com/webdataset/webdataset) is commonly used in large-scale training due to its efficiency and flexibility. The WebDataset format consists of simple tar files where samples share the same basename and have any number of associated extensions (e.g., jpg and txt within the same tar file). This structure allows for faster data access compared to loading individual files, and WebDatasets can even be streamed directly from HTTP or cloud storage.
While the most significant benefits of WebDataset are seen in large-scale training, these advantages also apply to smaller-scale training setups, such as those using kohya_ss.
I propose implementing support for WebDataset to leverage these efficiency gains. I'm happy to work on the implementation and answer any questions. Gauging interest would be helpful, so please let me know if this is something you'd like to see.
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names no repository files, tests, or entry points. Start by locating the existing dataset-loading path, then review the WebDataset tar structure and assess how samples, HTTP or cloud streaming, and associated extensions should fit; done means the project can train from WebDataset inputs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100