kohya-ss / kohya-ss/sd-scripts

WebDataset support

Open
#1,478 0 comments 5 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.2k
Forks
1.2k
Avg merge
11m
Merged PRs (30d)
2

Description

[WebDataset](https://github.com/webdataset/webdataset) is commonly used in large-scale training due to its efficiency and flexibility. The WebDataset format consists of simple tar files where samples share the same basename and have any number of associated extensions (e.g., jpg and txt within the same tar file). This structure allows for faster data access compared to loading individual files, and WebDatasets can even be streamed directly from HTTP or cloud storage.

While the most significant benefits of WebDataset are seen in large-scale training, these advantages also apply to smaller-scale training setups, such as those using kohya_ss.

I propose implementing support for WebDataset to leverage these efficiency gains. I'm happy to work on the implementation and answer any questions. Gauging interest would be helpful, so please let me know if this is something you'd like to see.

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names no repository files, tests, or entry points. Start by locating the existing dataset-loading path, then review the WebDataset tar structure and assess how samples, HTTP or cloud streaming, and associated extensions should fit; done means the project can train from WebDataset inputs.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.