Akegarasu / Akegarasu/lora-scripts

[Bug Report] Critical Performance Bottleneck: max_data_loader_n_workers is Hardcoded to 0, Severely Limiting High-End Hardware

Offen
#742 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
6.1k
Forks
699
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

#### **1. Environment (环境信息)**

* **CPU:** AMD Ryzen 7 5800X
* **GPU:** NVIDIA RTX 5080
* **RAM:** 32GB
* **OS:** [Windows 10 22H2]
* **Software Version:** [v1.13.0]

#### **2. Summary (问题概述)**

The GUI forcefully sets `max_data_loader_n_workers` to 0, creating a severe data-loading bottleneck that cripples the performance of modern hardware. My recent tests provide definitive proof: by simply increasing the `train_batch_size`, I was able to increase my training speed from **13 seconds/iteration** to just **2 seconds/iteration**. This demonstrates that the bottleneck is not in GPU computation, but in the inefficient, single-threaded data preparation pipeline, a direct result of `workers` being locked to 0.

GUI强制将 `max_data_loader_n_workers` 设置为0,这造成了严重的数据加载瓶颈,完全压制了现代硬件的性能。我最近的测试提供了决定性的证据:**仅通过增大 `train_batch_size`,我的训练速度就从 13秒/步 提升到了 2秒/步**。这无可辩驳地证明了瓶颈不在于GPU的计算能力,而在于效率低下的单线程数据准备流程——这正是 `workers` 被锁定为0的直接后果。

#### **3. The Experiment: Definitive Proof of a DataLoader Bottleneck (实验:数据加载器瓶颈的铁证)**

I conducted a series of tests, keeping all settings identical except for `train_batch_size` and `gradient_accumulation_steps`. The total effective batch size was kept constant at 8.

我进行了一系列测试,除了批次大小和梯度累积步数外,所有设置都保持不变。总的有效批次大小始终保持为8。

* **Test 1:**
* `train_batch_size = 1`
* `gradient_accumulation_steps = 8`
* **Result: ~13 s/it** (极慢)
* *Analysis: The single-threaded loader struggles to prepare even a single image, causing the GPU to wait for an extremely long time for each tiny batch.*

* **Test 2:**
* `train_batch_size = 2`
* `gradient_accumulation_steps = 4`
* **Result: ~11 s/it** (略有提升)
* *Analysis: The loader's overhead is slightly more amortized over two images, but the bottleneck remains severe.*

* **Test 3 (The Smoking Gun):**
* `train_batch_size = 4`
* `gradient_accumulation_steps = 0` (parameter removed, defaults to 1)
* **Result: ~2 s/it** (性能大幅提升!)
* *Analysis: By forcing the CPU to prepare a larger batch of 4 images at once, the GPU has more work to do per step. This longer GPU processing time effectively **hides the latency** of the slow, single-threaded CPU loader which is preparing the *next* batch in the background. A more efficient pipeline is formed, but it's a workaround, not a solution.*

**This 650% performance increase, achieved simply by changing the batching strategy, proves that the system is entirely bottlenecked by the data loader's high per-batch overhead, a classic symptom of `max_data_loader_n_workers=0`.**

**这个仅通过改变批处理策略就获得的650%的性能提升,证明了整个系统完全受限于数据加载器极高的“每批次固定开销”,这是 `max_data_loader_n_workers=0` 最典型的症状。**

#### **4. Expected Behavior (期望的行为)**

With `max_data_loader_n_workers` correctly set to a higher value (e.g., `8`), the CPU bottleneck would be eliminated. The training speed should be consistently fast (e.g., ~1-2 s/it) regardless of the batch size strategy, limited only by VRAM and the GPU's actual compute power.

当 `max_data_loader_n_workers` 被正确设置为一个更高的值(例如`8`)时,CPU瓶颈将被彻底消除。训练速度应该能够稳定地保持在高速水平(例如~1-2秒/步),而不再受批处理策略的影响,其唯一的限制应该是显存和GPU的实际计算能力。

#### **5. Full Training Configuration Used in Test 3 (测试3中使用的完整配置):**

```toml
[model_arguments]
model_train_type = "sdxl-lora"
pretrained_model_name_or_path = "E:/CustomApps/AIPainting/sd-webui-forge-aki-v1.0/models/Stable-diffusion/illustriousXL_v01.safenet..."

[dataset_arguments]
train_data_dir = "E:/CustomApps/AIPainting/lora_webui/train/Chishima_mako"
...
[training_arguments]
train_batch_size = 4
gradient_accumulation_steps = 1 # This was changed
...
max_data_loader_n_workers = 8 # This is the critical parameter being ignored.
...
(The rest of the configuration is the same as provided previously)
```

#### **6. Conclusion & Suggested Solution (结论与建议)**

The evidence is conclusive: the GUI is hardcoding `max_data_loader_n_workers=0`, creating a severe and unnecessary performance bottleneck that prevents users from utilizing their hardware.

**Solution:** Please expose the `max_data_loader_n_workers` parameter as a configurable integer field within the GUI, for example, under the "Advanced" or "Dataset" settings. This is a simple change that will unlock massive performance gains for your users. Thank you.

证据是决定性的:GUI硬编码了 `max_data_loader_n_workers=0`,制造了一个严重且不必要的性能瓶颈,阻碍了用户有效利用他们的硬件。

**解决方案:** 请在GUI中将 `max_data_loader_n_workers` 参数作为一个可配置的选项暴露出来(例如,放在“高级”或“数据集”设置下的一个数字输入框)。这是一个简单的改动,但它将为您的用户解锁巨大的性能提升。谢谢。

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.