Akegarasu / Akegarasu/lora-scripts

[Bug Report] Critical Performance Bottleneck: max_data_loader_n_workers is Hardcoded to 0, Severely Limiting High-End Hardware

Abierto
#742 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
6.1k
Forks
699
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

#### **1. Environment (环境信息)**

* **CPU:** AMD Ryzen 7 5800X
* **GPU:** NVIDIA RTX 5080
* **RAM:** 32GB
* **OS:** [Windows 10 22H2]
* **Software Version:** [v1.13.0]

#### **2. Summary (问题概述)**

The GUI forcefully sets `max_data_loader_n_workers` to 0, creating a severe data-loading bottleneck that cripples the performance of modern hardware. My recent tests provide definitive proof: by simply increasing the `train_batch_size`, I was able to increase my training speed from **13 seconds/iteration** to just **2 seconds/iteration**. This demonstrates that the bottleneck is not in GPU computation, but in the inefficient, single-threaded data preparation pipeline, a direct result of `workers` being locked to 0.

GUI强制将 `max_data_loader_n_workers` 设置为0,这造成了严重的数据加载瓶颈,完全压制了现代硬件的性能。我最近的测试提供了决定性的证据:**仅通过增大 `train_batch_size`,我的训练速度就从 13秒/步 提升到了 2秒/步**。这无可辩驳地证明了瓶颈不在于GPU的计算能力,而在于效率低下的单线程数据准备流程——这正是 `workers` 被锁定为0的直接后果。

#### **3. The Experiment: Definitive Proof of a DataLoader Bottleneck (实验:数据加载器瓶颈的铁证)**

I conducted a series of tests, keeping all settings identical except for `train_batch_size` and `gradient_accumulation_steps`. The total effective batch size was kept constant at 8.

我进行了一系列测试,除了批次大小和梯度累积步数外,所有设置都保持不变。总的有效批次大小始终保持为8。

* **Test 1:**
* `train_batch_size = 1`
* `gradient_accumulation_steps = 8`
* **Result: ~13 s/it** (极慢)
* *Analysis: The single-threaded loader struggles to prepare even a single image, causing the GPU to wait for an extremely long time for each tiny batch.*

* **Test 2:**
* `train_batch_size = 2`
* `gradient_accumulation_steps = 4`
* **Result: ~11 s/it** (略有提升)
* *Analysis: The loader's overhead is slightly more amortized over two images, but the bottleneck remains severe.*

* **Test 3 (The Smoking Gun):**
* `train_batch_size = 4`
* `gradient_accumulation_steps = 0` (parameter removed, defaults to 1)
* **Result: ~2 s/it** (性能大幅提升!)
* *Analysis: By forcing the CPU to prepare a larger batch of 4 images at once, the GPU has more work to do per step. This longer GPU processing time effectively **hides the latency** of the slow, single-threaded CPU loader which is preparing the *next* batch in the background. A more efficient pipeline is formed, but it's a workaround, not a solution.*

**This 650% performance increase, achieved simply by changing the batching strategy, proves that the system is entirely bottlenecked by the data loader's high per-batch overhead, a classic symptom of `max_data_loader_n_workers=0`.**

**这个仅通过改变批处理策略就获得的650%的性能提升,证明了整个系统完全受限于数据加载器极高的“每批次固定开销”,这是 `max_data_loader_n_workers=0` 最典型的症状。**

#### **4. Expected Behavior (期望的行为)**

With `max_data_loader_n_workers` correctly set to a higher value (e.g., `8`), the CPU bottleneck would be eliminated. The training speed should be consistently fast (e.g., ~1-2 s/it) regardless of the batch size strategy, limited only by VRAM and the GPU's actual compute power.

当 `max_data_loader_n_workers` 被正确设置为一个更高的值(例如`8`)时,CPU瓶颈将被彻底消除。训练速度应该能够稳定地保持在高速水平(例如~1-2秒/步),而不再受批处理策略的影响,其唯一的限制应该是显存和GPU的实际计算能力。

#### **5. Full Training Configuration Used in Test 3 (测试3中使用的完整配置):**

```toml
[model_arguments]
model_train_type = "sdxl-lora"
pretrained_model_name_or_path = "E:/CustomApps/AIPainting/sd-webui-forge-aki-v1.0/models/Stable-diffusion/illustriousXL_v01.safenet..."

[dataset_arguments]
train_data_dir = "E:/CustomApps/AIPainting/lora_webui/train/Chishima_mako"
...
[training_arguments]
train_batch_size = 4
gradient_accumulation_steps = 1 # This was changed
...
max_data_loader_n_workers = 8 # This is the critical parameter being ignored.
...
(The rest of the configuration is the same as provided previously)
```

#### **6. Conclusion & Suggested Solution (结论与建议)**

The evidence is conclusive: the GUI is hardcoding `max_data_loader_n_workers=0`, creating a severe and unnecessary performance bottleneck that prevents users from utilizing their hardware.

**Solution:** Please expose the `max_data_loader_n_workers` parameter as a configurable integer field within the GUI, for example, under the "Advanced" or "Dataset" settings. This is a simple change that will unlock massive performance gains for your users. Thank you.

证据是决定性的:GUI硬编码了 `max_data_loader_n_workers=0`,制造了一个严重且不必要的性能瓶颈,阻碍了用户有效利用他们的硬件。

**解决方案:** 请在GUI中将 `max_data_loader_n_workers` 参数作为一个可配置的选项暴露出来(例如,放在“高级”或“数据集”设置下的一个数字输入框)。这是一个简单的改动,但它将为您的用户解锁巨大的性能提升。谢谢。

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.