huggingface / huggingface/datasets
"eval_strategy": "no" perform evaluation
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Describe the bug
Hi,
I'm trying to train a diffusion model and I getting stuck with
```
strace -p 967873
strace: Process 967873 attached
ioctl(9, _IOC(_IOC_READ|_IOC_WRITE, 0x46, 0x2a, 0x20), 0x7fff18579d90) = 0
ioctl(9, _IOC(_IOC_READ|_IOC_WRITE, 0x46, 0x2a, 0x20), 0x7fff18579d90) = 0
```
But my issue is that it hangs when it tries to do eval. So I decided to `eval_strategy: "no"` to skip that previous issue but the code doesn't respect the fact that I explicitly aske to skip eval.
config.yaml
base_model: meta-llama/Llama-3.2-1B
# Automatically upload checkpoint and final model to HF
# hub_model_id: username/custom_model_name
# Continue training after requeueing.
# NOTE: resume_from_checkpoint must be empty
resume_from_checkpoint:
auto_resume_from_checkpoints: true
# Setting `pretraining_dataset` without explicitly setting `streaming: true` is deprecated.
# In a future release, streaming will not be automatically enabled when using pretraining_dataset.
# Please explicitly set `streaming: true` in your configuration to maintain current behavior.
# NOTE: Streaming currently only supports a single dataset. Multi-dataset support will be added soon.
streaming: false
# Streaming-specific settings
streaming_multipack_buffer_size: 10000 # NOTE: 50000 required >300GB RAM WTF?!
shuffle_merged_datasets: true
# For pretraining: control with pretrain_multipack_attn
pretrain_multipack_attn: true # prevent cross-attention between packed samples
dataset_prepared_path: last_run_prepared
- path: A.jsonl
ds_type: json
type: completion
- path: B.jsonl
ds_type: json
type: completion
- path: C.jsonl
ds_type: json
type: completion
- path: D.jsonl
ds_type: json
type: completion
eval_strategy: "no"
test_datasets:
- path: E.jsonl
ds_type: json
type: completion
plugins:
- axolotl.integrations.diffusion.DiffusionPlugin
diffusion:
noise_schedule: cosine
min_mask_ratio: 0.15
max_mask_ratio: 0.85
num_diffusion_steps: 128
eps: 5e-4
importance_weighting: true
mask_token_id: 128002
# NOTE: It looks like it is trying to run eval on train_dataloader().
generate_samples: true
# generate_samples: false
generation_interval: 500 # same as the save frequency
output_dir: ./outputs/model-out
sequence_len: 512
sample_packing: true
gradient_accumulation_steps: 4
micro_batch_size: 32
# NOTE: max_steps must be set when using streaming datasets.
# Trainer cannot infer dataset length for iterable datasets.
# max_steps: 100000
num_epochs: 3.0
warmup_ratio: 0.1
# Optimizer and scheduler
optimizer: adamw_8bit
# optimizer: adamw_torch_8bit
lr_scheduler: cosine
learning_rate: 3e-4
attn_implementation: sdpa
# Precision and performance
bf16: auto
tf32: true
# Logging and checkpointing
logging_steps: 5
save_strategy: steps
save_steps: 500
save_total_limit: 3
# Special tokens
special_tokens:
pad_token: "<|end_of_text|>"
use_tensorboard: true
wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:
load_best_model_at_end: false
# save_first_step: true # uncomment this to validate checkpoint saving works with your config
# fsdp_version: 2
# fsdp_config:
# offload_params: true
# state_dict_type: FULL_STATE_DICT
# auto_wrap_policy: TRANSFORMER_BASED_WRAP
# transformer_layer_cls_to_wrap: LlamaDecoderLayer
# reshard_after_forward: true
### Steps to reproduce the bug
```sh
accelerate launch \
--multi_gpu \
--num_processes="$SLURM_GPUS_ON_NODE" \
--num_machines="$SLURM_NNODES" \
--mixed_precision=bf16 \
-m axolotl.cli.train \
"$config" \
--output-dir="$output_dir" \
--dataloader-num-workers="$SLURM_CPUS_PER_TASK" \
--dataset-num-proc="$SLURM_TASKS_PER_NODE" \
--dataset-processes="$SLURM_TASKS_PER_NODE" \
--deepspeed=deepspeed_configs/zero1.json
```
### Expected behavior
The eval step shouldn't run.
### Environment info
Slurm cluster, using single node 4 GPUs and `deepspeed_configs/zero1.json`.
uv pip freeze
```
absl-py==2.4.0
accelerate==1.13.0
addict==2.4.0
adlfs==2026.4.0
aiobotocore==2.26.0
aiofiles==23.2.1
aiohappyeyeballs==2.6.1
aiohttp==3.13.5
aioitertools==0.13.0
aiosignal==1.4.0
annotated-doc==0.0.4
annotated-types==0.7.0
antlr4-python3-runtime==4.13.2
anyio==4.13.0
art==6.5
asttokens==3.0.1
attrs==26.1.0
autoawq==0.2.7.post3
-e file:///gpfs/work/larkins/TraLLM/axolotl/axolotl
axolotl-contribs-lgpl==0.0.7
axolotl-contribs-mit==0.0.6
azure-core==1.40.0
azure-datalake-store==0.0.53
azure-identity==1.25.3
azure-storage-blob==12.28.0
backoff==2.2.1
bitsandbytes==0.49.1
botocore==1.41.5
brotli==1.2.0
cbor2==6.0.1
certifi==2026.4.22
cffi==2.0.0
chardet==6.0.0.post1
charset-normalizer==3.4.7
circuitbreaker==2.1.3
click==8.3.3
cmake==4.3.2
colorama==0.4.6
coloredlogs==15.0.1
cryptography==46.0.7
cuda-bindings==13.2.0
cuda-pathfinder==1.5.4
cuda-toolkit==13.0.2
cut-cross-entropy @ git+https://github.com/axolotl-ai-cloud/ml-cross-entropy.git@fec1a888e6f4ad7e6270ea7b02186e56c76f5ac2
dataproperty==1.1.0
datasets==4.8.5
decorator==5.2.1
deepspeed==0.18.9
deepspeed-kernels==0.0.1.dev1698255861
dill==0.3.8
distro==1.9.0
einops==0.8.2
evaluate==0.4.1
executing==2.2.1
fastapi==0.136.1
fastcore==1.12.44
ffmpy==1.0.0
filelock==3.29.0
fire==0.7.1
fla-core==0.4.1
flash-linear-attention==0.4.1
frozenlist==1.8.0
fsspec==2025.3.0
gcsfs==2025.3.0
gitdb==4.0.12
gitpython==3.1.49
google-api-core==2.30.3
google-auth==2.50.0
google-auth-oauthlib==1.3.1
google-cloud-core==2.5.1
google-cloud-storage==3.10.1
google-crc32c==1.8.0
google-resumable-media==2.8.2
googleapis-common-protos==1.74.0
gradio==6.14.0
gradio-client==2.5.0
groovy==0.1.2
grpcio==1.80.0
grpclib==0.4.7
h11==0.16.0
h2==4.3.0
hf-gradio==0.4.1
hf-transfer==0.1.9
hf-xet==1.4.3
hjson==3.1.0
hpack==4.1.0
httpcore==1.0.9
httptools==0.7.1
httpx==0.28.1
huggingface-hub==1.13.0
humanfriendly==10.0
hyperframe==6.1.0
idna==3.13
immutabledict==4.2.0
ipython==9.13.0
ipython-pygments-lexers==1.1.1
isodate==0.7.2
jedi==0.20.0
jinja2==3.1.6
jmespath==1.1.0
joblib==1.5.3
jsonlines==4.0.0
jsonschema==4.26.0
jsonschema-specifications==2025.9.1
kernels==0.13.0
langdetect==1.0.9
liger-kernel==0.7.0
llvmlite==0.47.0
lm-eval==0.4.11
lxml==6.1.0
markdown==3.10.2
markdown-it-py==4.0.0
markupsafe==3.0.3
matplotlib-inline==0.2.2
mbstrdecoder==1.1.5
mdurl==0.1.2
mistral-common==1.11.0
modal==1.3.0.post1
more-itertools==11.0.2
mpmath==1.3.0
msal==1.36.0
msal-extensions==1.3.1
msgpack==1.1.2
multidict==6.7.1
multiprocess==0.70.16
networkx==3.6.1
ninja==1.13.0
nltk==3.9.4
numba==0.65.1
numexpr==2.14.1
numpy==2.3.5
nvidia-cublas==13.1.0.3
nvidia-cuda-cupti==13.0.85
nvidia-cuda-nvrtc==13.0.88
nvidia-cuda-runtime==13.0.96
nvidia-cudnn-cu13==9.19.0.56
nvidia-cufft==12.0.0.61
nvidia-cufile==1.15.1.6
nvidia-curand==10.4.0.35
nvidia-cusolver==12.0.4.66
nvidia-cusparse==12.6.3.3
nvidia-cusparselt-cu13==0.8.0
nvidia-ml-py==12.560.30
nvidia-nccl-cu13==2.28.9
nvidia-nvjitlink==13.0.88
nvidia-nvshmem-cu13==3.4.5
nvidia-nvtx==13.0.85
oauthlib==3.3.1
oci==2.173.1
ocifs==1.3.2
openenv-core==0.1.0
optimum==1.16.2
orjson==3.11.8
packaging==26.0
pandas==2.3.3
parso==0.8.7
pathvalidate==3.3.1
peft==0.19.1
pexpect==4.9.0
pillow==11.3.0
platformdirs==4.9.6
portalocker==3.2.0
posthog==6.7.11
prompt-toolkit==3.0.52
propcache==0.4.1
proto-plus==1.27.2
protobuf==5.29.6
psutil==7.2.2
ptyprocess==0.7.0
pudb==2025.1.5
pure-eval==0.2.3
py-cpuinfo==9.0.0
pyarrow==24.0.0
pyasn1==0.6.3
pyasn1-modules==0.4.2
pybind11==3.0.4
pycountry==26.2.16
pycparser==3.0
pydantic==2.10.6
pydantic-core==2.27.2
pydantic-extra-types==2.11.1
pydub==0.25.1
pygments==2.20.0
pyjwt==2.12.1
pyopenssl==26.2.0
pytablewriter==1.2.1
python-dateutil==2.9.0.post0
python-dotenv==1.0.1
python-multipart==0.0.27
pytz==2026.2
pyyaml==6.0.3
referencing==0.37.0
regex==2026.4.4
requests==2.33.1
requests-oauthlib==2.0.0
responses==0.18.0
rich==15.0.0
rouge-score==0.1.2
rpds-py==0.30.0
ruff==0.15.12
s3fs==2025.3.0
sacrebleu==2.6.0
safehttpx==0.1.7
safetensors==0.7.0
schedulefree==1.4.1
scikit-learn==1.4.2
scipy==1.17.1
semantic-version==2.10.0
sentencepiece==0.2.1
sentry-sdk==2.59.0
setuptools==81.0.0
shellingham==1.5.4
sigtools==4.0.1
six==1.17.0
smmap==5.0.3
sqlitedict==2.1.0
stack-data==0.6.3
starlette==0.52.1
sympy==1.14.0
synchronicity==0.11.1
tabledata==1.3.4
tabulate==0.10.0
tcolorpy==0.1.7
tensorboard==2.20.0
tensorboard-data-server==0.7.2
termcolor==3.3.0
threadpoolctl==3.6.0
tiktoken==0.12.0
tokenizers==0.22.2
toml==0.10.2
tomlkit==0.13.3
torch==2.11.0+cu130
torchao==0.17.0
torchvision==0.25.0+cu130
tqdm==4.67.3
tqdm-multiprocess==0.0.11
trackio==0.25.1
traitlets==5.15.0
transformers==5.5.4
triton==3.6.0
stack-data==0.6.3
starlette==0.52.1
sympy==1.14.0
synchronicity==0.11.1
tabledata==1.3.4
tabulate==0.10.0
tcolorpy==0.1.7
tensorboard==2.20.0
tensorboard-data-server==0.7.2
termcolor==3.3.0
threadpoolctl==3.6.0
tiktoken==0.12.0
tokenizers==0.22.2
toml==0.10.2
tomlkit==0.13.3
torch==2.11.0+cu130
torchao==0.17.0
torchvision==0.25.0+cu130
tqdm==4.67.3
tqdm-multiprocess==0.0.11
trackio==0.25.1
traitlets==5.15.0
transformers==5.5.4
triton==3.6.0
trl==1.1.0
typepy==1.3.5
typer==0.25.1
types-certifi==2021.10.8.3
types-toml==0.10.8.20260408
typing-extensions==4.15.0
typing-inspection==0.4.2
tzdata==2026.2
urllib3==2.6.3
urwid==4.0.0
urwid-readline==0.15.1
uvicorn==0.46.0
uvloop==0.22.1
wandb==0.26.1
watchfiles==1.1.1
wcwidth==0.7.0
websockets==15.0.1
werkzeug==3.1.8
word2number==1.1
wrapt==1.17.3
xformers==0.0.35
xxhash==3.7.0
yarl==1.23.0
zstandard==0.22.0
```
Contributor guide
Assessment
This issue has not been assessed yet.