deepspeedai / deepspeedai/DeepSpeed

[BUG] libaio on amd node

Open
#6,972 3 comments 0 reactions 3 assignees View on GitHub

@loadams is already working on this.

Since Jan 25, 2025.

bug training
Dominant language
Python
Stars
43.1k
Forks
5k
Avg merge
4d 15h
Merged PRs (30d)
112

Description

Hi, I installed libaio as

apt install libaio-dev

And I can see both .so and .h exist

root@b6410ec8bb69:/code/DeepSpeed# find / -name "libaio.so*" 2>/dev/null
/usr/lib/x86_64-linux-gnu/libaio.so.1
/usr/lib/x86_64-linux-gnu/libaio.so
/usr/lib/x86_64-linux-gnu/libaio.so.1.0.1
root@b6410ec8bb69:/code/DeepSpeed# find / -name "libaio.h" 2>/dev/null
/usr/include/libaio.h

And I did setup flags as:

echo 'export CFLAGS="-I/usr/include"' >> ~/.bashrc
echo 'export LDFLAGS="-L/usr/lib"' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH="/usr/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH"' >> ~/.bashrc
source ~/.bashrc

but when I do ds_report, it says async_io is not compatible

JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
fused_adam ............. [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_lion ............... [NO] ....... [OKAY]
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
evoformer_attn ......... [NO] ....... [NO]
fp_quantizer ........... [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
fused_lion ............. [NO] ....... [OKAY]
 [WARNING]  gds is not compatible with ROCM
gds .................... [NO] ....... [NO]
transformer_inference .. [NO] ....... [OKAY]
inference_core_ops ..... [NO] ....... [OKAY]
cutlass_ops ............ [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
ragged_device_ops ...... [NO] ....... [OKAY]
ragged_ops ............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
 [WARNING]  sparse_attn is not compatible with ROCM
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch']
torch version .................... 2.3.0a0+gitd2f9472
deepspeed install path ........... ['/code/DeepSpeed/deepspeed']
deepspeed info ................... 0.16.2+unknown, unknown, unknown
torch cuda version ............... None
torch hip version ................ 6.2.41134-65d174c3e
nvcc version ..................... None
deepspeed wheel compiled w. ...... torch 2.3, hip 6.2
shared memory (/dev/shm) size .... 910.48 GB

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.