kvcache-ai / kvcache-ai/AgentENV
OverlayBD direct I/O hard-codes 512-byte alignment and fails with EINVAL on 4096-byte-sector block devices
- Dominant language
- Rust
- Stars
- 3.5k
- Forks
- 309
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 66
Description
### Affected component
OverlayBD / image resolution
### AgentENV version
- Version or commit: `main` at b0d649c41112217ae3354eb189a6fb9996e5529b (2026-09-10)
### Host environment
- OS: Ubuntu 24.04 (Depot CI runner image `depot-ubuntu-24.04`)
- Kernel (`uname -a`): `6.12.8+` x86_64
- Architecture: x86_64
- Bare metal, VM, or cloud: cloud VM (Depot-hosted GitHub Actions runner), KVM available
- ublk support: yes
- CPU: 8 vCPU
- Memory: 31 GiB
- Storage/filesystem: root is ext4 on `/dev/mapper/root` (dm thin pool over virtio disks `vda`/`vdb`). The block device reports **4096-byte logical and physical sectors**:
```
$ lsblk -t
NAME ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED RQ-SIZE RA WSAME
vda 0 262144 2097152 4096 4096 1 mq-deadline 256 4096 0B
└─thinpool 0 262144 2097152 4096 4096 1 4096 0B
└─root 0 262144 2097152 4096 4096 1 4096 0B
$ findmnt -no SOURCE,FSTYPE,OPTIONS --target /tmp
/dev/mapper/root ext4 rw,noatime,lazytime,nobarrier
```
### Operation and configuration
```shell
# Test suite (TMPDIR defaults to /tmp on the 4096-byte-sector filesystem)
make test-ublk PROFILE=debug
# Minimal syscall-level reproduction of the same failure, no AgentENV involved
fio --name=p --directory=/tmp --ioengine=io_uring --direct=1 --bs=512 --size=1M --rw=write --iodepth=4
fio --name=p --directory=/tmp --ioengine=libaio --direct=1 --bs=512 --size=1M --rw=write --iodepth=4
fio --name=p --directory=/tmp --ioengine=psync --direct=1 --bs=512 --size=1M --rw=write --iodepth=4
# The same commands with --bs=4096 succeed.
```
Production exposure with default configuration is limited: `LocalFile::direct_io(true)` is only used for lower layers when the image config selects `ioEngine` libaio (`image_file.rs`, `direct_io = io_engine == IO_ENGINE_LIBAIO`), plus the lsmt direct-I/O paths. On such a host, enabling libaio makes every lower-layer open fail with `Invalid argument`, and the test suite always fails.
### Steps to reproduce
1. Use a Linux host whose filesystem for `TMPDIR` sits on a block device with `blockdev --getss` = 4096 (4K-native NVMe namespaces and some cloud/virtio disks.
2. Check out `main` and run `make test-ublk PROFILE=debug`.
3. Observe the O_DIRECT tests in `storage/overlaybd` fail with `EINVAL`/`EIO`.
4. Optionally run the `fio` commands above: 512-byte direct I/O fails, 4096-byte direct I/O succeeds, on any of the io_uring, libaio, or psync engines.
### Expected behavior
`storage/overlaybd/src/backend/local.rs` should derive the direct-I/O alignment from the backing device instead of hard-coding it:
```rust
const DIRECT_IO_ALIGNMENT: usize = 512;
```
Linux (ext4/xfs via iomap) rejects O_DIRECT requests unless offset, length and buffer are aligned to the device's logical block size (`bdev_logical_block_size`), which is 4096 on these hosts. The backend could query `BLKSSZGET` on the underlying device (or use `statx`'s `stx_dio_mem_align`/`stx_dio_offset_align` on kernels ≥ 6.1, which report exactly these limits for a file) when opening a direct-I/O file, and size `AlignedBuffer` and the offset/length checks from that value. Failing that, opening with `direct_io(true)` on such a device should return a clear error naming the required alignment rather than surfacing raw `EINVAL` on the first write.
### Actual behavior
Eight tests fail, every direct-I/O write or open returns `EINVAL` (or `EIO` from the lsmt create path):
```
test backend::local::tests::test_write_bytes_at_direct_io ... FAILED
test backend::local::tests::test_write_direct_at_aligned_ptr_fast_path ... FAILED
test backend::local::tests::test_write_direct_at_unaligned_ptr_bounce ... FAILED
test image::image_file::tests::test_image_file_open_local_lower_with_libaio_io_engine ... FAILED
test lsmt::file::tests::test_direct_io_close_seal ... FAILED
test lsmt::file::tests::test_direct_io_create_open ... FAILED
test lsmt::file::tests::test_direct_io_create_sparse ... FAILED
test lsmt::file::tests::test_direct_io_write_and_read ... FAILED
thread 'backend::local::tests::test_write_bytes_at_direct_io' panicked at storage/overlaybd/src/backend/local.rs:1060:44:
write_bytes_at: Invalid argument (os error 22)
thread 'backend::local::tests::test_write_direct_at_aligned_ptr_fast_path' panicked at storage/overlaybd/src/backend/local.rs:977:14:
aligned write: Invalid argument (os error 22)
thread 'backend::local::tests::test_write_direct_at_unaligned_ptr_bounce' panicked at storage/overlaybd/src/backend/local.rs:1007:14:
unaligned-ptr write via bounce buffer: Invalid argument (os error 22)
thread 'image::image_file::tests::test_image_file_open_local_lower_with_libaio_io_engine' panicked at storage/overlaybd/src/image/image_file.rs:3004:14:
open image with libaio: failed to open lower layer 0
Invalid argument (os error 22)
thread 'lsmt::file::tests::test_direct_io_close_seal' panicked at storage/overlaybd/src/lsmt/file/tests.rs:3172:10:
create direct-io LSMTFile failed: Input/output error (os error 5)
thread 'lsmt::file::tests::test_direct_io_write_and_read' panicked at storage/overlaybd/src/lsmt/file/tests.rs:3207:38:
direct-io write: Invalid argument (os error 22)
test result: FAILED. 310 passed; 8 failed; 1 ignored; 0 measured; 0 filtered out
```
### Logs and diagnostics
```
$ fio --name=p --directory=/tmp --ioengine=io_uring --direct=1 --bs=512 --size=1M --rw=write --iodepth=4
fio: io_u error on file /tmp/p.0.0: Invalid argument: write offset=0, buflen=512
fio: io_u error on file /tmp/p.0.0: Invalid argument: write offset=512, buflen=512
$ fio --name=p --directory=/tmp --ioengine=libaio --direct=1 --bs=512 --size=1M --rw=write --iodepth=4
fio: io_u error on file /tmp/p.0.0: Invalid argument: write offset=0, buflen=512
$ fio --name=p --directory=/tmp --ioengine=psync --direct=1 --bs=512 --size=1M --rw=write --iodepth=4
fio: io_u error on file /tmp/p.0.0: Invalid argument: write offset=0, buflen=512
# python3: os.pwrite on an O_DIRECT fd with a page-aligned buffer
/tmp: pwrite 512 aligned -> EINVAL
/tmp: pwrite 4096 aligned -> ok (4096)
# Same tests on a 512-byte-sector loop device mounted over /tmp on the same host
$ truncate -s 24G img && losetup --find --show --sector-size 512 img && mkfs.ext4 -q -F /dev/loopN && mount /dev/loopN /tmp
$ make test-ublk PROFILE=debug
test result: ok. 318 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out
```
### Reproduction frequency
Always
### Regression information
- Last known good: not a regression as far as I can tell, the constant has been 512 since the direct-I/O backend was introduced. The tests pass on hosts whose device reports 512-byte logical sectors (GitHub-hosted runners, 512e NVMe).
- First known bad: n/a
### Pre-submission checklist
- [x] I searched existing open and closed issues and did not find a duplicate.
- [x] I provided a minimal reproduction or explained why one cannot be provided.
- [x] I removed credentials, tokens, private image references, and other sensitive information.
- [x] This is not a security vulnerability.
Contributor guide
Research direction
Start in storage/overlaybd/src/backend/local.rs at DIRECT_IO_ALIGNMENT and the direct-I/O write tests, then inspect the libaio path in image/image_file.rs and the direct-I/O tests in lsmt/file/tests.rs. Run make test-ublk PROFILE=debug on a 4096-byte-sector filesystem and compare with the provided 512-byte loop-device result. Done means the eight listed tests pass without EINVAL or EIO on both sector sizes, with alignment errors reported clearly if needed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux, rust
- Domain
- backend, operating-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 67/100