lincc-frameworks / lincc-frameworks/iops-profiler
Some reads not captured on Linux
- Dominant language
- Python
- Stars
- 6
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
**Bug report**
I have [a 1 GiB file](https://huggingface.co/datasets/MultimodalUniverse/desi/blob/main/edr_sv3/train-00000-of-00007.parquet), and I'm getting different results when I read it with standard Python tooling and pyarrow; pyarrow bytes read are unrealistically small.
```
with open('train-00000-of-00007.parquet', 'rb') as gh:
%iops data = gh.read()
del data
```
```
======================================================================
IOPS Profile Results (strace (per-process))
======================================================================
Execution Time: 18.2150 seconds
Read Operations: 2
Write Operations: 0
Total Operations: 2
Bytes Read: 1.02 GB (1,091,305,162 bytes)
Bytes Written: 0.00 B (0 bytes)
Total Bytes: 1.02 GB (1,091,305,162 bytes)
----------------------------------------------------------------------
IOPS: 0.11 operations/second
Throughput: 57.14 MB/second
======================================================================
```
```
import pyarrow.parquet as pq
%iops pq.read_table('train-00000-of-00007.parquet')
```
```
======================================================================
IOPS Profile Results (strace (per-process))
======================================================================
Execution Time: 19.7621 seconds
Read Operations: 3
Write Operations: 3
Total Operations: 6
Bytes Read: 3.63 MB (3,808,731 bytes)
Bytes Written: 13.05 KB (13,360 bytes)
Total Bytes: 3.65 MB (3,822,091 bytes)
----------------------------------------------------------------------
IOPS: 0.30 operations/second
Throughput: 188.87 KB/second
======================================================================
```
I tried to do `sync; sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'`, but it didn't help.
**Environment Information**
Linux 6.8.0, x86_64, etx4, python 3.13, pyarrow 23, iops_profiler 0.2.0, ipython 9.9.0
**Before submitting**
Please check the following:
- [x] I have described the situation in which the bug arose, including what code was executed, and any applicable data others will need to reproduce the problem.
- [x] I have included information about my environment, including the version of this package (e.g. `iops_profiler.__version__`)
- [x] I have included available evidence of the unexpected behavior (including error messages, screenshots, and/or plots) as well as a description of what I expected instead.
- [ ] If I have a solution in mind, I have provided an explanation and/or pseudocode and/or task list.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.