pytorch / pytorch/pytorch

Slow reading tensor meta-data

Open
#175,252 5 comments 0 reactions 0 assignees View on GitHub
bot-triaged module: performance module: serialization triaged
Dominant language
Python
Stars
103k
Forks
29.5k
PR merge metrics
PR metrics pending

Description

I opened a small PR https://github.com/pytorch/pytorch/pull/175239 to fix the following issue.

Consider the following loop that only reads meta-data of tensors:

```python
with safe_open(st_file, framework="pt", device="cpu") as f:
tensor_keys = f.keys()
for name in tensor_keys:
tensor = f.get_tensor(name)
total_bytes += tensor.element_size() * tensor.numel()
total_tensors += 1
del tensor
````

It is slow on systems where `read_ahead_kb` for a file system is configured to a large value, for example 32MiB. This is do to an unwanted data access. For example if I have a 5GB tensor files with 20 tensors, it will read a total of 640MiB from the file. When you have 100 of these files and you were only trying to read tensor meta-data for starting up, it would read 64,000MiB in total. That takes noticeable time.

Configuring a high value for `readahead_kb` increases performance when doing full serial load of tensors on some file systems. The files are already set with `posix_fadvise(fd, 0, static_cast(size), POSIX_FADV_SEQUENTIAL);` so we are telling the kernel that are going to do sequential, so it is allowing it to pick the large size.

The linked PR fixes the issue. Can we discuss it?

cc @jerryzh168 @mruberry @mikaylagawarecki

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.