AcademySoftwareFoundation / AcademySoftwareFoundation/OpenImageIO
many open files can use lots of memory
- Dominant language
- C++
- Stars
- 2.4k
- Forks
- 698
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 47
Description
If we set the `max_open_files` very high (and of course make sure the OS is setup to support this many files), then loading in, for instance, 50,000 textures can use up 5.5GB of unaccounted memory even if the texture cache stays small. I believe the reason for this is that each open texture stores data that is not counted against the texture cache limit. For instance, if we look at OpenEXR (note, similar overhead was seen for both tiff and exr), we see in its `close()` method that closing exrs release a bunch of memory:
```
bool
OpenEXRInput::close ()
{
delete m_input_multipart;
delete m_scanline_input_part;
delete m_tiled_input_part;
delete m_deep_scanline_input_part;
delete m_deep_tiled_input_part;
delete m_input_scanline;
delete m_input_tiled;
delete m_input_stream;
init (); // Reset to initial state
return true;
}
```
I don't know if this makes up the source of the majority of the overhead or if maybe the overhead comes from some other bookkeeping in OIIO, but at least it helps to indicate that unaccounted overhead does exist in OIIO.
I'm not sure what the proper way of handling this is. One option is to factor this overhead into the texture cache. The tricky part would be to figure out when it's better to close a file handle and when it's better to evict cache data (maybe you'd want to close files that have the least amount of data in the cache or files that have been open longest without being used?). Another option is to let OIIO continue to use up the extra memory, but at least have it tabulate this memory overhead and include this memory usage in the logs it produces so that users can know what is causing the 5.5GB of unaccounted memory.
Contributor guide
Research direction
Reproduce the 50,000-texture case with a high max_open_files value, then inspect the texture cache and open-texture handling alongside OpenEXRInput::close(). Measure which memory remains outside the cache and determine whether the issue should account for that overhead or change file-closing and eviction behavior; done means the 5.5GB discrepancy is explained and the chosen behavior is covered by tests or logs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100