Additional ext4-backed mount point for high-throughput I/O caching (besides `/kaggle/working`)
- Dominant language
- Python
- Stars
- 2.7k
- Forks
- 1k
- Avg merge
- 7d 14h
- Merged PRs (30d)
- 2
Description
## 🚀 Feature
Provide an additional mount point backed by a physical device (`ext4`), separate from `/kaggle/working`, to use as a high-throughput I/O cache (e.g., `/tmp`).
## Motivation
* The container root (`/`) appears to be provided via an overlay filesystem; write/metadata performance on overlay can be slower or less predictable than on a host-backed volume such as `/kaggle/working`.
* Some applications default to `/tmp` for I/O-intensive caches; on overlay this default can become an I/O bottleneck.
* `/kaggle/working` is relatively fast and can be an alternatives to `/tmp`, but its capacity is limited to \~20 GB per session, which is sometimes insufficient to host checkpoint of LLMs (\~100GB).
* An additional fast endpoint for caching has the potential to better utilize the instance’s compute resources (e.g., T4, P100) that are often bottlenecked by disk I/O.
## Additional context
I benchmarked sequential I/O and observed both writes and reads to be slower on `/tmp` (overlay) than on `/kaggle/working` (host-backed). Write throughput on `/tmp` is especially unstable; it sometimes gets ~5–10× lower in my tests.
* [🕵️ I/O Performance Test in Kaggle Kernel](https://www.kaggle.com/code/tatamikenn/i-o-performance-test-in-kaglle-kernel)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the issue’s I/O benchmark and the existing container mount configuration; no implementation files or tests are named in the report. Define the additional physical-device-backed ext4 mount separately from /kaggle/working, then verify that it provides a larger, faster cache path without changing the existing mount behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, linux
- Domain
- infrastructure, operating-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100