Kaggle / Kaggle/docker-python

Additional ext4-backed mount point for high-throughput I/O caching (besides `/kaggle/working`)

未關閉
#1,506 0 則留言 1 個 reaction 已指派 0 人 在 GitHub 檢視
enhancement
主要語言
Python
星號
2.7k
分支
1k
平均合併
7 天 14 小時
30 天內合併 PR
2

描述

## 🚀 Feature

Provide an additional mount point backed by a physical device (`ext4`), separate from `/kaggle/working`, to use as a high-throughput I/O cache (e.g., `/tmp`).

## Motivation

* The container root (`/`) appears to be provided via an overlay filesystem; write/metadata performance on overlay can be slower or less predictable than on a host-backed volume such as `/kaggle/working`.
* Some applications default to `/tmp` for I/O-intensive caches; on overlay this default can become an I/O bottleneck.
* `/kaggle/working` is relatively fast and can be an alternatives to `/tmp`, but its capacity is limited to \~20 GB per session, which is sometimes insufficient to host checkpoint of LLMs (\~100GB).
* An additional fast endpoint for caching has the potential to better utilize the instance’s compute resources (e.g., T4, P100) that are often bottlenecked by disk I/O.

## Additional context

I benchmarked sequential I/O and observed both writes and reads to be slower on `/tmp` (overlay) than on `/kaggle/working` (host-backed). Write throughput on `/tmp` is especially unstable; it sometimes gets ~5–10× lower in my tests.

* [🕵️ I/O Performance Test in Kaggle Kernel](https://www.kaggle.com/code/tatamikenn/i-o-performance-test-in-kaglle-kernel)

貢獻指南

這個儲存庫沒有索引到貢獻指南

研究方向

先檢視 issue 的 I/O 基準測試和現有的容器掛載設定;報告中未指出任何實作檔案或測試。將由實體裝置支援的額外 ext4 掛載與 /kaggle/working 分開定義,然後驗證它是否提供更大、更快的快取路徑,同時不改變現有的掛載行為。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
docker, linux
領域
infrastructure, operating-systems
Issue 類型
功能
難度
5/5
預估耗時
一週以上
活躍度
停滯
描述清晰度
基本清楚
新手友好度
25/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。