Kaggle / Kaggle/docker-python

Additional ext4-backed mount point for high-throughput I/O caching (besides `/kaggle/working`)

Abierto
#1,506 0 comentarios 1 reacción 0 asignados Ver en GitHub
enhancement
Lenguaje dominante
Python
Estrellas
2.7k
Forks
1k
Merge medio
7 d 14 h
PR fusionados (30 d)
2

Descripción

## 🚀 Feature

Provide an additional mount point backed by a physical device (`ext4`), separate from `/kaggle/working`, to use as a high-throughput I/O cache (e.g., `/tmp`).

## Motivation

* The container root (`/`) appears to be provided via an overlay filesystem; write/metadata performance on overlay can be slower or less predictable than on a host-backed volume such as `/kaggle/working`.
* Some applications default to `/tmp` for I/O-intensive caches; on overlay this default can become an I/O bottleneck.
* `/kaggle/working` is relatively fast and can be an alternatives to `/tmp`, but its capacity is limited to \~20 GB per session, which is sometimes insufficient to host checkpoint of LLMs (\~100GB).
* An additional fast endpoint for caching has the potential to better utilize the instance’s compute resources (e.g., T4, P100) that are often bottlenecked by disk I/O.

## Additional context

I benchmarked sequential I/O and observed both writes and reads to be slower on `/tmp` (overlay) than on `/kaggle/working` (host-backed). Write throughput on `/tmp` is especially unstable; it sometimes gets ~5–10× lower in my tests.

* [🕵️ I/O Performance Test in Kaggle Kernel](https://www.kaggle.com/code/tatamikenn/i-o-performance-test-in-kaglle-kernel)

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.