duckdb / duckdb/duckdb-python

Translate unsupported fsspec modification times into a typed DuckDB error

Abierto
#584 2 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
186
Forks
113
Merge medio
13 h 29 min
PR fusionados (30 d)
17

Descripción

## What happens?

When a filesystem registered through `Connection.register_filesystem()` cannot provide a modification time, querying the `last_modified` column of `read_blob` surfaces the raw Python exception instead of returning `NULL`.

One concrete case is `gcsfs`: `GCSFileSystem.modified()` indexes `info(path)["mtime"]`, while some GCS object metadata (and synthesized directory entries) has no `mtime`. The resulting `KeyError: 'mtime'` currently reaches a DuckDB query as an untyped error and can abort callers such as DuckLake `CHECKPOINT`.

This is the Python/fsspec half of:

- duckdb/ducklake#1042
- duckdb/ducklake#1336

## Why the error loses its type

`PythonFilesystem::GetLastModifiedTime()` currently calls `filesystem.modified()` directly. A Python exception crosses the nanobind boundary as `std::exception`; when DuckDB later constructs `ErrorData(ex)`, it has `ExceptionType::INVALID`, so extensions cannot distinguish unsupported modification-time metadata from unrelated I/O failures without inspecting the message.

DuckDB core already has the desired downstream behavior: `DirectFileReader` converts `ExceptionType::NOT_IMPLEMENTED` failures for optional file metadata columns to `NULL`. An `older_than` predicate then naturally retains only files whose age is unknown.

## Suggested direction

Translate the Python filesystem capability error at the adapter boundary in `PythonFilesystem::GetLastModifiedTime()`:

1. Catch `nb::python_error`.
2. Map Python `NotImplementedError` to DuckDB `NotImplementedException`.
3. Consider a narrowly scoped compatibility mapping for the current gcsfs `KeyError('mtime')`, or coordinate with gcsfs so missing `mtime` is reported as `NotImplementedError`.
4. Re-throw every unrelated Python exception unchanged.

Please add a regression test with a small custom fsspec filesystem whose `modified()` is unsupported, asserting that `read_blob(...).last_modified` is `NULL` while other filesystem errors still surface.

This keeps provider-specific exception classification in the Python filesystem adapter instead of requiring downstream extensions to accumulate message-matching helpers.

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

Comienza en PythonFilesystem::GetLastModifiedTime() y sigue cómo DirectFileReader gestiona los errores de metadatos opcionales. Añade una prueba de regresión usando un sistema de archivos fsspec personalizado con un comportamiento modified() no compatible y ejecútala mediante read_blob(...).last_modified. El trabajo estará terminado cuando las horas de modificación no compatibles devuelvan NULL, mientras que los errores del sistema de archivos no relacionados sigan propagándose.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
cpp, python
Área
database, testing
Tipo de issue
Error
Dificultad
4/5
Tiempo estimado
3-5 días
Estado de actividad
Activo
Claridad
Bastante claro
Aptitud para principiantes
68/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.