python / python/cpython

Add converter and formatter parameters to csv.reader and csv.writer

Abierto
#155,097 0 comentarios 1 reacción 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

extension-modules type-feature
Lenguaje dominante
Python
Estrellas
77.2k
Forks
35.9k
Métricas de merge de PR
Métricas de PR pendientes

Descripción

Feature or enhancement

Proposal:

The conversion between Python values and CSV fields is hard-coded in both directions. The reader converts unquoted fields with float(), and only in the QUOTE_NONNUMERIC and QUOTE_STRINGS modes. The writer converts every non-string value with str().

This is the common cause of several open issues:

  • gh-74232 -- bool is written unquoted as True, which cannot be read back.
  • gh-98485 -- the same for complex; Fraction and IntEnum are affected too, and Decimal silently round-trips through float.
  • gh-110852 -- there is no way to write floats with a fixed precision, because preformatted strings are quoted in the QUOTE_NONNUMERIC mode.
  • gh-85002 -- there is no way to reject values which are neither strings nor numbers.

I propose two parameters, mirroring parse_float in json:

  • csv.reader(f, converter=None) -- called as converter(index, field) instead of float().
  • csv.writer(f, formatter=None) -- called as formatter(index, value) instead of str(). It must return a string.

index is the 0-based position of the field in the record. Both default to None, which keeps the current behavior. The hooks only replace the existing calls -- what is not passed to float() or str() now is not passed to them either. Quoting is still decided by the original value.

The index goes first, like in enumerate(). This also makes a wrong one-argument callable fail at once: converter=int raises TypeError on the first field instead of taking the index as the base.

The index makes the hooks per-column, which is what the dtype and converters parameters of pandas.read_csv() are used for:

>>> types = [str, int, Decimal, Fraction]
>>> list(csv.reader(['spam,42,1.10,1/2'], quoting=csv.QUOTE_NONNUMERIC,
...                 converter=lambda i, field: types[i](field)))
[['spam', 42, Decimal('1.10'), Fraction(1, 2)]]
>>> def money(index, value):
...     return format(value, '.2f') if index == 2 else str(value)
>>> csv.writer(sys.stdout, formatter=money).writerow(['a', 1, 0.0, 3.14159])
a,1,0.00,3.14159

gh-85002 no longer needs a parameter of its own -- a strict writer is a formatter which refuses everything except numbers.

I have a working prototype (about 90 lines in Modules/_csv.c).

Open question: should these be parameters of the reader and the writer, or attributes of the dialect? A dialect is a portable description of the file syntax -- it is registered under a global name, sniffed, and copied -- so keeping callables out of it seems better.

Linked PRs
  • gh-155099

Guía de contribución

Abrir la guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Línea de trabajo

Comienza con Modules/_csv.c y los puntos de entrada de reader/writer descritos en la propuesta; revisa las rutas de conversión existentes de float() y str(), así como el prototipo mencionado allí. Compara las alternativas de parámetros y dialectos y, después, utiliza el trabajo enlazado gh-155099 para determinar la API acordada y los criterios de finalización.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
python
Área
data
Tipo de issue
Nueva funcionalidad
Dificultad
5/5
Tiempo estimado
Más de una semana
Estado de actividad
Estancado
Claridad
Bastante claro
Aptitud para principiantes
25/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.