Consider using numpy loadtxt under the hood for fast ASCII reading
- Dominant language
- Python
- Stars
- 5.3k
- Forks
- 2.2k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 74
Description
### Description
Interesting progress from numpy who now have a C-based CSV parser built in as `loadtxt`. See the copied numpy announce below.
I have not looked at it, but I wonder if it is worth investigating to replace our custom ASCII fast C reader. Maybe the answer is a simple "no". The obvious benefit is greatly reducing maintenance of this difficult code. I suspect the numpy version will have better speed and memory performance as well.
Downsides:
- There are some things built in to the astropy fast reader that might not work out of box. E.g. the FastCsv reader supports missing elements at the end as masked values.
- Non-small amount of work to fix something that is not really broken. But it might be a clean and well-defined GSoC project.
- Not clear if the very careful handling of Fortran formats and other details from @dhomeier made it to the numpy parser.
Cc: @dhomeier @hamogu
### Numpy announce
https://github.com/numpy/numpy/pull/20580
is now merged. This moves `np.loadtxt` to C. Mainly making it much
faster. There are also some other improvements and changes though:
* It now supports `quotechar='"'` to support Excel dialect CSV.
* Parsing some numbers is stricter (e.g. removed support for `_`
or hex float parsing by default).
* `max_rows` now actually counts rows and not lines. A warning
is given if this makes a difference (blank lines).
* Some exception will change, parsing failures now (almost) always
give an informative `ValueError`.
* `converters=callable` is now valid to provide a single converter
for all columns.
### Additional context
Contributor guide
Research direction
Start by comparing Astropy's custom ASCII fast C reader and FastCsv behavior with numpy.loadtxt, using the linked NumPy change as context. Check support for masked values from missing trailing elements, Fortran formats, parsing strictness, exceptions, and performance or memory behavior. Done means documenting compatibility and performance findings and determining whether replacement is feasible.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- data, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100