astropy / astropy/astropy

Consider using numpy loadtxt under the hood for fast ASCII reading

Open
#12,829 12 comments 0 reactions 0 assignees View on GitHub
Feature Request io.ascii
Dominant language
Python
Stars
5.3k
Forks
2.2k
Avg merge
1d 18h
Merged PRs (30d)
74

Description

### Description

Interesting progress from numpy who now have a C-based CSV parser built in as `loadtxt`. See the copied numpy announce below.

I have not looked at it, but I wonder if it is worth investigating to replace our custom ASCII fast C reader. Maybe the answer is a simple "no". The obvious benefit is greatly reducing maintenance of this difficult code. I suspect the numpy version will have better speed and memory performance as well.

Downsides:
- There are some things built in to the astropy fast reader that might not work out of box. E.g. the FastCsv reader supports missing elements at the end as masked values.
- Non-small amount of work to fix something that is not really broken. But it might be a clean and well-defined GSoC project.
- Not clear if the very careful handling of Fortran formats and other details from @dhomeier made it to the numpy parser.

Cc: @dhomeier @hamogu

### Numpy announce

https://github.com/numpy/numpy/pull/20580

is now merged. This moves `np.loadtxt` to C. Mainly making it much
faster. There are also some other improvements and changes though:

* It now supports `quotechar='"'` to support Excel dialect CSV.
* Parsing some numbers is stricter (e.g. removed support for `_`
or hex float parsing by default).
* `max_rows` now actually counts rows and not lines. A warning
is given if this makes a difference (blank lines).
* Some exception will change, parsing failures now (almost) always
give an informative `ValueError`.
* `converters=callable` is now valid to provide a single converter
for all columns.
### Additional context

Contributor guide

Open the contributing guide

Research direction

Start by comparing Astropy's custom ASCII fast C reader and FastCsv behavior with numpy.loadtxt, using the linked NumPy change as context. Check support for masked values from missing trailing elements, Fortran formats, parsing strictness, exceptions, and performance or memory behavior. Done means documenting compatibility and performance findings and determining whether replacement is feasible.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.