astropy / astropy/asdf-astropy
setting compression on individual columns of astropy table
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 19
- Forks
- 22
- Avg merge
- 6h 1m
- Merged PRs (30d)
- 1
Description
Is there a syntax to set compression on individual columns of an astropy table? In the following example, using all_array_compression compresses the columns, but using AsdfFile.set_array_compression() does not.
import numpy as np
from astropy.table import Table
import asdf
t = Table(data=dict(col=np.ones(1)))
with asdf.AsdfFile(tree=dict(table=t)) as af:
af.write_to('test.asdf')
with asdf.AsdfFile(tree=dict(table=t)) as af:
af.write_to('test_compressed.asdf',
all_array_compression='zlib',
)
with asdf.AsdfFile(tree=dict(table=t)) as af:
af.set_array_compression(af['table']['col'], 'zlib') # this has no effect
af.write_to('test_compressed_col.asdf')
Comparing test.asdf with test_compressed_col.asdf, we see that they have identical checksums (and there's no zlib tag at the beginning of the binary block). So the set_array_compression had no effect.
I've tried variants of this like
set_array_compression(af['table']['col'].base, 'zlib')
set_array_compression(af['table']['col'].data, 'zlib')
but I couldn't get it to work.
I did dig around in the source code a bit, and it looked to me like it's trying to compare the ultimate ndarray base to check if two arrays are the same, but maybe a copy is being made somewhere that's thwarting this detection.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the reproduced calls to AsdfFile.set_array_compression() and the table column expressions af['table']['col'], .base, and .data. Read the related source logic that compares ndarray bases, then reproduce the checksum and binary-block comparison from the example. Done means per-column compression works or the limitation and correct syntax are documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100