developmentseed / developmentseed/zarrista

Add free-threaded tests with pytest-run-parallel

Open
#70 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
45
Forks
2
Avg merge
2d 4h
Merged PRs (30d)
7

Description

Follow up from https://github.com/developmentseed/zarrista/pull/69

----

Claude:

## Follow-up: free-threaded (3.14t) CI testing

PR #69 adds the **wheel builds** for free-threaded CPython 3.14 (`cp314-cp314t`)
and scaffolds the abi3t (3.15+) wheel. It intentionally does **not** add CI that
*tests* zarrista on a free-threaded interpreter — this issue tracks that work,
which turned out to need a test-suite restructure rather than a CI one-liner.

### Why it was deferred

Spiking the 3.14t test job surfaced three blockers:

1. **`numcodecs` re-enables the GIL.** `numcodecs.blosc` hasn't declared that it
can run without the GIL, so importing it on 3.14t silently turns the GIL back
on:
> `RuntimeWarning: The global interpreter lock (GIL) has been enabled to load
> module 'numcodecs.blosc' ...`

Since nearly all tests pull in zarr-python (→ numcodecs), running the current
suite on 3.14t gives a **false** free-threaded signal — the GIL is on.

2. **Missing free-threaded wheels → slow / degraded.** `icechunk` and
`numcodecs` compile from source on 3.14t (no `cp314t` wheels yet), and
`google-crc32c` (via icechunk) falls back to its pure-Python implementation:
> `As the c extension couldn't be imported, google-crc32c is using a pure
> python implementation that is significantly slower.`

3. **Tests use zarr-python as the fixture *writer*.** 8 of 10 test files call
`zarr.create_array` / `zarr.open_group` to produce fixtures, then read them
back with zarrista. These are really *interop/integration* tests. We have
almost no coverage that exercises zarrista's **own** read+write round-trip
(`ArrayBuilder.create` / `store_chunk` / `store_metadata` → `Array` read).

### Proposed plan

- [ ] Split the suite into **own** tests (zarrista-only read/write) vs
**integration** tests (zarr-python / icechunk / arro3 interop), via markers
or a `tests/integration/` directory.
- [ ] Add zarrista-only round-trip tests (create + store chunks + read back,
compared against numpy) so there's real own-code coverage.
- [ ] Split dependency groups so the free-threaded job installs only GIL-safe
deps (avoid zarr/numcodecs/icechunk).
- [ ] Add a `3.14t` CI job that runs **only** the own tests on the free-threaded
interpreter.
- [ ] Revisit `pytest-run-parallel` for genuine thread-safety stress — note it
runs the *same* test in N threads sharing one fixture, so any test that
mutates shared store state flakes; concurrency tests must use per-thread
isolated stores.
- [ ] Re-enable integration tests on 3.14t once `numcodecs`/`icechunk` ship
free-threaded wheels and declare GIL-safety.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the tests that use zarr.create_array and zarr.open_group, then inspect the zarrista-only entry points ArrayBuilder.create, store_chunk, store_metadata, and Array reads. Separate own tests from integration tests, add isolated round-trip coverage compared with numpy, and verify a 3.14t CI job runs only GIL-safe tests without shared-store concurrency flakes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.