pydata / pydata/xarray

Repo layout proposal

Open
#10,089 18 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic-internals
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

What is your issue?

As part of the efforts described in https://github.com/pydata/xarray/pull/10039, I added https://github.com/pydata/xarray/pull/10088, and noticed the repo layout has arguably not kept up with the code growth over the past decade. This isn't the most pressing issue, but it does make the returns to refactors lower, since we're moving lines from 11K LOC files to 1K LOC files, rather than anything smaller.

(Even if you think LLMs aren't that useful / aren't going to get better / etc; these changes would still make the repo easier for people to navigate...)

In particular, 2/3 of our code is in xarray/core — 66873 LOC vs 97118 LOC in xarray

I can imagine splitting this up into a few categories:

  • compat — dask_array_*, npcompat, pdcompat, array_api_compat
  • compute / computation — computation, arithmetic, nanops, weighted, the curvefit that's currently in dataset, rolling, rolling_exp, maybe missing
  • reshape / align / merge (need a better name) — merge, alignment, concat

I'd propose having each of those be paths within xarray/. Then there's more freedom to make new files within those paths relative to the current state, where a new file means adding onto a very long list of files in xarray/core.

I'm not confident on how much disruption that can cause to existing PRs. I think if we land them as commits which mostly just move the files, then git will mostly handle merges well. We can start slowly and see how it goes...

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the current xarray/core layout and the changes in PRs #10039 and #10088. Compare the proposed compat, computation, and reshape/align/merge groupings, then resolve naming and migration scope with maintainers. Done means an agreed staged layout plan that accounts for existing pull requests.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, pandas, python
Domain
developer-experience, tooling
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.