NCAS-CMS / NCAS-CMS/PyActiveStorage

Multiple operations instead of means

Open
#33 8 comments 0 reactions 1 assignee View on GitHub

@bnlawrence is already working on this.

Since Oct 27, 2022.

excalibur
Dominant language
Python
Stars
6
Forks
2
PR merge metrics
No merged PRs in 30d

Description

In working through the implications of implementing means in chunks, it is notable that once missing data is in play, we need to return two numbers from the reduce_chunk method: the sum, and the count, because means over chunks will be needed to be weighted by the actual number of values being meaned.

There are a number of ways we could implement this:

  1. Always return (X, N), where X is the expected operation, and N the number of values contributing
  2. Only return (X, N) when required (e.g. for means) otherwise return (X,None) or (X,)
  3. Return X, except when it needs to be (X,N)
  4. Something else.

The something else option could be slightly more interesting: do we think it's a smart idea to say we could chain a series of methods and expect a series of results, in a lightweight sort of caching?

Obvious use cases would be:

  • mean = sum, count
  • range = min, max
  • sqmean = sum(squares), sum, count

This could be facilitated by handing not just "a method" but a list of 1.. many methods, and expect back a list of 1..many results.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.