NCAS-CMS / NCAS-CMS/cf-python

Improve performance related to `cf.Field.autocylcic`

Open
#903 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

dataset read enhancement performance
Dominant language
Python
Stars
150
Forks
23
Avg merge
1d 11h
Merged PRs (30d)
2

Description

Now that we have #851 (Improve detection of cyclic axes), I realise that we no longer need to run cf.Field.autocyclic during cf.Field.set_construct, because autocyclic will get end up being run anyway, if required, whenever cf.Field.cyclic() is called.

This doesn't change any functionality, but will speed up the reading of remote datasets. Currently, even if you set cache=False to cf.read, set_construct ends up accessing the cyclic axis bounds values from the dataset, which can be slow.

Pushing the overhead of autocyclic to when cyclicity is actually needed makes sense because i) you may well not need that information in your workflow and ii) the overhead of the operation will likely make the autocyclic overhead not seem so bad.

Note that once autocyclic has been run once, it caches the values it need from disk, so is faster the next time.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing cf.Field.set_construct and its interaction with cf.Field.autocyclic and cf.Field.cyclic. Remove the eager cyclicity work from set_construct while preserving the existing lazy behavior, then verify that cyclicity is still detected when cyclic() is called and that remote dataset reads avoid unnecessary bounds access.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.