RadioAstronomySoftwareGroup / RadioAstronomySoftwareGroup/pyuvdata

performance monitoring and benchmarking

Open
#729 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

UVData
Dominant language
Python
Stars
102
Forks
35
Avg merge
1d 20h
Merged PRs (30d)
14

Description

Occasionally I hear that pyuvdata is slow, though without further investigation this complaint is impossible to decouple from the size of the data being read.* However since we do not currently track execution time of our various tasks, it is possible for a change to be introduced which increases execution time. This is difficult to monitor at scale because large files and extended execution times are not easily supported within the current testing infrastructure. Here are a few possible things we could do:

  1. Monitor the execution time of all tests. This is a crude metric as we know that execution time can be affected by exogenous factors related to the underlying cloud infrastructure or install times.
  2. Monitor the execution time of specific existing tests. This would be a more precise datum than all tests which would expose things that grossly affect read time, but since the test files are small would not expose issues that scale badly with times, freqs, etc.
  3. Add tests which focus on timing but use the existing test files. This could include things like reading the file many times and averaging the read time, generating a large number of files and concat-read them, etc.
  4. Setups that require more resources. Its not clear what the break point is.

*The following is a digest of a discussion on the 3 Dec 2019 pyuvdata telecon.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no specific files, tests, or entry points. First review the existing test files and testing infrastructure, then determine which benchmarking approach is accepted; the work is done when a defined, repeatable way to detect execution-time regressions is agreed and implemented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.