microsoft / microsoft/Qcodes

data_set.get_parameter_data gets slower with increasing values of start/end

Open
#1,915 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
459
Forks
359
Avg merge
3d 6h
Merged PRs (30d)
73

Description

Our dataset comprises 4 dependent parameters, which depend on 1 or 2 of the 2 independent parameters. `interdeps.dependencies` returns:
```
{ParamSpecBase('PDt_gain', 'numeric', 'Gain', ''): (ParamSpecBase('N_stepd', 'numeric', 'number of stepper stepd steps', ''),),
ParamSpecBase('PDt', 'array', 'PDt', 'V'): (ParamSpecBase('N_stepd', 'numeric', 'number of stepper stepd steps', ''),
ParamSpecBase('Time', 'array', 'Time', 's')),
ParamSpecBase('PDr', 'array', 'PDr', 'V'): (ParamSpecBase('N_stepd', 'numeric', 'number of stepper stepd steps', ''),
ParamSpecBase('Time', 'array', 'Time', 's')),
ParamSpecBase('AWG', 'array', 'AWG', 'V'): (ParamSpecBase('N_stepd', 'numeric', 'number of stepper stepd steps', ''),
ParamSpecBase('Time', 'array', 'Time', 's'))}
```
The number of results contained in the dataset is typically on the order of 1000 per dependent parameters (`ds.number_of_results` returns 4000). The dependent parameters of type array typically contain 200000 values (`len(ds.get_parameter_data('PDt', start=0, end=0)['PDt']['PDt'][0])` returns 200000).
We analyze these datasets row by row, and it is not practical to load the whole dataset at once in RAM. We thus use the `start` and `end` parameters of `ds.get_parameter_data`.

We observe a significant increase in the time taken to process `ds.get_parameter_data('PDt', start=n, end=n)['PDt']` for increasing values of `n`.
```
%%timeit
ds.get_parameter_data('PDt', start=0, end=0)['PDt']
```
returns `11.6 ms ± 137 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)`
```
%%timeit
ds.get_parameter_data('PDt', start=600, end=600)['PDt']
```
returns `929 ms ± 11.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)`

### System
Windows 10

fork of qcodes master branch, last common ancestor 48287481f16400189034e9b433a823e4aeae9cc

Contributor guide

Open the contributing guide

Research direction

Reproduce the reported timings with Dataset.get_parameter_data('PDt', start=n, end=n) using the dataset sizes and dependency structure described here. Trace that entry point and compare processing for start/end at 0 and 600; done means retrieving a row remains practical as n increases without loading the whole dataset into RAM.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.