data_set.get_parameter_data gets slower with increasing values of start/end
- Dominant language
- Python
- Stars
- 459
- Forks
- 359
- Avg merge
- 3d 6h
- Merged PRs (30d)
- 73
Description
Our dataset comprises 4 dependent parameters, which depend on 1 or 2 of the 2 independent parameters. `interdeps.dependencies` returns:
```
{ParamSpecBase('PDt_gain', 'numeric', 'Gain', ''): (ParamSpecBase('N_stepd', 'numeric', 'number of stepper stepd steps', ''),),
ParamSpecBase('PDt', 'array', 'PDt', 'V'): (ParamSpecBase('N_stepd', 'numeric', 'number of stepper stepd steps', ''),
ParamSpecBase('Time', 'array', 'Time', 's')),
ParamSpecBase('PDr', 'array', 'PDr', 'V'): (ParamSpecBase('N_stepd', 'numeric', 'number of stepper stepd steps', ''),
ParamSpecBase('Time', 'array', 'Time', 's')),
ParamSpecBase('AWG', 'array', 'AWG', 'V'): (ParamSpecBase('N_stepd', 'numeric', 'number of stepper stepd steps', ''),
ParamSpecBase('Time', 'array', 'Time', 's'))}
```
The number of results contained in the dataset is typically on the order of 1000 per dependent parameters (`ds.number_of_results` returns 4000). The dependent parameters of type array typically contain 200000 values (`len(ds.get_parameter_data('PDt', start=0, end=0)['PDt']['PDt'][0])` returns 200000).
We analyze these datasets row by row, and it is not practical to load the whole dataset at once in RAM. We thus use the `start` and `end` parameters of `ds.get_parameter_data`.
We observe a significant increase in the time taken to process `ds.get_parameter_data('PDt', start=n, end=n)['PDt']` for increasing values of `n`.
```
%%timeit
ds.get_parameter_data('PDt', start=0, end=0)['PDt']
```
returns `11.6 ms ± 137 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)`
```
%%timeit
ds.get_parameter_data('PDt', start=600, end=600)['PDt']
```
returns `929 ms ± 11.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)`
### System
Windows 10
fork of qcodes master branch, last common ancestor 48287481f16400189034e9b433a823e4aeae9cc
Contributor guide
Research direction
Reproduce the reported timings with Dataset.get_parameter_data('PDt', start=n, end=n) using the dataset sizes and dependency structure described here. Trace that entry point and compare processing for start/end at 0 and 600; done means retrieving a row remains practical as n increases without loading the whole dataset into RAM.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100