Memory leak when downloading beamline data with Databroker 1
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 38
- Forks
- 29
- PR merge metrics
- No merged PRs in 30d
Description
Memory leak was observed at HXN beamline when make_hdf was used from IPython to download large batch of run data (~70 runs). After ~40-50 files the workstation runs out of memory. Data Broker 0.13.3 provides API to clear cache, which is explicitely used by PyXRF. In Data Broker 1.0 cache is expected to be cleared automatically.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the make_hdf entry point used from IPython and compare its handling of large batches with the Data Broker 1 cache behavior described here. Reproduce the download with roughly 70 runs and determine whether memory remains bounded; completion means the workstation no longer exhausts memory during the batch.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100