MIT-LCP / MIT-LCP/wfdb-python

Performance issue while reading the data with FLAC file format via HTTP.

Open
#426 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jupyter Notebook
Stars
853
Forks
322
PR merge metrics
No merged PRs in 30d

Description

I tested the performance of wfdb python library for reading waveforms via HTTP.

  • mimic3wdb
dtstart = datetime.datetime.now()
wfdb.rdsamp('3000003', pn_dir='mimic3wdb/1.0/30/3000003')
print(datetime.datetime.now() - dtstart)
# results 0:00:21.143365
  • mitdb
dtstart = datetime.datetime.now()
wfdb.rdsamp('100', pn_dir='mitdb/1.0.0')
print(datetime.datetime.now() - dtstart)
# results 0:00:02.764091

It looks great. However, when I tried to read the mimic4wdb which has FLAC format, there was a significant performance decrease.

  • mimic4wdb
dtstart = datetime.datetime.now()
wfdb.rdsamp('81739927', pn_dir='mimic4wdb/0.1.0/waves/p100/p10014354/81739927')
print(datetime.datetime.now() - dtstart)
# results 0:07:26.220685

This issue was resolved when I cached files with the buffering = -2 for openurl function in _url.py.
-> results 0:00:37.388115

After digging a bit more, I figured out that this problem is caused by repeatedly calling read function frame by frame in the _cdata_io function in soundfile.py. Whenever the read function is called, session.request in _url.py is called and HTTP communication is established. This can cause significant performance problem and also make stress to the web server.

So it seems a good idea to change buffering=-2 to default until this is fixed. Reducing the number of requests is much more efficient in both improving the performance and reducing the load of the web server.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the provided wfdb.rdsamp timing examples for mimic3wdb, mitdb, and mimic4wdb. Then inspect _url.py and the _cdata_io path in soundfile.py, focusing on repeated read calls and the buffering=-2 behavior. Done means the FLAC-over-HTTP case avoids unnecessary HTTP communication and shows improved timing without increasing server requests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.