Performance issue while reading the data with FLAC file format via HTTP.

Open
#426 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
python
Domain
networking, performance

Research direction

Reproduce the provided wfdb.rdsamp timing examples for mimic3wdb, mitdb, and mimic4wdb. Then inspect _url.py and the _cdata_io path in soundfile.py, focusing on repeated read calls and the buffering=-2 behavior. Done means the FLAC-over-HTTP case avoids unnecessary HTTP communication and shows improved timing without increasing server requests.

Written by the indexing model from the issue text.

Description

I tested the performance of wfdb python library for reading waveforms via HTTP.

  • mimic3wdb
dtstart = datetime.datetime.now()
wfdb.rdsamp('3000003', pn_dir='mimic3wdb/1.0/30/3000003')
print(datetime.datetime.now() - dtstart)
# results 0:00:21.143365
  • mitdb
dtstart = datetime.datetime.now()
wfdb.rdsamp('100', pn_dir='mitdb/1.0.0')
print(datetime.datetime.now() - dtstart)
# results 0:00:02.764091

It looks great. However, when I tried to read the mimic4wdb which has FLAC format, there was a significant performance decrease.

  • mimic4wdb
dtstart = datetime.datetime.now()
wfdb.rdsamp('81739927', pn_dir='mimic4wdb/0.1.0/waves/p100/p10014354/81739927')
print(datetime.datetime.now() - dtstart)
# results 0:07:26.220685

This issue was resolved when I cached files with the buffering = -2 for openurl function in _url.py.
-> results 0:00:37.388115

After digging a bit more, I figured out that this problem is caused by repeatedly calling read function frame by frame in the _cdata_io function in soundfile.py. Whenever the read function is called, session.request in _url.py is called and HTTP communication is established. This can cause significant performance problem and also make stress to the web server.

So it seems a good idea to change buffering=-2 to default until this is fixed. Reducing the number of requests is much more efficient in both improving the performance and reducing the load of the web server.

Dominant language
Jupyter Notebook
Stars
853
Forks
322
No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from MIT-LCP/wfdb-python

All issues in MIT-LCP/wfdb-python

Similar issues

More Networking issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.