slow access to s3 resources (initial 3-4s lag)

Open

#504 opened on Apr 8, 2017 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty: 4/5
Estimated time: 3-5 days
Newbie friendliness: 25/100

Issue type: Bug
Clarity: Needs clarification
Activity status: Stale
Tech stack: c
Domain: networking

Research direction

Reproduce the delay with the two samtools view S3 commands and compare them with the direct curl pipeline, including after the BAI is cached. Inspect the initial S3 request path and any timeout or libcurl behavior involved. Done means identifying the source of the 1–2 second lag and confirming whether it can be removed without affecting local BAM access.

Written by the indexing model from the issue text.

Description

Hi,

We've noticed that samtools/htslib seems to have a small time penalty when making an initial connection to any s3 resource (local s3 or amazonaws s3).

e.g.
1-2 second delay
samtools view "https://s3-us-west-1.amazonaws.com/ilmn.htslib-test/test.bam
samtools view "https://s3-us-west-1.amazonaws.com/ilmn.htslib-test/test.bam" chr3:49000000-50000000

However a direct curl seems to begin streaming in < 1 second
curl -XGET "https://s3-us-west-1.amazonaws.com/ilmn.htslib-test/test.bam" | samtools view

This time delay does not appear to be due to the initial download of the BAI, as it still happens after the index is cached, nor does it appear to be the result of the s3 auth (above is a public object)

Are there any sort of wait/timeout(s) happening during the initial call? Or something with libcurl?

Access times for local BAMs are close to instant (as expected).
We (@nhjohnson) expect a small network related time delay, but is it possible something else is causing the lag?

Thanks !

Dominant language: C
Stars: 950
Forks: 475
Avg merge: 3d 13h
Merged PRs (30d): 11

Contributor guide

Open the contributing guide

First steps

Read the whole issue, then the project's contributing guide.
Comment on the issue to say you are picking it up — it saves two people doing the same work.
Fork the repository and make your change on a branch.
Open a pull request that references the issue number.