internetarchive / internetarchive/dweb-mirror

Search is slow to respond

Open
#257 1 comment 0 reactions 0 assignees View on GitHub
2SC/ON Performance
Dominant language
JavaScript
Stars
330
Forks
34
PR merge metrics
No merged PRs in 30d

Description

STR
Try a search for something new
Watch the DM console
Notice how browser pauses while DM is working to do something - not sure what,

Output of console is e.g.
```
dweb-transports:httptools Fetch of https://dweb.me/arc/archive.org/advancedsearch?output=json&q=byron%20bay&rows=30&page=1&sort[]=-downloads&and[]=&save=yes&fl=identifier%2Ctitle%2Ccollection%2Cmediatype%2Cdownloads%2Ccreator%2Cnum_reviews%2Cpublicdate%2Citem_count%2Cloans__status__status completed +4s
dweb-mirror:MirrorFS Already cached existing file with matching sha1 /Volumes/Transcend/archiveorg/THEBLUESROOM64911JULY2018MIDGEMARSDENTRIBUTE/__ia_thumb.jpg +34s
dweb-mirror:MirrorFS Already cached existing file with matching sha1 /Volumes/Transcend/archiveorg/THEBLUESROOM6484JULY2018HOUROFBLUESPOWERWITHDARCYPERRY/__ia_thumb.jpg +5ms
dweb-mirror:MirrorFS Already cached existing file with matching sha1 /Volumes/Transcend/archiveorg/THEBLUESROOM65018JULY2018BELLYFULLABLUESWITHFATTMAX/__ia_thumb.jpg +693ms
dweb-mirror:MirrorFS Already cached existing file with matching sha1 /Volumes/Transcend/archiveorg/eits2012-07-27.aud/__ia_thumb.jpg +30ms
dweb-mirror:MirrorFS Already cached existing file with matching sha1 /Volumes/Transcend/archiveorg/jbt2003-07-20.flac/__ia_thumb.jpg +385ms
dweb-mirror:MirrorFS Already cached existing file with matching sha1 /Volumes/Transcend/archiveorg/JimmyNooneJrWithDaviesRhythmic5605SweetLorraine/__ia_thumb.jpg +740ms
dweb-mirror:MirrorFS Already cached existing file with matching sha1 /Volumes/Transcend/archiveorg/PoorMansWhiskey2016-03-04/__ia_thumb.jpg +1s
dweb-transports:httptools p_httpfetch: https://dweb.me/info '' +9s
dweb-transports:httptools Fetch of https://dweb.me/info completed +3s
GET /arc/archive.org/advancedsearch?output=json&q=byron%20bay&rows=30&page=1&sort[]=-downloads&and[]=&save=yes&fl=identifier%2Ctitle%2Ccollection%2Cmediatype%2Cdownloads%2Ccreator%2Cnum_reviews%2Cpublicdate%2Citem_count%2Cloans__status__status - 200 25159 29693.270 ms
```
Which looks like its checking/fetching thumbnails before returning

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce a search for “byron bay” and inspect the DM console and the /arc/archive.org/advancedsearch request shown in the report. Trace why thumbnail or cache work occurs before the response completes; done means the search no longer pauses while that unrelated work is performed, with the reported request timing improved.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
performance, search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.