emscripten-core / emscripten-core/emscripten

Warn in docs that Chrome doesn't cache large .data and .wasm files

Open
#12,212 12 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
27.6k
Forks
3.6k
Avg merge
1d 1h
Merged PRs (30d)
105

Description

I have a primitive http server that sets ETag for all files, including html/js/data/wasm. However, Chrome seems to not want to cache any file larger than a few dozen megabytes. This leads to reloading large data files, and this is slow-ish even on localhost.

My question on SO about this: https://stackoverflow.com/questions/63891436/chrome-refuses-to-cache-binary-data-files, code:
```python
import os
import hashlib
import http.server

root = '.'

mime = {
'.manifest': 'text/cache-manifest',
'.html': 'text/html',
'.png': 'image/png',
'.jpg': 'image/jpg',
'.svg': 'image/svg+xml',
'.css': 'text/css',
'.js': 'application/x-javascript',
'.wasm': 'application/wasm',
'.data': 'application/octet-stream',
}
mime_fallback = 'application/octet-stream'

def md5(file_path):
hash = hashlib.md5()
with open(file_path, 'rb') as f:
hash.update(f.read())
return hash.hexdigest()

cache = {os.path.join(root, f) : md5(f) for f in os.listdir() if any(map(f.endswith, mime)) and os.path.isfile(f)}

class EtagHandler(http.server.BaseHTTPRequestHandler):
def do_GET(self, body = True):
self.protocol_version = 'HTTP/1.1'
self.path = os.path.join(root, self.path.lstrip('/') + ('index.html' if self.path == '/' else ''))

if not os.path.exists(self.path) or not os.path.isfile(self.path):
self.send_response(404)

elif self.path not in cache or cache[self.path] != self.headers.get('If-None-Match'):
content_type = ([content_type for ext, content_type in sorted(mime.items(), reverse = True) if self.path.endswith(ext)] + [mime_fallback])[0]
with open(self.path, 'rb') as f:
content = f.read()

self.send_response(200)
self.send_header('Content-Length', len(content))
self.send_header('Content-Type', content_type)
self.send_header('ETag', cache[self.path])
self.end_headers()
self.wfile.write(content)

else:
self.send_response(304)
self.send_header('ETag', cache[self.path])
self.end_headers()

if __name__ == '__main__':
PORT = 8080
print("serving at port", PORT)
httpd = http.server.HTTPServer(("", PORT), EtagHandler)
httpd.serve_forever()
```

One way forward may be to force `file_packager` to generate chunks of 10Mb and have them load in parallel (same for large wasm files).

Maybe related: https://stackoverflow.com/questions/60646737/how-to-cache-large-fie-in-chrome

Related: https://github.com/emscripten-core/emscripten/issues/4711. Maybe worth providing an option for IndexedDB caching for wasm in the meanwhile...

The problem with .wasm is also very acute, since it's happening at every module reload. Would you have an advice on dumping/reloading heap / other state? So that I could try to reset the module (at least softly)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.