emscripten-core / emscripten-core/emscripten
Warn in docs that Chrome doesn't cache large .data and .wasm files
- Dominant language
- C++
- Stars
- 27.6k
- Forks
- 3.6k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 105
Description
I have a primitive http server that sets ETag for all files, including html/js/data/wasm. However, Chrome seems to not want to cache any file larger than a few dozen megabytes. This leads to reloading large data files, and this is slow-ish even on localhost.
My question on SO about this: https://stackoverflow.com/questions/63891436/chrome-refuses-to-cache-binary-data-files, code:
```python
import os
import hashlib
import http.server
root = '.'
mime = {
'.manifest': 'text/cache-manifest',
'.html': 'text/html',
'.png': 'image/png',
'.jpg': 'image/jpg',
'.svg': 'image/svg+xml',
'.css': 'text/css',
'.js': 'application/x-javascript',
'.wasm': 'application/wasm',
'.data': 'application/octet-stream',
}
mime_fallback = 'application/octet-stream'
def md5(file_path):
hash = hashlib.md5()
with open(file_path, 'rb') as f:
hash.update(f.read())
return hash.hexdigest()
cache = {os.path.join(root, f) : md5(f) for f in os.listdir() if any(map(f.endswith, mime)) and os.path.isfile(f)}
class EtagHandler(http.server.BaseHTTPRequestHandler):
def do_GET(self, body = True):
self.protocol_version = 'HTTP/1.1'
self.path = os.path.join(root, self.path.lstrip('/') + ('index.html' if self.path == '/' else ''))
if not os.path.exists(self.path) or not os.path.isfile(self.path):
self.send_response(404)
elif self.path not in cache or cache[self.path] != self.headers.get('If-None-Match'):
content_type = ([content_type for ext, content_type in sorted(mime.items(), reverse = True) if self.path.endswith(ext)] + [mime_fallback])[0]
with open(self.path, 'rb') as f:
content = f.read()
self.send_response(200)
self.send_header('Content-Length', len(content))
self.send_header('Content-Type', content_type)
self.send_header('ETag', cache[self.path])
self.end_headers()
self.wfile.write(content)
else:
self.send_response(304)
self.send_header('ETag', cache[self.path])
self.end_headers()
if __name__ == '__main__':
PORT = 8080
print("serving at port", PORT)
httpd = http.server.HTTPServer(("", PORT), EtagHandler)
httpd.serve_forever()
```
One way forward may be to force `file_packager` to generate chunks of 10Mb and have them load in parallel (same for large wasm files).
Maybe related: https://stackoverflow.com/questions/60646737/how-to-cache-large-fie-in-chrome
Related: https://github.com/emscripten-core/emscripten/issues/4711. Maybe worth providing an option for IndexedDB caching for wasm in the meanwhile...
The problem with .wasm is also very acute, since it's happening at every module reload. Would you have an advice on dumping/reloading heap / other state? So that I could try to reset the module (at least softly)
Contributor guide
Assessment
This issue has not been assessed yet.