CentreForDigitalHumanities / CentreForDigitalHumanities/Textcavator
Large CSV downloads
- Dominant language
- Python
- Stars
- 12
- Forks
- 3
- Avg merge
- 3d 3m
- Merged PRs (30d)
- 9
Description
**Is your feature request related to a problem? Please describe.**
Depending on user download limit, very large CSV files can be generated . Creation of these files is chunked, so this poses no problem for the server. However, downloading the files is limited to the time offered by Apache's proxy timeout (by default: 30 seconds). There is no timeout value that would fix this problem, since it is highly dependent on internet connection. For downloads into several hundred MB's, this timeout would be exceedingly large.
**Describe the solution you'd like**
Possible solutions with notes:
- Don't serve large files
- Take some limit, for example 65.000 (Excel's row limit). Everything above that is beyond the responsibility of I-Analyzer.
- Serve the files directly through Apache
- No Apache proxy timeouts, because no proxy
- No possibility to authenticate with Django, would expose (possible) protected data publicly
- Chunk large CSV files based on some row limit, e.g. 50.000. Above that number of rows, generate multiple files. Make frontend download these files separately.
- Use `X-SendFile` headers
- Interesting solution. This allows Django to authenticate and decide if a file should be served.
- Header is added to the response that Apache interprets and uses to directly serve a file, bypassing the proxy.
- The Apache plugin enabling this hasn't been updated since 2011. Backoffice does not approve.
- The [NGINX equivalent](https://www.nginx.com/resources/wiki/start/topics/examples/x-accel/) is builtin. Would need to Switch I-Analyzer to NGINX. Backoffice approves.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.