CentreForDigitalHumanities / CentreForDigitalHumanities/Textcavator

Large CSV downloads

Open
#1,474 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
12
Forks
3
Avg merge
3d 3m
Merged PRs (30d)
9

Description

**Is your feature request related to a problem? Please describe.**
Depending on user download limit, very large CSV files can be generated . Creation of these files is chunked, so this poses no problem for the server. However, downloading the files is limited to the time offered by Apache's proxy timeout (by default: 30 seconds). There is no timeout value that would fix this problem, since it is highly dependent on internet connection. For downloads into several hundred MB's, this timeout would be exceedingly large.

**Describe the solution you'd like**
Possible solutions with notes:
- Don't serve large files
- Take some limit, for example 65.000 (Excel's row limit). Everything above that is beyond the responsibility of I-Analyzer.
- Serve the files directly through Apache
- No Apache proxy timeouts, because no proxy
- No possibility to authenticate with Django, would expose (possible) protected data publicly
- Chunk large CSV files based on some row limit, e.g. 50.000. Above that number of rows, generate multiple files. Make frontend download these files separately.
- Use `X-SendFile` headers
- Interesting solution. This allows Django to authenticate and decide if a file should be served.
- Header is added to the response that Apache interprets and uses to directly serve a file, bypassing the proxy.
- The Apache plugin enabling this hasn't been updated since 2011. Backoffice does not approve.
- The [NGINX equivalent](https://www.nginx.com/resources/wiki/start/topics/examples/x-accel/) is builtin. Would need to Switch I-Analyzer to NGINX. Backoffice approves.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.