Add limit on the size in KB of data returned from a single query
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 11.5k
- Forks
- 904
- Avg merge
- 4d 17h
- Merged PRs (30d)
- 18
Description
Datasette limits the number of rows returned to 1,000 and limits the time spent executing a SQL query to 1000ms - and both of these limits can be customized.
It does not have a limit on the size of the response returned. It's possible to compose maliciously large SQL responses in a small number of rows using mechanisms like the group_concat() aggregate function. It would be good to avoid malicious SQL creating 100MB+ responses and potentially crashing the server.
I think the easiest place to implement that is here:
Currently we use cursor.fetchmany() to fetch up to 1,001 rows at once. Instead, we could switch to iterating through cursor.fetchone() (or just using for row in cursor) and keeping a running tally of the size of the response as we go - maybe just using rough_response_size += len(str(row)). If that goes above a certain threshold we can terminate the response with an error, like we do with timelimits.
The bigger challenge here is understanding how well this approach works and what impact it will have on overall Datasette performance. I think I need #33 for this.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in datasette/app.py at the linked lines, where cursor.fetchmany() currently retrieves up to 1,001 rows, and review the related work in issue #33. Determine how to track response size while iterating rows and how the limit should terminate the response, then assess the performance impact. Done means oversized query responses are rejected without affecting normal results or the existing row and time limits.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sql, sqlite
- Domain
- api, backend, security
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100