More advanced connection pooling
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 11.5k
- Forks
- 904
- Avg merge
- 4d 17h
- Merged PRs (30d)
- 18
Description
We need a much smarter way of handling database connections.
Today, connections are simple: Datasette runs a number of threads (defaults to 3) and each thread gets a threadlocal read-only (or immutable) connection to each attached database - opened on demand.
For Datasette Library (#417) I want to support potentially hundreds of attached databases. Datasette Edit (#567) is going to introduce a need for writable connections too.
I'd also like to be able to run joins across multiple databases (#283) which further complicates things.
Supporting thousands of open SQLite connections at once feels like it won't provide good enough performance (though I should benchmark that to be sure). Some kind of connection pooling is likely to be necessary.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files or tests are named. Start by tracing Datasette's current thread-local, read-only or immutable connection handling, then review related issues #417, #567, and #283. Benchmark the cost of many open SQLite connections before proposing pooling that accounts for hundreds of attached databases, writable connections, and joins across databases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sqlite
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100