Automattic / Automattic/liveblog
Remove WPCOM_Liveblog_Entry_Query::get_all_entries_asc()
- Dominant language
- PHP
- Stars
- 313
- Forks
- 129
- Avg merge
- 18h 33m
- Merged PRs (30d)
- 1
Description
This function is used in a few spots to (primarily) get all entries and then slice out the desired ones in `WPCOM_Liveblog::get_entries_by_time()`, and then to calculate pagination in that function and `WPCOM_Liveblog::get_entries_paged()`.
Normally this works fine, querying the list from memcached then slicing out relevant ones for a given timestamp interval is a fast operation. However, this falls over flat on large datasets. The default memcache max object size is 1MB, and on Liveblog posts with many entries, that can be exceeded, which ends up reading lots of data at once from the DB, as well as writing lots of data to memcache (which is immediately dropped).
`get_comments()` now has Date Query support, and it has some caching internally (although admittedly it's not great b/c the cache key uses the latest comment from the whole site).
To solve the large LB problem, we need to move away from this monolithic cache structure and query the DB, then add caching that only stores small chunks of data.
Contributor guide
Research direction
Start by reading WPCOM_Liveblog_Entry_Query::get_all_entries_asc(), WPCOM_Liveblog::get_entries_by_time(), WPCOM_Liveblog::get_entries_paged(), and get_comments(). Trace how entry lists are read from the database and memcached, then define a chunked query and caching approach that avoids the 1MB object limit while preserving timestamp filtering and pagination.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- php
- Domain
- backend, databases
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100