learningequality / learningequality/studio
Optimize the publishing task
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 191
- Forks
- 307
- Avg merge
- 5d 6h
- Merged PRs (30d)
- 10
Description
Issue 1:
We've found that some UN Women channels that are KA-level in size are exceeding 8GB of RAM usage and crashing the worker. We have made the RAM limit burstable to 16GB for now, but we really need to figure out a way to get publishing memory usage under control
Issue 2:
Publishing some UN Women channels is taking close to 24 hours to complete, and in some cases near the end we end up seeing Google Storage errors.
Next Steps
- Use bulk creation mechanisms to create the content nodes, in a mirror of the node duplication code
- Store the checksum of all the assessment items on generation and only regenerate if changed
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the publishing task and comparing it with the node duplication code, as suggested in the issue. Check how assessment items are generated and how content nodes are created. Done means publishing uses bulk creation, skips unchanged assessment items via stored checksums, and addresses the reported memory, runtime, and Google Storage failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python
- Domain
- backend, cloud, content, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100