dmarx / dmarx/twitterMonitor

migrate to a more robust database solution

Open
#36 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Handling database locks from conflicting components is slowing down the tweet scraping mechanism, which is damaging the ability of the app to perform trend detection. Need to swap out sqlite3 for a database engine that handles locking better. To consider:

* postgres
* mysql

Additionally: the title daemone is writing titles directly into the entities table at the moment. this means that even if I resolve database level locks, the title_daemon and scraper will still encounter locking conflicts when trying to write into the entities table. Should factor out titles into a separate table to eliminate table level locking conflicts.

Should consider swapping out raw SQL for SQLAlchemy to make swapping out backends easier.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.