chaoss / chaoss/CollectOSS

Full table scans on every batch in messages and events collection

Open
#146 2 comments 0 reactions 0 assignees View on GitHub
challenging first issue
Dominant language
Python
Stars
13
Forks
17
Avg merge
6h 59m
Merged PRs (30d)
1

Description

> [!NOTE]
> Migrated from [augurlabs/augur#3440](https://github.com/augurlabs/augur/issues/3440)
> Originally opened by `@shlokgilda` on 2025-12-04

---

**Description:**
Both `messages.py` and `events.py` rebuild issue/PR URL mappings on every batch instead of once at the start. This causes redundant full table scans that scale with data volume.

**messages.py:156-164** - queries all issues and PRs per 20-message batch:
```python
issues = augur_db.session.query(Issue).filter(Issue.repo_id == repo_id).all()
prs = augur_db.session.query(PullRequest).filter(PullRequest.repo_id == repo_id).all()
```
**events.py:237-253** - rebuilds mappings per 500-event batch via `_get_map_from_*` methods.

Impact:
- 1000 messages -> 50 full scans of issues AND PRs tables
- 10000 events -> 40 full scans total

Expected behavior:
Build mappings once before the batch loop, pass as parameters. See augurlabs/augur#3439 for the expected pattern.

Suggested fix:
1. Move mapping queries outside batch loops
2. Build `issue_url_to_id_map` and `pr_url_to_id_map` once
3. Pass mappings to processing functions
4. Possibly also increase messages batch size from 20 to 1000 (unless this was an intentional design choice that wasn't documented in the code)

Files:
- augur/tasks/github/messages.py
- augur/tasks/github/events.py

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.