AOSSIE-Org / AOSSIE-Org/PictoPy

Feat: Implement Incremental Indexing Engine for Scalable Library Management

Offen
#1,186 5 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
283
Forks
679
Ø Merge
7 T. 2 Std.
Gemergte PRs (30 T.)
3

Beschreibung

## 🚀 Problem

Currently, library rescans may reprocess many or all images, even if they haven't changed.
For large libraries (10k+ images), this leads to:

- Slow refresh times
- Unnecessary ML reprocessing
- High CPU usage
- Poor scalability

There is no reliable mechanism to detect:
- Newly added files
- Modified files
- Deleted files
- Unchanged files

---

## 💡 Proposed Solution

Implement a true **Incremental Indexing Engine** that:

- Stores file metadata (e.g., path, size, last modified time, optional hash)
- Processes only new or modified files
- Skips unchanged files
- Removes deleted files from the database
- Cleans associated embeddings and thumbnails

---

## 🎯 Expected Benefits

- Significant performance improvement for large libraries
- Reduced redundant ML inference
- Better database consistency
- Production-grade scalability

---

## 🛠️ Implementation Overview

1. Extend image table schema to store file metadata
2. Add comparison logic during rescans
3. Update indexing pipeline to process only changed files
4. Handle cleanup for deleted files

---

## 🧪 Acceptance Criteria

- Only new/modified files are processed during rescan
- Deleted files are removed from DB
- No redundant ML processing occurs
- Works reliably on large datasets (10k+ images)

---

## 🙋 I would like to work on this issue.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.