AOSSIE-Org / AOSSIE-Org/PictoPy

Feat:Optimize face embedding storage in SQLite using raw float32 BLOB with backward compatibility

Aberta
#1,497 7 comentários 0 reações 0 responsáveis Ver no GitHub
enhancement
Linguagem predominante
Python
Estrelas
283
Forks
679
Merge médio
7d 2h
PRs com merge (30d)
3

Descrição

### Describe the feature

### Issue
Currently, `faces.py` stores face embeddings as JSON strings (`TEXT`), whereas `image_embeddings.py` and `video_frames.py` already store embeddings as raw `float32` BLOBs .

Storing embeddings as JSON strings introduces unnecessary overhead in storage size, CPU deserialization cycles, and numeric formatting precision.

---

### Impact
1. **Storage Reduction**: 512-dim `float32` vectors take 2 KB as binary BLOB vs 8–10 KB as JSON text which will be crucial in case of large data.
2. **Fast Deserialization**: `np.frombuffer()` eliminates the CPU overhead of `json.loads()` when querying thousands of faces for clustering and search.

---

### Backward Compatibility
- **Startup Data Migration**: In `db_create_faces_table()`, migrate existing `typeof(embeddings) = 'text'` rows to BLOB in-place.
- **Dual-mode Deserializer**: Add a fallback helper supporting both `bytes` (`np.frombuffer`) and legacy `str` (`json.loads`).

#### Proposed Changes
- faces.py: Update insertion and reader queries to write/read BLOB, plus add one-time migration logic.
- test_faces_db.py: Add tests for BLOB storage and legacy JSON backward compatibility.

### Add ScreenShots

@Maintainer I would like to solve this.

### Record

- [x] I agree to follow this project's Code of Conduct
- [x] I want to work on this issue

Guia de contribuição

Abrir o guia de contribuição

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.