AOSSIE-Org / AOSSIE-Org/PictoPy
Feat:Optimize face embedding storage in SQLite using raw float32 BLOB with backward compatibility
- Linguagem predominante
- Python
- Estrelas
- 283
- Forks
- 679
- Merge médio
- 7d 2h
- PRs com merge (30d)
- 3
Descrição
### Describe the feature
### Issue
Currently, `faces.py` stores face embeddings as JSON strings (`TEXT`), whereas `image_embeddings.py` and `video_frames.py` already store embeddings as raw `float32` BLOBs .
Storing embeddings as JSON strings introduces unnecessary overhead in storage size, CPU deserialization cycles, and numeric formatting precision.
---
### Impact
1. **Storage Reduction**: 512-dim `float32` vectors take 2 KB as binary BLOB vs 8–10 KB as JSON text which will be crucial in case of large data.
2. **Fast Deserialization**: `np.frombuffer()` eliminates the CPU overhead of `json.loads()` when querying thousands of faces for clustering and search.
---
### Backward Compatibility
- **Startup Data Migration**: In `db_create_faces_table()`, migrate existing `typeof(embeddings) = 'text'` rows to BLOB in-place.
- **Dual-mode Deserializer**: Add a fallback helper supporting both `bytes` (`np.frombuffer`) and legacy `str` (`json.loads`).
#### Proposed Changes
- faces.py: Update insertion and reader queries to write/read BLOB, plus add one-time migration logic.
- test_faces_db.py: Add tests for BLOB storage and legacy JSON backward compatibility.
### Add ScreenShots
@Maintainer I would like to solve this.
### Record
- [x] I agree to follow this project's Code of Conduct
- [x] I want to work on this issue
Guia de contribuição
Avaliação
Esta issue ainda não foi avaliada.