AOSSIE-Org / AOSSIE-Org/PictoPy
Feat:Optimize face embedding storage in SQLite using raw float32 BLOB with backward compatibility
- 主要语言
- Python
- 星标
- 283
- 派生
- 679
- 平均合并
- 7 天 2 小时
- 30 天内合并 PR
- 3
描述
### Describe the feature
### Issue
Currently, `faces.py` stores face embeddings as JSON strings (`TEXT`), whereas `image_embeddings.py` and `video_frames.py` already store embeddings as raw `float32` BLOBs .
Storing embeddings as JSON strings introduces unnecessary overhead in storage size, CPU deserialization cycles, and numeric formatting precision.
---
### Impact
1. **Storage Reduction**: 512-dim `float32` vectors take 2 KB as binary BLOB vs 8–10 KB as JSON text which will be crucial in case of large data.
2. **Fast Deserialization**: `np.frombuffer()` eliminates the CPU overhead of `json.loads()` when querying thousands of faces for clustering and search.
---
### Backward Compatibility
- **Startup Data Migration**: In `db_create_faces_table()`, migrate existing `typeof(embeddings) = 'text'` rows to BLOB in-place.
- **Dual-mode Deserializer**: Add a fallback helper supporting both `bytes` (`np.frombuffer`) and legacy `str` (`json.loads`).
#### Proposed Changes
- faces.py: Update insertion and reader queries to write/read BLOB, plus add one-time migration logic.
- test_faces_db.py: Add tests for BLOB storage and legacy JSON backward compatibility.
### Add ScreenShots
@Maintainer I would like to solve this.
### Record
- [x] I agree to follow this project's Code of Conduct
- [x] I want to work on this issue
贡献指南
评估
这个 Issue 还没有评估数据。