AOSSIE-Org / AOSSIE-Org/PictoPy

Feat:Optimize face embedding storage in SQLite using raw float32 BLOB with backward compatibility

未關閉
#1,497 7 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
enhancement
主要語言
Python
星號
283
分支
679
平均合併
7 天 2 小時
30 天內合併 PR
3

描述

### Describe the feature

### Issue
Currently, `faces.py` stores face embeddings as JSON strings (`TEXT`), whereas `image_embeddings.py` and `video_frames.py` already store embeddings as raw `float32` BLOBs .

Storing embeddings as JSON strings introduces unnecessary overhead in storage size, CPU deserialization cycles, and numeric formatting precision.

---

### Impact
1. **Storage Reduction**: 512-dim `float32` vectors take 2 KB as binary BLOB vs 8–10 KB as JSON text which will be crucial in case of large data.
2. **Fast Deserialization**: `np.frombuffer()` eliminates the CPU overhead of `json.loads()` when querying thousands of faces for clustering and search.

---

### Backward Compatibility
- **Startup Data Migration**: In `db_create_faces_table()`, migrate existing `typeof(embeddings) = 'text'` rows to BLOB in-place.
- **Dual-mode Deserializer**: Add a fallback helper supporting both `bytes` (`np.frombuffer`) and legacy `str` (`json.loads`).

#### Proposed Changes
- faces.py: Update insertion and reader queries to write/read BLOB, plus add one-time migration logic.
- test_faces_db.py: Add tests for BLOB storage and legacy JSON backward compatibility.

### Add ScreenShots

@Maintainer I would like to solve this.

### Record

- [x] I agree to follow this project's Code of Conduct
- [x] I want to work on this issue

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。