AOSSIE-Org / AOSSIE-Org/PictoPy

Feat:Optimize face embedding storage in SQLite using raw float32 BLOB with backward compatibility

未关闭
#1,497 7 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
enhancement
主要语言
Python
星标
283
派生
679
平均合并
7 天 2 小时
30 天内合并 PR
3

描述

### Describe the feature

### Issue
Currently, `faces.py` stores face embeddings as JSON strings (`TEXT`), whereas `image_embeddings.py` and `video_frames.py` already store embeddings as raw `float32` BLOBs .

Storing embeddings as JSON strings introduces unnecessary overhead in storage size, CPU deserialization cycles, and numeric formatting precision.

---

### Impact
1. **Storage Reduction**: 512-dim `float32` vectors take 2 KB as binary BLOB vs 8–10 KB as JSON text which will be crucial in case of large data.
2. **Fast Deserialization**: `np.frombuffer()` eliminates the CPU overhead of `json.loads()` when querying thousands of faces for clustering and search.

---

### Backward Compatibility
- **Startup Data Migration**: In `db_create_faces_table()`, migrate existing `typeof(embeddings) = 'text'` rows to BLOB in-place.
- **Dual-mode Deserializer**: Add a fallback helper supporting both `bytes` (`np.frombuffer`) and legacy `str` (`json.loads`).

#### Proposed Changes
- faces.py: Update insertion and reader queries to write/read BLOB, plus add one-time migration logic.
- test_faces_db.py: Add tests for BLOB storage and legacy JSON backward compatibility.

### Add ScreenShots

@Maintainer I would like to solve this.

### Record

- [x] I agree to follow this project's Code of Conduct
- [x] I want to work on this issue

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。