kubeflow / kubeflow/docs-agent
Replace MinIO with SeaweedFS as Milvus Object Storage Backend
- Dominant language
- Python
- Stars
- 42
- Forks
- 111
- Avg merge
- 6d 23m
- Merged PRs (30d)
- 2
Description
## Overview
This issue proposes replacing MinIO with SeaweedFS as the S3-compatible object storage backend for our Milvus deployment. While Milvus itself remains the vector database, the underlying storage layer for segments and indexes should be migrated to SeaweedFS to address licensing concerns and optimize for our specific workload.
## Motivation
### 1. Licensing Compliance (Primary Driver)
MinIO recently transitioned its open-source license from **Apache 2.0** to **AGPL v3**.
- **AGPL v3** is a strong copyleft license that can impose significant compliance obligations, potentially affecting the distribution and deployment of our software stack.
- **SeaweedFS** is licensed under **Apache 2.0**, which is permissive and fully compatible with our project's licensing requirements. Using SeaweedFS eliminates the legal ambiguity and risk associated with AGPL v3 dependencies.
### 2. Performance Optimization for Small Files
Milvus architecture involves writing numerous small files (segments, index chunks, logs) to object storage.
- **MinIO** is optimized for large sequential throughput but can have higher overhead for massive numbers of small files (the "lots of small files" problem).
- **SeaweedFS** is specifically designed to handle billions of files efficiently. It achieves $O(1)$ disk seek time for file access by storing file metadata in memory (or LevelDB/RocksDB) and appending file data to large volumes. This architecture aligns perfectly with the write patterns of a vector database like Milvus.
### 3. Resource Efficiency
- SeaweedFS generally exhibits a lighter memory and CPU footprint compared to MinIO for comparable workloads, making it more cost-effective for both development and production environments.
## Technical Feasibility
SeaweedFS provides a high-fidelity **S3-compatible API**.
- Milvus connects to object storage using the standard S3 protocol.
- By configuring the SeaweedFS S3 gateway, we can use it as a drop-in replacement.
- No changes to Milvus application logic or our internal Python code (`pymilvus` usage) are required. The migration is purely at the infrastructure/configuration level.
## Proposed Implementation
1. **Container Swap**: Replace the `minio/minio` container in our Kubernetes manifests and Docker Compose files with `chrislusf/seaweedfs`.
2. **Configuration**: Update Milvus configuration env vars (`MINIO_ADDRESS`, etc.) to point to the SeaweedFS S3 endpoint (default port `8333`).
3. **Data Persistence**: Ensure SeaweedFS volume and filer data are persisted to appropriate PersistentVolumes.
## Verification Plan
To ensure a safe migration, the following verification steps have been performed (and are automated in the corresponding PR):
1. **S3 API Compliance**: Verify basic S3 operations (create bucket, put/get/delete object) against SeaweedFS.
2. **Milvus Connectivity**: Confirm Milvus establishes a connection to the storage backend without errors.
3. **End-to-End Vector Operations**:
- Create Collection (using project-specific 768-dim schema).
- Insert Vectors (ensure data is flushed to storage).
- Build Index (`IVF_FLAT` / `COSINE`).
- Execute ANN Search (validate accuracy of results).
- Perform Deletions and Updates.
- Validate Persistence (restart Milvus and ensure data is still queryable).
## Deliverables
- [ ] `docker-compose.yaml` for local development.
- [ ] Updated Kubernetes `Deployment` manifests.
- [ ] Verification test proving functional parity.
Contributor guide
Research direction
Start by locating docker-compose.yaml and the Kubernetes Deployment manifests mentioned in the deliverables, then review the Milvus storage environment variables and existing persistence settings. Verify SeaweedFS S3 create, put, get, and delete operations before testing Milvus connectivity and the listed vector operations. Done means both deployment paths use persistent SeaweedFS storage and the verification test passes after a restart.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker-compose, kubernetes, python
- Domain
- databases, devops, infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100