[Feature] Roadmap for Paimon C++ 0.4.0
- Lenguaje dominante
- C++
- Estrellas
- 65
- Forks
- 25
- Merge medio
- 2 d 12 h
- PR fusionados (30 d)
- 80
Descripción
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar.
### Motivation
Looking ahead, Paimon C++ will focus on several areas:
- Storage and retrieval optimizations for vertical workloads, including efficient MAP storage, richer data types such as VECTOR, and complete index support.
- Making newly written data queryable in real time through pluggable real-time writes and memory/disk union reads.
- Following up on important capabilities from the broader Paimon community, including changelog production, Format Table, and extensions to Data Evolution.
### Solution
#### 1. Storage and retrieval optimizations for vertical workloads
- [x] Support shared-shredding columnar storage for `MAP` based on [PIP-43](https://cwiki.apache.org/confluence/spaces/PAIMON/pages/430408347/PIP-43+Columnar+Storage+Optimization+for+MAP+Type+in+Paimon), including adaptive physical columns, field mappings, overflow storage, and end-to-end reads and writes.
- [ ] Support richer data types, starting with `VECTOR` based on [PIP-40](https://cwiki.apache.org/confluence/spaces/PAIMON/pages/399279132/PIP-40+Introduce+a+new+Vector+data+type), including schema representation, storage, reads, writes, and Data Evolution.
- [ ] Complete File Index support by adding index generation to the existing read path.
- [ ] Continue improving storage layout, predicate pushdown, column pruning, point lookups, and index-assisted retrieval for workload-specific scenarios.
#### 2. Making real-time data queryable
- [ ] Support pluggable real-time writes and memory/disk union reads based on [PIP-46](https://cwiki.apache.org/confluence/spaces/PAIMON/pages/444334302/PIP-46+Support+pluggable+real-time+writes+and+memory+disk+union+reads+for+Paimon+C), tracked by [#158](https://github.com/apache/paimon-cpp/issues/158).
- [ ] Make data held by an active writer queryable before it is committed into a snapshot.
- [ ] Guarantee consistent query results without missing or duplicate rows while writing, committing, and reclaiming memory segments concurrently.
- [ ] Support append tables first, followed by primary-key tables, deletion-vector mode, and additional production scenarios.
#### 3. Following up on Paimon community capabilities
- [ ] Support changelog production, including `input`, `lookup`, and `full-compaction` modes, tracked by [#174](https://github.com/apache/paimon-cpp/issues/174).
- [ ] Support Format Table reads and writes for Hive-style file directories, tracked by [#170](https://github.com/apache/paimon-cpp/issues/170).
- [ ] Extend Data Evolution to work with deletion vectors, tracked by [#169](https://github.com/apache/paimon-cpp/issues/169).
- [ ] Extend Data Evolution to primary-key tables and support compaction across evolved field groups, tracked by [#204](https://github.com/apache/paimon-cpp/issues/204).
- [ ] Continue following relevant Paimon features and make them available through native Paimon C++ APIs where appropriate.
Each roadmap item should be delivered through focused issues and reviewable pull requests, with unit tests, integration tests, documentation, and compatibility validation where applicable.
### Anything else?
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Guía de contribución
Línea de trabajo
Comienza con las issues específicas enlazadas, especialmente #158 para escrituras en tiempo real y lecturas de unión, #174 para la generación del changelog, #170 para Format Table y #169 o #204 para Data Evolution. Lee los documentos PIP referenciados cuando corresponda y elige un elemento de roadmap con un alcance definido. Para darlo por terminado se requiere un cambio específico y revisable, con las pruebas unitarias y de integración pertinentes, documentación y validación de compatibilidad.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- cpp
- Área
- databases
- Tipo de issue
- Nueva funcionalidad
- Dificultad
- 5/5
- Tiempo estimado
- Más de una semana
- Estado de actividad
- Tranquilo
- Claridad
- Necesita aclaración
- Aptitud para principiantes
- 25/100