apache / apache/incubator-pegasus
support new “data export” feature to replace using backup
- Dominant language
- C++
- Stars
- 2.1k
- Forks
- 328
- PR merge metrics
- No merged PRs in 30d
Description
## Feature Request
**Is your feature request related to a problem? Please describe:**
Now if we want to dump/export data fastly, we need use [backup](http://pegasus.apache.org/administration/cold-backup). However, the backup feature is not friendly for dumping data, for example:
* Don't support dump immediately, we need create a task and wait excuting
* The path contain redundant sub-dir such as `policy name` etc. and don‘t support custom path
Above and some other question result in dumping data is complex and especially using [Pegasus-Spark](https://github.com/pegasus-kv/pegasus-spark) to read the dumped data.
**Describe the feature you'd like:**
I expect the command line should be simple as follow:
```shell
# dump/export once
# target is hdfs, sub-path is optional, default can be pegasus/cluster_name/table_name
>> dump/export table_name hdfs://url sub-path
# target is fds, sub-path is optional, default can be pegasus/cluster_name/table_name
>> dump/export table_name endpoint bucket sub-path
# dump/export periodicly
>> dump/export table_name hdfs://url sub-path start_time periodic_time
```
finaly, the data path is:
```
root/pegasus/cluster/table/timestamp/partition/file.sst
```
and then, user can use [Pegasus-Spark](https://github.com/pegasus-kv/pegasus-spark) to read data directly but no need we must offer the `policy name`,`cluster_name`, `table_name`, `fds/hdfs config`.
**Describe alternatives you've considered:**
Since the `cold backup` code is complex, so we no need refactor it to achive the above result, but can re-implement new feature
**Teachability, Documentation, Adoption, Migration Strategy:**
Contributor guide
Research direction
Start by tracing the existing cold backup implementation and its command-line entry points, then review how Pegasus-Spark consumes exported data. Define the scope for one-time and periodic exports to HDFS or FDS, including optional sub-paths and the requested timestamp/partition/file.sst layout; done means the commands work and the output is directly readable by Pegasus-Spark.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, spark
- Domain
- cli, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100