apache / apache/incubator-pegasus

support new “data export” feature to replace using backup

Open
#582 0 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
C++
Stars
2.1k
Forks
328
PR merge metrics
No merged PRs in 30d

Description

## Feature Request

**Is your feature request related to a problem? Please describe:**

Now if we want to dump/export data fastly, we need use [backup](http://pegasus.apache.org/administration/cold-backup). However, the backup feature is not friendly for dumping data, for example:
* Don't support dump immediately, we need create a task and wait excuting
* The path contain redundant sub-dir such as `policy name` etc. and don‘t support custom path

Above and some other question result in dumping data is complex and especially using [Pegasus-Spark](https://github.com/pegasus-kv/pegasus-spark) to read the dumped data.

**Describe the feature you'd like:**

I expect the command line should be simple as follow:
```shell
# dump/export once
# target is hdfs, sub-path is optional, default can be pegasus/cluster_name/table_name
>> dump/export table_name hdfs://url sub-path
# target is fds, sub-path is optional, default can be pegasus/cluster_name/table_name
>> dump/export table_name endpoint bucket sub-path

# dump/export periodicly
>> dump/export table_name hdfs://url sub-path start_time periodic_time
```
finaly, the data path is:
```
root/pegasus/cluster/table/timestamp/partition/file.sst
```
and then, user can use [Pegasus-Spark](https://github.com/pegasus-kv/pegasus-spark) to read data directly but no need we must offer the `policy name`,`cluster_name`, `table_name`, `fds/hdfs config`.

**Describe alternatives you've considered:**

Since the `cold backup` code is complex, so we no need refactor it to achive the above result, but can re-implement new feature

**Teachability, Documentation, Adoption, Migration Strategy:**

Contributor guide

Open the contributing guide

Research direction

Start by tracing the existing cold backup implementation and its command-line entry points, then review how Pegasus-Spark consumes exported data. Define the scope for one-time and periodic exports to HDFS or FDS, including optional sub-paths and the requested timestamp/partition/file.sst layout; done means the commands work and the output is directly readable by Pegasus-Spark.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, spark
Domain
cli, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.