Proposal: Redesign of Pegasus Scanner

## Proposal Redesign of Pegasus Scanner

### Background

Pegasus provides three interfaces `on_get_scanner` `on_scan` and `on_clear_scanner` , for clients to execute scanning tasks.

If we want to full scan the whole table, at first, the client will call `on_get_scanner` on each partition, and then partitions return a `context_id` which is a random number generated by the server to record some parameters such as `hash_key_filter_type`, `batch_size` and the context of this scanning task.

Secondly, the client uses this `context_id` to call `on_scan` and completes scanning in the corresponding partition in turn. Servers will scan the whole data of the table on the disk, and return compliant value to the client in batches.

If the tasking end or any error happened, the client will call `on_clear_scanner` to clear its context_id on the server.

### Problem Statement

In actual use, such a design will cause some problems.

1. **prefix scan is too slow**

If we execute this scanning task:

```shell
full_scan --hash_key_filter_type prefix --hash_key_filter_pattern 2021-04-21
```

Server will scan all the data in the table, then returns the prefix match key of the pattern. But we can speed it up by using prefix seeking futures of RocksDB.

2. **scanning task is easily failed**

Although we have a batch size to limit the scan time, it does not work if the data is sparse. In the case above, we need to scan almost the whole partition but it is possible that there is no row which matches the prefix,then it will be easy to timeout.

### Proposal

**For problem 1**

1. Pegasus store key schema in RocksDB is like `[hashkey_len(2bytes)][hashkey][sortkey]`, so we can't directly use prefix seeking. But we can prefix seek `[01][prefix_pattern]`,`[02][prefix_pattern]`,`[03][prefix_pattern]`...`[65535][prefix_pattern]` in RocksDB.
2. Client can parallelly scan all the partitions instead of one by one.

**For problem 2**

1. We can set a `HeartbeatCheck` during scanning like [Hbase StoreScanner](https://github.com/apache/hbase/blob/048ca4e43fdf8b341c9ade5a9d455f627fc76041/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java#L591), pegasus sever sends heartbeat packets periodically to avoid timeout, which performed like a stream.

2. We can change the way to count batch size: compliant value number -> already scan value number

No repository files or tests are named. Start by tracing the on_get_scanner, on_scan, and on_clear_scanner entry points and the full_scan client; review the RocksDB key layout and the referenced Hbase StoreScanner heartbeat behavior. Done requires an agreed design and implementation covering prefix scans, partition parallelism, heartbeat handling, and scan-count batching.

Written by the indexing model from the issue text.