SHAREdis, a redis protocol compatible layer for TiKV

Project Incubating Request

Summary

As a caching service, Redis has become the first choice for most companies with its efficient performance, rich data types, and comprehensive interfaces. However, since all data in Redis is in memory, when the amount of data is huge, the cost will be very expensive. Therefore, many companies have developed their own caching services based on open source frameworks such as RocksDB. With the high-efficiency performance of SSD disks and the append log mode of the LSM tree, the service cost is greatly reduced at the cost of a certain loss of read and write efficiency.

Because self-developed kv store often requires large R&D and time costs, TiKV, as an excellent open source KV storage service, has powerful cluster management and data synchronization functions, making it a suitable choice. But firstly, TiKV only supports reading and writing of key-value type data, which greatly limits business application scenarios. Secondly, TiKV uses a unique client protocol. If the business is switched from Redis, the sdk needs to be changed, and the service migration cost is high.

This project hopes to encapsulate and expand the key-value interface of TiKV on top of the advantages of TiKV, and use the Redis client protocol to implement Redis data types and interface operations, while shielding the complex cluster management inside the distributed system and data synchronization operation, thereby providing a distributed SSD cache computing layer service that can replace/supplement Redis.

Motivation

This proposal mainly solves the following problems:

1. TiKV only supports reading and writing of key-value type data, which greatly limits business application scenarios
2. TiKV uses a unique client protocol. If the business is switched from Redis, the sdk needs to be changed, which needs high R&D costs.
3. Storage and computing are not separated and do not have elastic scalability

Current Situation

1. Redis data type & interface support: implements some commonly used types, such as kv, set, hash.
2. The proxy layer is developed using Go, the client-service interaction protocol is thrift, and currently supports c++, java, go and python
3. The existing design and code will inevitably have potential deficiencies and bugs

Targets

1. The support of Redis protocol, aiming to realize all common types & interfaces
2. The client can use the unified SDK to call Redis & TiKV
3. The identification and separate storage of cold and hot data can be automatically derived based on statistical analysis of calls. Users can manually or automatically put cold data in kv store and hot data in redis without additional development costs

Architecture diagram

Estimated Time

No files, tests, or concrete entry points are named. Start by reviewing the architecture diagram and the stated proxy layer, Redis protocol, thrift interaction, and current Redis type support; the scope would need to be narrowed before work can be considered done.