The `pagecheckpoint` tool reading from S3 path is broken
Open
Nobody has claimed this yet.
component/storage
severity/moderate
type/bug
- Dominant language
- C++
- Stars
- 1k
- Forks
- 423
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 24
Description
Bug Report
Please answer these questions before submitting your issue. Thanks!
1. Minimal reproduce step (Required)
> mc ls k79/jayson/j20/s122/manifest/mf_40
[2024-03-01 15:09:39 CST] 133KiB STANDARD mf_40
-- parsing the manifest on local disk is OK
> mc cp k79/jayson/j20/s122/manifest/mf_40 ./
> tiup tiflash:v7.5.1 pagecheckpoint --manifest mf_40 | head
file_format: 1
local_sequence: 48929
last_local_sequence: 48925
create_at_ms: 1709276979024
writer_info {
...
-- parsing the manifest on S3 is failed
> tiup tiflash:v7.5.1 pagecheckpoint --manifest s3://j20/s123/manifest/mf_43 --endpoint=http://10.2.12.79:9000 --access_key_id <> --secret_access_key <> --bucket jayson
The component `tiflash` version v7.5.1 is not installed; downloading from repository.
download https://tiup-mirrors.pingcap.com/tiflash-v7.5.1-linux-amd64.tar.gz 250.59 MiB / 250.59 MiB 100.00% 39.20 MiB/s
Starting component `tiflash`: /DATA/disk1/ra_common/.tiup/components/tiflash/v7.5.1/tiflash/tiflash pagecheckpoint --manifest s3://j20/s123/manifest/mf_43 --endpoint=http://10.2.12.79:9000 --access_key_id minioadmin --secret_access_key --bucket jayson
Code: 49, e.displayText() = DB::Exception: Check file_path_.find("://") == std::string::npos failed: Unsupported protocol in path s3://j20/s123/manifest/mf_43, e.what() = DB::Exception, Stack trace:
0x1eb246e DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, int) [tiflash+32187502]
dbms/src/Common/Exception.h:46
0x78568f2 DB::FileProvider::newRandomAccessFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::EncryptionPath const&, std::__1::shared_ptr<DB::ReadLimiter> const&, int) const [tiflash+126183666]
dbms/src/Encryption/FileProvider.cpp:47
0x20fc919 DB::PS::CheckpointTool::mainEntry(int, char**) [tiflash+34588953]
dbms/src/Storages/Page/tools/Checkpoint/CheckpointTool.cpp:206
0x1e9ec4c main [tiflash+32107596]
dbms/src/Server/main.cpp:173
0x7f7abf385eb0 __libc_start_call_main [libc.so.6+261808]
0x7f7abf385f60 __libc_start_main_alias_2 [libc.so.6+261984]
0x1b873e9 <unknown symbol> [tiflash+28865513]
2. What did you expect to see? (Required)
pagecheckpoint can parse the manifest from S3 directly, ref https://github.com/pingcap/tiflash/pull/7914
3. What did you see instead (Required)
Errors as above
4. What is your TiFlash version? (Required)
v7.5.1
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at dbms/src/Storages/Page/tools/Checkpoint/CheckpointTool.cpp:206 and inspect dbms/src/Encryption/FileProvider.cpp:47, using the S3 reproduction and PR 7914 as context. Done means pagecheckpoint parses an S3 manifest directly instead of rejecting the s3:// path; verify with the command shown in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- cli
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100