pingcap / pingcap/tiflash

The `pagecheckpoint` tool reading from S3 path is broken

Open
#8,810 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

component/storage severity/moderate type/bug
Dominant language
C++
Stars
1k
Forks
423
Avg merge
1d 15h
Merged PRs (30d)
24

Description

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)
>  mc ls k79/jayson/j20/s122/manifest/mf_40
[2024-03-01 15:09:39 CST] 133KiB STANDARD mf_40

-- parsing the manifest on local disk is OK
> mc cp k79/jayson/j20/s122/manifest/mf_40 ./
> tiup tiflash:v7.5.1 pagecheckpoint --manifest mf_40 | head
file_format: 1
local_sequence: 48929
last_local_sequence: 48925
create_at_ms: 1709276979024
writer_info {
...

-- parsing the manifest on S3 is failed
>  tiup tiflash:v7.5.1 pagecheckpoint --manifest s3://j20/s123/manifest/mf_43 --endpoint=http://10.2.12.79:9000 --access_key_id <> --secret_access_key <> --bucket jayson
The component `tiflash` version v7.5.1 is not installed; downloading from repository.
download https://tiup-mirrors.pingcap.com/tiflash-v7.5.1-linux-amd64.tar.gz 250.59 MiB / 250.59 MiB 100.00% 39.20 MiB/s
Starting component `tiflash`: /DATA/disk1/ra_common/.tiup/components/tiflash/v7.5.1/tiflash/tiflash pagecheckpoint --manifest s3://j20/s123/manifest/mf_43 --endpoint=http://10.2.12.79:9000 --access_key_id minioadmin --secret_access_key --bucket jayson
Code: 49, e.displayText() = DB::Exception: Check file_path_.find("://") == std::string::npos failed: Unsupported protocol in path s3://j20/s123/manifest/mf_43, e.what() = DB::Exception, Stack trace:


       0x1eb246e        DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, int) [tiflash+32187502]
                        dbms/src/Common/Exception.h:46
       0x78568f2        DB::FileProvider::newRandomAccessFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::EncryptionPath const&, std::__1::shared_ptr<DB::ReadLimiter> const&, int) const [tiflash+126183666]
                        dbms/src/Encryption/FileProvider.cpp:47
       0x20fc919        DB::PS::CheckpointTool::mainEntry(int, char**) [tiflash+34588953]
                        dbms/src/Storages/Page/tools/Checkpoint/CheckpointTool.cpp:206
       0x1e9ec4c        main [tiflash+32107596]
                        dbms/src/Server/main.cpp:173
  0x7f7abf385eb0        __libc_start_call_main [libc.so.6+261808]
  0x7f7abf385f60        __libc_start_main_alias_2 [libc.so.6+261984]
       0x1b873e9        <unknown symbol> [tiflash+28865513]
2. What did you expect to see? (Required)

pagecheckpoint can parse the manifest from S3 directly, ref https://github.com/pingcap/tiflash/pull/7914

3. What did you see instead (Required)

Errors as above

4. What is your TiFlash version? (Required)

v7.5.1

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at dbms/src/Storages/Page/tools/Checkpoint/CheckpointTool.cpp:206 and inspect dbms/src/Encryption/FileProvider.cpp:47, using the S3 reproduction and PR 7914 as context. Done means pagecheckpoint parses an S3 manifest directly instead of rejecting the s3:// path; verify with the command shown in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.