pingcap / pingcap/tidb-operator

Skip rclone in the `tidb-lightning` chart

Open
#6,142 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
1.3k
Forks
540
Avg merge
3d 2h
Merged PRs (30d)
18

Description

Feature Request

Is your feature request related to a problem? Please describe:

https://github.com/pingcap/tidb-operator/blob/639482c0d2b92561b274bcc0c58df47716776103/charts/tidb-lightning/values.yaml#L30-L31

To import from cloud storage, as of current version (v1.7.0-alpha.3) of TiDB Lightning Helm Chart it relies on rclone to copy those files locally. This was introduced 6 years ago #817, but since then Lightning has supported downloading from cloud storage natively.

The additional rclone pod complicates deployment and credential configuration.

Describe the feature you'd like:

Make it possible to directly use an s3:// / gcs:// etc data source directly, without any volume mount for the data-source

None of the current dataSource works:

  • local only accepts an actual path for hostPath
  • remote is rclone-based
  • adhoc requires a PVC

Describe alternatives you've considered:

We just give up on the Helm chart and schedule a custom Lightning job.

Teachability, Documentation, Adoption, Migration Strategy:

pingcap/docs-tidb-operator#2727

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with charts/tidb-lightning/values.yaml around the dataSource settings at lines 30-31, then trace the chart templates for the local, remote, and adhoc modes and the rclone pod. Confirm how the chart currently handles cloud sources and volume mounts. Done means an s3://, gcs://, or similar source can be used directly without a data-source volume or rclone pod.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.