stackabletech / stackabletech/airflow-operator

External resources: replace PVC-usage with dependency sources

Open
#150 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
32
Forks
6
Avg merge
1d 7h
Merged PRs (30d)
13

Description

As a user I want to be able to make external resources and dependencies available to pods.

Previously this was implemented using PVCs, which was not independent of cluster capabilities (Access modes for PVCs etc.). A alternative solution is to dispense with PVCs and instead implement a dependencySources that will use a complex enum to expose different sources. In this example below, this will be an S3Bucket:-

---
apiVersion: s3.stackable.tech/v1alpha1
kind: S3Bucket
metadata:
  name: my-bucket-resource1
spec:
  bucketName: my-example-bucket1
  connection:
    reference: my-connection-resource
---
apiVersion: spark.stackable.tech/v1alpha1
kind: SparkApplication
...
spec:
  ...
  driver:
    ...
    dependencySources:
        # Use "#[serde(flatten)]" in a complex enum called e.g. "Source" 
        # allowing s3Bucket (and maybe hdfsFolder or url later on)
        - s3Bucket: # Option<S3BucketRef>
            inline: [...]
            # or
            reference: my-bucket-resource1

When a dependencySources is declared for a role, the contents of this source will be copied to an internally defined path (not visible to the user) on the respective pods, from where it can be accessed as required. In other words, where previously a job was started to copy resources to a (usually RWX) PVC - which was then used to back a volume mounted for roles - the same mechanism will be used without using shared storage. This adds some redundancy, but a) is independent of PV storage classes and b) is more transparent for the user.

If #177 has not been implemented, then:

  • remove PVC-usage from tests and documentation
    • N.B. inform @backstreetkiwi when this is complete so that node labels can be removed from T2 cluster nodes

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review the current PVC-based tests and documentation, and check whether issue #177 has been implemented before deciding on the remaining work. Done means dependencySources can expose external sources such as S3Bucket without shared PVC storage, or, if #177 is absent, PVC usage is removed from the tests and documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, rust
Domain
backend, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.