apache / apache/beam

[Feature Request]: Support for s3 as a cache store for InteractiveRunner(Flink) with jupyterlab

Open
#25,158 0 comments 0 reactions 0 assignees View on GitHub
awaiting triage flink new feature P2
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

### What would you like to happen?

Current only GCS appears to be supported location for distributed cachestore with interactive(flink) runner when using beam from jupterlab. (https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/interactive/README.md#status)
it would be helpful to also have support for s3, so that minio can be used as well.

### Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

### Issue Components

- [ ] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [X] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner

Contributor guide

Open the contributing guide

Research direction

Start with the InteractiveRunner documentation and the Interactive README section linked in the issue, then trace how the Flink runner configures the distributed cache store and its current GCS location. Done means InteractiveRunner can use an S3-compatible store, including MinIO, and the supported configuration is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter
Domain
cloud, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.