apache / apache/beam

[Feature Request]: Support reading from BigTable Continuous Materialized Views in BigtableIO

Open
#38,053 3 comments 0 reactions 1 assignee Claimed by @psud View on GitHub
io new feature P2 stale
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What would you like to happen?

The BigTable connector via `BigtableIO.Read` currently only supports reading from BigTable tables via `withTableId()`. There is no way to read from the new [continuous materialized view tables](https://cloud.google.com/bigtable/docs/continuous-materialized-views) (CMV), which are a separate resource type (`materializedViews/` vs `tables/`).

The underlying google-cloud-bigtable Java client already supports CMVs — `Query.create()` and sampleRowKeys() both accept materialized view resource names as of [v2.55.0](https://github.com/googleapis/java-bigtable/releases/tag/v2.55.0). However, `BigtableIO.Read `has no option to specify a materialized view name, so this capability is not accessible from Beam pipelines.

**Use case:**
We use BigTable CMVs to maintain pre-aggregated index views over our data. We need to read from these views in Dataflow pipelines for downstream processing. Currently we can only access CMVs via the REST API or the Java client directly, but not through BigtableIO.

**Expected behavior:**
A `withMaterializedViewName(String)` method (or similar) on `BigtableIO.Read`, mutually exclusive with withTableId(), that routes the read through the existing client support for materialized views.

### Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

### Issue Components

- [ ] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [x] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.