apache / apache/hudi

Explore fat jar option for gcs-connector lib used during GCS Ingestion

Open
#15,458 0 comments 0 reactions 0 assignees View on GitHub
area:ingest from-jira priority:high type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Currently, the GCS Ingestion (HUDI-4850) expects recent versions of Jars like protobuf and Guava to be provided to spark-submit explicitly, to override older versions shipped with Spark. These Jars are used by the gcs-connector which is a library from Google that helps connect to GCS. For more details see [https://docs.google.com/document/d/1VfvtdvhXw6oEHPgZ_4Be2rkPxIzE0kBCNUiVDsXnSAA/edit#] (section titled "Configure Spark to use newer versions of some Jars").

See if it's possible to create a shaded+fat jar of gcs-connector for this use case instead, and avoid specifying things to spark-submit on the command line.

An alternate approach to consider for the long term is HUDI-4930 (slim bundles).

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-4931
- Type: Task
- Epic: https://issues.apache.org/jira/browse/HUDI-1896

---

## Comments

28/Sep/22 08:14;pramodbiligiri;Some useful references regarding this:
- GCP docs on Cloud Storage connector: [https://cloud.google.com/dataproc/docs/concepts/connectors/cloud-storage]
- Hudi docs on GCS connectivity: https://hudi.apache.org/docs/gcs_hoodie/

 ;;;

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the GCS Ingestion context in HUDI-4850 and the linked GCS connector, GCP Cloud Storage connector, and Hudi GCS connectivity documentation. Investigate whether a shaded, fat gcs-connector jar can replace the explicit protobuf and Guava spark-submit dependencies. Done means documenting or implementing a viable approach that avoids those command-line overrides, while considering HUDI-4930 as the long-term alternative.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, java, spark
Domain
cloud, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.