apache / apache/hudi

The hudi docker demo failed to execute the delta-streamer and ingest to stock_ticks_cow table in HDFS

Open
#15,214 0 comments 0 reactions 0 assignees View on GitHub
area:ingest from-jira priority:medium type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

When I execute the following code in container adhoc-2:
{code:java}
spark-submit \
  --class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer $HUDI_UTILITIES_BUNDLE \
  --table-type COPY_ON_WRITE \
  --source-class org.apache.hudi.utilities.sources.JsonKafkaSource \
  --source-ordering-field ts  \
  --target-base-path /user/hive/warehouse/stock_ticks_cow \
  --target-table stock_ticks_cow --props /var/demo/config/kafka-source.properties \
  --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider {code}
An error is as follows:
{code:java}
root@adhoc-2:/opt# spark-submit \
>   --class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer $HUDI_UTILITIES_BUNDLE \
>   --table-type COPY_ON_WRITE \
>   --source-class org.apache.hudi.utilities.sources.JsonKafkaSource \
>   --source-ordering-field ts  \
>   --target-base-path /user/hive/warehouse/stock_ticks_cow \
>   --target-table stock_ticks_cow --props /var/demo/config/kafka-source.properties \
>   --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider
Exception in thread "main" org.apache.spark.SparkException: Cannot load main class from JAR file:/opt/%C2%A0
    at org.apache.spark.deploy.SparkSubmitArguments.error(SparkSubmitArguments.scala:657)
    at org.apache.spark.deploy.SparkSubmitArguments.loadEnvironmentArguments(SparkSubmitArguments.scala:221)
    at org.apache.spark.deploy.SparkSubmitArguments.(SparkSubmitArguments.scala:116)
    at org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$1.(SparkSubmit.scala:907)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.parseArguments(SparkSubmit.scala:907)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:81)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) {code}
When I check the environment variable $HUDI_UTILITIES_BUNDLE, I got this:
{code:java}
root@adhoc-2:/opt# echo $HUDI_UTILITIES_BUNDLE
/var/hoodie/ws/docker/hoodie/hadoop/hive_base/target/hoodie-utilities.jar {code}
But, I can't find the jar file:
{code:java}
root@adhoc-2:/opt# ls -ltr /var/hoodie/ws/docker/hoodie/hadoop/hive_base/target/hoodie-utilities.jar
ls: cannot access '/var/hoodie/ws/docker/hoodie/hadoop/hive_base/target/hoodie-utilities.jar': No suchfile or directory {code}
When I try find this:
{code:java}
root@adhoc-2:/opt# find /var/hoodie/ws -name "hudi-utilities-bundle*.0.jar" | xargs ls -ltr
-rw-r--r-- 1 root root 60631874 Jun  8 07:41 /var/hoodie/ws/hudi-examples/hudi-examples-spark/target/lib/hudi-utilities-bundle_2.11-0.11.0.jar
-rw-r--r-- 1 root root 60631874 Jun  8 07:41 /var/hoodie/ws/hudi-cli/target/lib/hudi-utilities-bundle_2.11-0.11.0.jar
-rw-r--r-- 1 root root 60631874 Jun  8 07:41 /var/hoodie/ws/packaging/hudi-utilities-bundle/target/hudi-utilities-bundle_2.11-0.11.0.jar {code}
So I tried to modify the environment variable  $HUDI_UTILITIES_BUNDLE, and resubmit the command, it worked:
{code:java}
root@adhoc-2:/opt# spark-submit \
>   --class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer $HUDI_UTILITIES_BUNDLE \
>   --table-type COPY_ON_WRITE \
>   --source-class org.apache.hudi.utilities.sources.JsonKafkaSource \
>   --source-ordering-field ts  \
>   --target-base-path /user/hive/warehouse/stock_ticks_cow \
>   --target-table stock_ticks_cow --props /var/demo/config/kafka-source.properties \
>   --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider
22/06/09 01:43:34 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/06/09 01:43:35 WARN SchedulerConfGenerator: Job Scheduling Configs will not be in effect as spark.scheduler.mode is not set to FAIR at instantiation time. Continuing without scheduling configs
22/06/09 01:43:36 WARN DFSPropertiesConfiguration: Cannot find HUDI_CONF_DIR, please set it as the dirof hudi-defaults.conf
22/06/09 01:43:36 WARN DFSPropertiesConfiguration: Properties file file:/etc/hudi/conf/hudi-defaults.conf not found. Ignoring to load props file
22/06/09 01:43:36 WARN SparkContext: Using an existing SparkContext; some configuration may not take effect.
22/06/09 01:43:37 WARN KafkaUtils: overriding enable.auto.commit to false for executor
22/06/09 01:43:37 WARN KafkaUtils: overriding auto.offset.reset to none for executor
22/06/09 01:43:37 ERROR KafkaUtils: group.id is null, you should probably set it
22/06/09 01:43:37 WARN KafkaUtils: overriding executor group.id to spark-executor-null
22/06/09 01:43:37 WARN KafkaUtils: overriding receive.buffer.bytes to 65536 see KAFKA-3135
22/06/09 01:43:38 WARN HoodieBackedTableMetadata: Metadata table was not found at path /user/hive/warehouse/stock_ticks_cow/.hoodie/metadata
00:05  WARN: Timeline-server-based markers are not supported for HDFS: base path /user/hive/warehouse/stock_ticks_cow.  Falling back to direct markers.
00:06  WARN: Timeline-server-based markers are not supported for HDFS: base path /user/hive/warehouse/stock_ticks_cow.  Falling back to direct markers.
00:08  WARN: Timeline-server-based markers are not supported for HDFS: base path /user/hive/warehouse/stock_ticks_cow.  Falling back to direct markers. {code}
I could view the data had been written in the HDFS:
{code:java}
root@adhoc-2:/opt# hdfs dfs -ls /user/hive/warehouse/stock_ticks_cow/*/*/*/*
Found 1 items
drwxr-xr-x   - root supergroup          0 2022-06-09 01:43 /user/hive/warehouse/stock_ticks_cow/.hoodie/metadata/.hoodie/.aux/.bootstrap
-rw-r--r--   1 root supergroup       8056 2022-06-09 01:43 /user/hive/warehouse/stock_ticks_cow/.hoodie/metadata/.hoodie/00000000000000.deltacommit
-rw-r--r--   1 root supergroup       3035 2022-06-09 01:43 /user/hive/warehouse/stock_ticks_cow/.hoodie/metadata/.hoodie/00000000000000.deltacommit.inflight
-rw-r--r--   1 root supergroup          0 2022-06-09 01:43 /user/hive/warehouse/stock_ticks_cow/.hoodie/metadata/.hoodie/00000000000000.deltacommit.requested
-rw-r--r--   1 root supergroup       8139 2022-06-09 01:43 /user/hive/warehouse/stock_ticks_cow/.hoodie/metadata/.hoodie/20220609014338711.deltacommit
-rw-r--r--   1 root supergroup       3035 2022-06-09 01:43 /user/hive/warehouse/stock_ticks_cow/.hoodie/metadata/.hoodie/20220609014338711.deltacommit.inflight
-rw-r--r--   1 root supergroup          0 2022-06-09 01:43 /user/hive/warehouse/stock_ticks_cow/.hoodie/metadata/.hoodie/20220609014338711.deltacommit.requested
-rw-r--r--   1 root supergroup        599 2022-06-09 01:43 /user/hive/warehouse/stock_ticks_cow/.hoodie/metadata/.hoodie/hoodie.properties
-rw-r--r--   1 root supergroup        124 2022-06-09 01:43 /user/hive/warehouse/stock_ticks_cow/.hoodie/metadata/files/.files-0000_00000000000000.log.1_0-0-0
-rw-r--r--   1 root supergroup      21951 2022-06-09 01:43 /user/hive/warehouse/stock_ticks_cow/.hoodie/metadata/files/.files-0000_00000000000000.log.1_0-10-10
-rw-r--r--   1 root supergroup         93 2022-06-09 01:43 /user/hive/warehouse/stock_ticks_cow/.hoodie/metadata/files/.hoodie_partition_metadata
-rw-r--r--   1 root supergroup         96 2022-06-09 01:43 /user/hive/warehouse/stock_ticks_cow/2018/08/31/.hoodie_partition_metadata
-rw-r--r--   1 root supergroup     436884 2022-06-09 01:43 /user/hive/warehouse/stock_ticks_cow/2018/08/31/7610b058-8df2-484a-ba70-881feef7195e-0_0-36-35_20220609014338711.parquet {code}
So my question is whether I need to modify $HUDI_UTILITIES_BUNDLE ?

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-4211
- Type: Bug

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by checking how HUDI_UTILITIES_BUNDLE is set in the Docker demo container and compare its value with the bundle paths found under /var/hoodie/ws. Re-run the shown spark-submit command with the documented environment setup; done means the demo resolves the utilities JAR and ingests data into stock_ticks_cow without manually changing the variable.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, hadoop, java, spark
Domain
data-engineering, devops, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.