4paradigm / 4paradigm/OpenMLDB

sbin deploy can't set HADOOP_CONF_DIR for taskmanager

Open
#3,159 2 comments 0 reactions 0 assignees View on GitHub
enhancement high-priority tools
Dominant language
C++
Stars
1.7k
Forks
331
Avg merge
12d 12h
Merged PRs (30d)
1

Description

and spark extra conf, e.g. hdfs-site.xml, needs to add in spark/conf before deploy. -- HADOOP_CONF_DIR设置为openmldb/conf应该可以?
但taskmanager读取需要start.sh时有环境,sbin不会传去host里
just deploy conf for fix config options

More

stat-all.sh没有recover工具时最好failed,除非force执行,不然用户很容易重启后表有问题,但当时无法察觉
start-all.sh里的recovering没有进度,前台也没法看,没有多开shell时很麻烦
recovering本来就是在tablet、nameserver都启动后的操作,在线集群可以独立recovering,不做不必要的stop

yarn job如果日志出现connect 0.0.0.0:8032,说明yarn-site.xml没有正确load。使用了默认yarn地址。

Contributor guide

Open the contributing guide

Research direction

Look at sbin deploy scripts to see how environment variables like HADOOP_CONF_DIR are passed to taskmanager. Examine start.sh and related host deployment logic. The fix likely involves modifying deployment scripts to propagate configuration. Check spark/conf handling for hdfs-site.xml. Also review stat-all.sh and start-all.sh for recovery behavior improvements mentioned in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, shell, spark
Domain
backend, cli, devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.