byzer-org / byzer-org/byzer-shell

Byzer Shell 默认配置文件复用 $BYZER_HOME/conf/byzer.properties

Open
#7 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
6
Forks
2
PR merge metrics
No merged PRs in 30d

Description

当前版本默认的配置文件是执行 `byzer-shell` 的目录下的 `.mlsql.config` 文件,这个和产品包的体系内容不搭,需要在 byzer-shell 能够以 `$BYZER_HOME/conf/byzer.properties` 和 `$BYZER_HOME/conf/byzer.properties.override` 文件来控制配置。

配置大概区分为三个部分
```
java.home=/Users/allwefantasy/projects/mlsql/dist/mlsql-lang/jdk8
engine.memory=6048m
# engine.url=http://127.0.0.1:9004/
# engine.url=http://127.0.0.1:9004/

engine.spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension
engine.spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog
engine.spark.shuffle.spill.batchSize=1000
engine.spark.shuffle.spill.numElementsForceSpillThreshold=10000
engine.spark.sql.shuffle.partitions=8
engine.spark.shuffle.accurateBlockThreshold=5242880
engine.spark.memory.fraction=0.1
engine.spark.memory.storageFraction=0.1
engine.spark.sql.inMemoryColumnarStorage.batchSize=1000
engine.spark.sql.files.maxPartitionBytes=5242880
engine.spark.sql.parquet.columnarReaderBatchSize=1000
engine.spark.sql.pivotMaxValues=1000
engine.spark.unsafe.sorter.spill.reader.buffer.size=1m
engine.spark.file.transferTo=false
engine.spark.shuffle.file.buffer=1m
engine.spark.shuffle.unsafe.file.ouput.buffer=1m
engine.spark.io.compression.lz4.blockSize=128k

user.owner=admin
user.access_token=123
user.outputSize=100
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.