Skip to content

M
mobvista-dmp
Commit 911d96f8 by fan.jiang
Options

fix reyun_daily.sh

parent ec44af2e
Showing with 1 addition and 0 deletions
azkaban/ali/reyun/reyun_daily.sh
...@@ -28,6 +28,7 @@ spark-submit --class mobvista.dmp.datasource.reyun.ReyunDaily \ ...@@ -28,6 +28,7 @@ spark-submit --class mobvista.dmp.datasource.reyun.ReyunDaily \
--conf spark.default.parallelism=2000 \ --conf spark.default.parallelism=2000 \
--conf spark.sql.shuffle.partitions=2000 \ --conf spark.sql.shuffle.partitions=2000 \
--conf spark.sql.broadcastTimeout=1200 \ --conf spark.sql.broadcastTimeout=1200 \
--conf spark.hadoop.mapreduce.input.fileinputformat.input.dir.recursive=true \
--conf spark.sql.autoBroadcastJoinThreshold=31457280 \ --conf spark.sql.autoBroadcastJoinThreshold=31457280 \
--master yarn --deploy-mode cluster --executor-memory 6g --driver-memory 4g --executor-cores 3 --num-executors 60 \ --master yarn --deploy-mode cluster --executor-memory 6g --driver-memory 4g --executor-cores 3 --num-executors 60 \
../../${JAR} \ ../../${JAR} \
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment