Skip to content
Projects
Groups
Snippets
Help
This project
Loading...
Sign in / Register
Toggle navigation
M
mobvista-dmp
Project
Overview
Details
Activity
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
王金锋
mobvista-dmp
Commits
644e85a1
Commit
644e85a1
authored
3 years ago
by
WangJinfeng
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
fix rtdmp
parent
7890533e
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
7 additions
and
1 deletion
+7
-1
rtdmp.sh
azkaban/rtdmp/rtdmp.sh
+7
-1
No files found.
azkaban/rtdmp/rtdmp.sh
View file @
644e85a1
...
...
@@ -10,6 +10,12 @@ date_path=$(date +%Y/%m/%d/%H -d "-1 hour $today")
INPUT
=
"s3://mob-emr-test/dataplatform/rtdmp_pre/
${
date_path
}
"
old_date_path
=
$(
date
+%Y/%m/%d/%H
-d
"-1 hour
$today
"
)
OLD_MERGE_INPUT
=
"s3://mob-emr-test/dataplatform/DataWareHouse/data/dwh/audience_merge/
${
old_date_path
}
"
check_await
${
MERGE_INPUT
}
/_SUCCESS
OUTPUT
=
"s3://mob-emr-test/dataplatform/DataWareHouse/data/dwh/audience_merge/
${
date_path
}
"
spark-submit
--class
mobvista.dmp.datasource.rtdmp.RTDmpMain
\
...
...
@@ -19,7 +25,7 @@ spark-submit --class mobvista.dmp.datasource.rtdmp.RTDmpMain \
--conf
spark.kryoserializer.buffer.max
=
512m
\
--conf
spark.kryoserializer.buffer
=
64m
\
--master
yarn
--deploy-mode
cluster
\
--executor-memory
18g
--driver-memory
4
g
--executor-cores
5
--num-executors
40
\
--executor-memory
18g
--driver-memory
6
g
--executor-cores
5
--num-executors
40
\
.././DMP.jar
\
-datetime
${
date_time
}
-input
${
INPUT
}
-output
${
OUTPUT
}
-coalesce
200
...
...
This diff is collapsed.
Click to expand it.
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment