Skip to content
Projects
Groups
Snippets
Help
This project
Loading...
Sign in / Register
Toggle navigation
M
mobvista-dmp
Project
Overview
Details
Activity
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
王金锋
mobvista-dmp
Commits
d4fe6410
Commit
d4fe6410
authored
Dec 20, 2021
by
WangJinfeng
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
init id_mapping
parent
2b5304c6
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
5 additions
and
6 deletions
+5
-6
IDMappingGraphx.scala
.../mobvista/dmp/datasource/id_mapping/IDMappingGraphx.scala
+5
-6
No files found.
src/main/scala/mobvista/dmp/datasource/id_mapping/IDMappingGraphx.scala
View file @
d4fe6410
...
...
@@ -94,15 +94,14 @@ class IDMappingGraphx extends CommonSparkJob with Serializable {
processVertex
(
date
,
row
,
idSet
,
idMainSet
)
}).
flatMap
(
l
=>
l
)
val
maxGraph
=
vertex
.
filter
(
kv
=>
{
// 非主ID生成OneID
val
multiOneIDRDD
=
vertex
.
filter
(
kv
=>
{
!
idMainSet
.
contains
(
kv
.
_2
.
_3
)
}).
combineByKey
(
(
v
:
(
String
,
String
,
String
))
=>
Set
(
v
),
(
c
:
Set
[(
String
,
String
,
String
)],
v
:
(
String
,
String
,
String
))
=>
c
++
Seq
(
v
),
(
c1
:
Set
[(
String
,
String
,
String
)],
c2
:
Set
[(
String
,
String
,
String
)])
=>
c1
++
c2
)
val
multiOneIDRDD
=
maxGraph
.
map
(
rs
=>
{
).
map
(
rs
=>
{
platform
match
{
case
"ios"
=>
updateOneID
(
rs
,
Constant
.
iosMainIDSet
)
...
...
@@ -111,12 +110,12 @@ class IDMappingGraphx extends CommonSparkJob with Serializable {
}
}).
flatMap
(
l
=>
l
)
// 主ID生成OneID
val
singleOneIDRDD
=
vertex
.
filter
(
kv
=>
{
idMainSet
.
contains
(
kv
.
_2
.
_3
)
// kv._2.size == 1
}).
map
(
kv
=>
{
val
oneID
=
new
JSONObject
()
oneID
.
put
(
kv
.
_2
.
_1
,
kv
.
_2
.
_2
)
oneID
.
put
(
kv
.
_2
.
_1
,
MobvistaConstant
.
String2JSONObject
(
kv
.
_2
.
_2
)
)
(
kv
.
_1
,
oneID
.
toJSONString
,
kv
.
_2
.
_3
)
})
/*
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment