ImportEntityTransforms.md 5.5 KB
Newer Older
1 2 3 4 5 6 7 8 9 10 11
---
name: Import Entity Transforms
route: /ImportEntityTransforms
menu: Documentation
submenu: Import/Export
---

import  themen  from 'theme/styles/styled-colors';
import  * as theme  from 'react-syntax-highlighter/dist/esm/styles/hljs';
import SyntaxHighlighter from 'react-syntax-highlighter';

12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43
# (New) Entity Transforms Framework

#### Background

During Import Process, entity transforms are required to make changes to the entity before it gets committed to the database. These modifications are necessary to make the entity conform to the environment it is going to reside. The Import Process provided a mechanism to do that.

#### Transformation Framework

A transformation framework allows a mechanism to selectively transform an entity or specific attributes of that entity.

To achieve this, the framework, provides:

* Way to set a condition that needs to be satisfied for a transformation to be applied.
* Action to be taken on the entity once the condition is met.

The existing transformation frameworks allowed this to happen.

#### Reason for New Transformation Framework

While the existing framework provided the basic benefits of transformation framework, it did not have support for some of the commonly used Atlas types. Which meant that users of this framework would have to meticulously define transformations for every type they are working with. This can be tedious and potentially error prone.
The new framework addresses this problem by providing built-in transformations for some of the commonly used types. It can also be extended to accommodate new types.

#### Approach

The approach used by the new transformation framework creates a transformation by:
* Specifying a condition.
* Specifying action(s) to be taken if condition is met.

##### Conditions

Following are built-in conditions.

44
|**Condition Types**                          | **Description**    |
45 46 47 48 49
-----------------------------------------|-----------------|
ENTITY_ALL                | Any/every entity               |
ENTITY_TOP_LEVEL          | Entity that is the top-level entity. This is also the entity present specified in _AtlasExportRequest_.|
EQUALS                    | Entity attribute equals to the one specified in the condition. |
EQUALS_IGNORE_CASE        | Entity attribute equals to the one specified in the condition ignoring case. |
50
STARTS_WITH               | Entity attribute starts with. |
51 52 53 54 55 56
STARTS_WITH_IGNORE_CASE   | Entity attribute starts with ignoring case. |
HAS_VALUE                 | Entity attribute has value. |


##### Actions

57
|**Action Type**        | *Description**                                 |
58 59 60 61 62 63 64 65 66
-------------------|----------------------------------------------|
ADD_CLASSIFICATION | Add classifiction                            |
REPLACE_PREFIX     | Replace value starting with another value.   |
TO_LOWER           | Convert value of an attribute to lower case. |
SET                | Set the value of an attribute                |
CLEAR              | Clear value of an attribute                  |

#### Built-in Transforms

67
##### Add Classification
68 69

During import, hive_db entity whose _qualifiedName_ is _stocks@cl1_ will get the classification _clSrcImported_.
70 71 72

<SyntaxHighlighter wrapLines={true} language="json" style={theme.dark}>
{`{
73 74 75 76 77 78
    "conditions": {
        "hive_db.qualifiedName": "stocks@cl1"
    },
    "action": {
        "__entity": "ADD_CLASSIFICATION: clSrcImported"
    }
79 80
}`}
</SyntaxHighlighter>
81 82 83

Every imported entity will get the classification by simply changing the condition. The __entity is special condition which matches entity.

84 85 86

<SyntaxHighlighter wrapLines={true} language="json" style={theme.dark}>
{`{
87 88 89 90 91 92
    "conditions": {
        "__entity": ""
    },
    "action": {
        "__entity": "ADD_CLASSIFICATION: clSrcImported"
    }
93 94
}`}
</SyntaxHighlighter>
95 96 97

To add classification to only the top-level entity (entity that is used as starting point for an export), use:

98 99
<SyntaxHighlighter wrapLines={true} language="json" style={theme.dark}>
{`{
100 101 102 103 104 105
    "conditions": {
        "__entity": "topLevel:"
    },
    "action": {
        "__entity": "ADD_CLASSIFICATION: clSrcImported"
    }
106 107 108
}`}
</SyntaxHighlighter>

109
##### Replace Prefix
110 111 112 113

This action works on string values. The first parameter is the prefix that is searched for a match, once matched, it is replaced with the provided replacement string.

The sample below searches for _/aa/bb/_, once found replaces it with _/xx/yy/_.
114 115 116

<SyntaxHighlighter wrapLines={true} language="json" style={theme.dark}>
{`{
117 118 119 120 121 122
    "conditions": {
        "hdfs_path.clusterName": "EQUALS: CL1"
    },
    "action": {
        "hdfs_path.path": "REPLACE_PREFIX: = :/aa/bb/=/xx/yy/"
    }
123 124
}`}
</SyntaxHighlighter>
125

126
##### To Lower
127 128 129

Entity whose hdfs_path.clusterName is CL1 will get its path attribute converted to lower case.

130
<SyntaxHighlighter wrapLines={true} language="json" style={theme.dark}>
131
{`{
132 133 134 135 136 137
    "conditions": {
        "hdfs_path.clusterName": "EQUALS: CL1"
    },
    "action": {
        "hdfs_path.path": "TO_LOWER:"
    }
138 139
}`}
</SyntaxHighlighter>
140

141
##### Clear
142 143 144

Entity whose hdfs_path.clusterName has value set, will get its _replicatedTo_ attribute value cleared.

145
<SyntaxHighlighter wrapLines={true} language="json" style={theme.dark}>
146
{`{
147 148 149 150 151 152
    "conditions": {
        "hdfs_path.clusterName": "HAS_VALUE:"
    },
    "action": {
        "hdfs_path.replicatedTo": "CLEAR:"
    }
153 154
}`}
</SyntaxHighlighter>
155 156 157 158


#### Additional Examples

159
Please look at [these tests](https://github.com/apache/atlas/blob/master/intg/src/test/java/org/apache/atlas/entitytransform/TransformationHandlerTest.java) for examples using Java classes.