name: Incremental Export
route: /IncrementalExport
menu: Documentation
submenu: Import/Export

import themen from 'theme/styles/styled-colors'; import * as theme from 'react-syntax-highlighter/dist/esm/styles/hljs'; import SyntaxHighlighter from 'react-syntax-highlighter';

Incremental Export

Background

Incremental export allows for export of entities after a specified timestamp. This allows for synchronization capability to be built as it makes payloads lighter.

Export Options

New fetchType added to indicate incremental export. This option can be used with any matchType. When fetchType is incremental, it is necessary to specify the changeMarker option for incremental export to function, else full export will be performed.

{{ "itemsToExport": [ { "typeName": "hive_db", "uniqueAttributes": { "qualifiedName": "stocks@cl1" } } ], "options": { "fetchType": "incremental", "changeMarker": 10000 } }}

Getting Change Marker

The very first call to export with fetchType set to incremental should be made with changeMarker set to 0. This will perform a full export. The AtlasExportResult will have the changeMarker set to a value. This is the value that should be used for subsequent call to Export.

Skip Lineage Option

Export can be performed by skipping lineage information. This avoids all lineage information getting into the exported file.

Benefit of Incremental Export

The real benefit of incremental export comes when export is done with skipLineage option set to true. This greatly improves performance when fetching entities that have changed since the last export.