Commit c9924fdc by rmani Committed by Madhan Neethiraj

ATLAS-2603: [Docs] Add Atlas Kafka Bridge Documents

parent 620943ed
---+ Kafka Atlas Bridge
---++ Kafka Model
The default Kafka model includes the following types:
* Entity types:
* kafka_topic
* super-types: !DataSet
* attributes: name, owner, description, type, classifications, term, clustername, topic , partitionCount, qualifiedName
The entities are created and de-duped using unique qualified name. They provide namespace and can be used for querying as well:
* topic.qualifiedName - <topic>@<clusterName>
---++ Setup
binary files are present in apache-atlas-<release-vesion>-SNAPSHOT-kafka-hook.tar.gz
Copy apache-atlas-kafka-hook-<release-verion>-SNAPSHOT/hook/kafka folder to <atlas package>/hook/ directory
Copy apache-atlas-kafka-hook-<release-verion>-SNAPSHOT/hook-bin folder to <atlas package>/hook-bin/ directory
* Copy <atlas-conf>/atlas-application.properties to the Kafka conf directory.
---++ Importing Kafka Metadata
org.apache.atlas.Kafka.bridge.KafkaBridge imports the Kafka metadata into Atlas using the model defined above. import-kafka.sh command can be used to facilitate this.
<verbatim>
Usage 1: <atlas package>/hook-bin/import-kafka.sh
Usage 2: <atlas package>/hook-bin/import-kafka.sh [-n <namespace regex> OR --namespace <namespace regex >] [-t <table regex > OR --table <table regex>]
Usage 3: <atlas package>/hook-bin/import-kafka.sh [-f <filename>]
File Format:
topic1
topic2
topic3
</verbatim>
The logs are in <atlas package>/logs/import-kafka.log
Refer [[Configuration][Configuration]] for notification related configurations
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment