Bridge-Kafka.twiki 1.45 KB
Newer Older
1
---+ Apache Atlas Hook for Apache Kafka
2 3

---++ Kafka Model
4
Kafka model includes the following types:
5 6 7
   * Entity types:
      * kafka_topic
         * super-types: !DataSet
8
         * attributes: qualifiedName, name, description, owner, topic, uri, partitionCount
9

10 11 12 13 14
Kafka entities are created and de-duped in Atlas using unique attribute qualifiedName, whose value should be formatted as detailed below.
Note that qualifiedName will have topic name in lower case.
<verbatim>
   topic.qualifiedName: <topic>@<clusterName>
</verbatim>
15 16 17


---++ Setup
18 19 20 21 22
      Binary files are present in apache-atlas-<release-version>-kafka-hook.tar.gz

      Copy apache-atlas-kafka-hook-<release-version>/hook/kafka folder to <atlas package>/hook/    directory

      Copy apache-atlas-kafka-hook-<release-version>/hook-bin folder to  <atlas package>/hook-bin directory
23 24

---++ Importing Kafka Metadata
25 26 27 28 29 30 31 32 33 34 35 36 37
Apache Atlas provides a command-line utility, import-kafka.sh, to import metadata of Apache Kafka topics into Apache Atlas.
This utility can be used to initialize Apache Atlas with topics present in Apache Kafka.
This utility supports importing metadata of a specific topic or all topics.

<verbatim>
Usage 1: <atlas package>/hook-bin/import-kafka.sh
Usage 2: <atlas package>/hook-bin/import-kafka.sh [-t <topic prefix> OR --topic <topic prefix>]
Usage 3: <atlas package>/hook-bin/import-kafka.sh [-f <filename>]
         File Format:
            topic1
            topic2
            topic3
</verbatim>