name: Kafka
route: /HookKafka
menu: Documentation
submenu: Hooks

import themen from 'theme/styles/styled-colors'; import * as theme from 'react-syntax-highlighter/dist/esm/styles/hljs'; import SyntaxHighlighter from 'react-syntax-highlighter'; import Img from 'theme/components/shared/Img'

Apache Atlas Hook for Apache Kafka

Kafka Model

Kafka model includes the following types:

• Entity types:
  • kafka_topic
    • super-types: DataSet
    • attributes: qualifiedName, name, description, owner, topic, uri, partitionCount

Kafka entities are created and de-duped in Atlas using unique attribute qualifiedName, whose value should be formatted as detailed below. Note that qualifiedName will have topic name in lower case. {topic.qualifiedName: <topic>@<clusterName>}

Setup

Binary files are present in apache-atlas--kafka-hook.tar.gz

Copy apache-atlas-kafka-hook-/hook/kafka folder to /hook/ directory

Copy apache-atlas-kafka-hook-/hook-bin folder to /hook-bin directory

Importing Kafka Metadata

Apache Atlas provides a command-line utility, import-kafka.sh, to import metadata of Apache Kafka topics into Apache Atlas. This utility can be used to initialize Apache Atlas with topics present in Apache Kafka. This utility supports importing metadata of a specific topic or all topics.

{sage 1: <atlas package>/hook-bin/import-kafka.sh Usage 2: <atlas package>/hook-bin/import-kafka.sh [-t <topic prefix> OR --topic <topic prefix>] Usage 3: <atlas package>/hook-bin/import-kafka.sh [-f <filename>] File Format: topic1 topic2 topic3}