Skip to content

A
atlas
Commit 0143486a by Shwetha GS
Options

ATLAS-823 Atlas should use external HBase and SOLR (tbeerbower via shwethags)

parent 2e02ae62
Showing with 105 additions and 61 deletions
distro/pom.xml
...@@ -28,15 +28,11 @@ ...@@ -28,15 +28,11 @@
<artifactId>atlas-distro</artifactId> <artifactId>atlas-distro</artifactId>
<description>Apache Atlas Distribution</description> <description>Apache Atlas Distribution</description>
<name>Apache Atlas Distribution</name> <name>Apache Atlas Distribution</name>
<properties> <properties>
<hbase.dir>${project.build.directory}/hbase</hbase.dir> <titan.storage.backend>berkeleyje</titan.storage.backend>
<hbase.tar>http://archive.apache.org/dist/hbase/${hbase.version}/hbase-${hbase.version}-bin.tar.gz</hbase.tar> <titan.index.backend>elasticsearch</titan.index.backend>
<hbase.folder>hbase-${hbase.version}</hbase.folder> <hbase.embedded>false</hbase.embedded>
<solr.embedded>false</solr.embedded>
<solr.dir>${project.build.directory}/solr</solr.dir>
<solr.tar>http://archive.apache.org/dist/lucene/solr/${solr.version}/solr-${solr.version}.tgz</solr.tar>
<solr.folder>solr-${solr.version}</solr.folder>
</properties> </properties>
<profiles> <profiles>
...@@ -76,10 +72,55 @@ ...@@ -76,10 +72,55 @@
<build> <build>
<plugins> <plugins>
<plugin> <plugin>
<artifactId>maven-assembly-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>single</goal>
</goals>
<phase>package</phase>
<configuration>
<descriptors>
<descriptor>src/main/assemblies/standalone-package.xml</descriptor>
<descriptor>src/main/assemblies/src-package.xml</descriptor>
</descriptors>
<finalName>apache-atlas-${project.version}</finalName>
<tarLongFileMode>gnu</tarLongFileMode>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
<!-- embedded_services profile to package and configure embedded hbase and solr with the distribution -->
<profile>
<id>embedded_services</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<properties>
<titan.storage.backend>hbase</titan.storage.backend>
<titan.index.backend>solr5</titan.index.backend>
<hbase.embedded>true</hbase.embedded>
<solr.embedded>true</solr.embedded>
<hbase.dir>${project.build.directory}/hbase</hbase.dir>
<hbase.tar>http://archive.apache.org/dist/hbase/${hbase.version}/hbase-${hbase.version}-bin.tar.gz</hbase.tar>
<hbase.folder>hbase-${hbase.version}</hbase.folder>
<solr.dir>${project.build.directory}/solr</solr.dir>
<solr.tar>http://archive.apache.org/dist/lucene/solr/${solr.version}/solr-${solr.version}.tgz</solr.tar>
<solr.folder>solr-${solr.version}</solr.folder>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId> <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-antrun-plugin</artifactId> <artifactId>maven-antrun-plugin</artifactId>
<version>1.7</version> <version>1.7</version>
<executions> <executions>
<!-- package hbase -->
<execution> <execution>
<id>hbase</id> <id>hbase</id>
<phase>generate-resources</phase> <phase>generate-resources</phase>
...@@ -109,6 +150,7 @@ ...@@ -109,6 +150,7 @@
</target> </target>
</configuration> </configuration>
</execution> </execution>
<!-- package solr -->
<execution> <execution>
<id>solr</id> <id>solr</id>
<phase>generate-resources</phase> <phase>generate-resources</phase>
...@@ -140,32 +182,13 @@ ...@@ -140,32 +182,13 @@
</execution> </execution>
</executions> </executions>
</plugin> </plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>single</goal>
</goals>
<phase>package</phase>
<configuration>
<descriptors>
<descriptor>src/main/assemblies/standalone-package.xml</descriptor>
<descriptor>src/main/assemblies/src-package.xml</descriptor>
</descriptors>
<finalName>apache-atlas-${project.version}</finalName>
<tarLongFileMode>gnu</tarLongFileMode>
</configuration>
</execution>
</executions>
</plugin>
</plugins> </plugins>
</build> </build>
</profile> </profile>
</profiles> </profiles>
<build> <build>
<outputDirectory>target/bin</outputDirectory> <outputDirectory>target</outputDirectory>
<resources> <resources>
<resource> <resource>
<directory>src/bin</directory> <directory>src/bin</directory>
...@@ -173,6 +196,12 @@ ...@@ -173,6 +196,12 @@
<includes> <includes>
<include>**/*.py</include> <include>**/*.py</include>
</includes> </includes>
<targetPath>bin</targetPath>
</resource>
<resource>
<directory>src/conf</directory>
<filtering>true</filtering>
<targetPath>conf</targetPath>
</resource> </resource>
</resources> </resources>
<plugins> <plugins>
......
distro/src/bin/atlas_config.py
...@@ -388,7 +388,7 @@ def is_hbase(confdir): ...@@ -388,7 +388,7 @@ def is_hbase(confdir):
return grep(confdir, HBASE_STORAGE_CONF_ENTRY) is not None return grep(confdir, HBASE_STORAGE_CONF_ENTRY) is not None
def is_hbase_local(confdir): def is_hbase_local(confdir):
if os.environ.get(MANAGE_LOCAL_HBASE, "True").lower() == 'false': if os.environ.get(MANAGE_LOCAL_HBASE, "False").lower() == 'false':
return False return False
confdir = os.path.join(confdir, CONF_FILE) confdir = os.path.join(confdir, CONF_FILE)
...@@ -419,7 +419,7 @@ def is_solr(confdir): ...@@ -419,7 +419,7 @@ def is_solr(confdir):
return grep(confdir, SOLR_INDEX_CONF_ENTRY) is not None return grep(confdir, SOLR_INDEX_CONF_ENTRY) is not None
def is_solr_local(confdir): def is_solr_local(confdir):
if os.environ.get(MANAGE_LOCAL_SOLR, "True").lower() == 'false': if os.environ.get(MANAGE_LOCAL_SOLR, "False").lower() == 'false':
return False return False
confdir = os.path.join(confdir, CONF_FILE) confdir = os.path.join(confdir, CONF_FILE)
......
distro/src/bin/atlas_start.py
...@@ -118,8 +118,6 @@ def main(): ...@@ -118,8 +118,6 @@ def main():
mc.run_solr(mc.solrBinDir(atlas_home), "start", mc.get_solr_zk_url(confdir), mc.solrPort(), logdir) mc.run_solr(mc.solrBinDir(atlas_home), "start", mc.get_solr_zk_url(confdir), mc.solrPort(), logdir)
print "solr started." print "solr started."
#solr indexes
if mc.is_solr(confdir):
print "setting up solr collections..." print "setting up solr collections..."
mc.create_solr_collection(mc.solrBinDir(atlas_home), mc.solrConfDir(atlas_home), "vertex_index", logdir) mc.create_solr_collection(mc.solrBinDir(atlas_home), mc.solrConfDir(atlas_home), "vertex_index", logdir)
mc.create_solr_collection(mc.solrBinDir(atlas_home), mc.solrConfDir(atlas_home), "edge_index", logdir) mc.create_solr_collection(mc.solrBinDir(atlas_home), mc.solrConfDir(atlas_home), "edge_index", logdir)
......
distro/src/conf/atlas-application.properties
...@@ -18,11 +18,11 @@ ...@@ -18,11 +18,11 @@
######### Graph Database Configs ######### ######### Graph Database Configs #########
# Graph Storage # Graph Storage
#atlas.graph.storage.backend=berkeleyje atlas.graph.storage.backend=${titan.storage.backend}
#atlas.graph.storage.directory=${sys:atlas.home}/data/berkley atlas.graph.storage.directory=${sys:atlas.home}/data/berkley
#Hbase as stoarge backend #Hbase as storage backend
atlas.graph.storage.backend=hbase #atlas.graph.storage.backend=hbase
#For standalone mode , specify localhost #For standalone mode , specify localhost
#for distributed mode, specify zookeeper quorum here - For more information refer http://s3.thinkaurelius.com/docs/titan/current/hbase.html#_remote_server_mode_2 #for distributed mode, specify zookeeper quorum here - For more information refer http://s3.thinkaurelius.com/docs/titan/current/hbase.html#_remote_server_mode_2
atlas.graph.storage.hostname=localhost atlas.graph.storage.hostname=localhost
...@@ -30,7 +30,7 @@ atlas.graph.storage.hbase.regions-per-server=1 ...@@ -30,7 +30,7 @@ atlas.graph.storage.hbase.regions-per-server=1
atlas.graph.storage.lock.wait-time=10000 atlas.graph.storage.lock.wait-time=10000
#Solr #Solr
atlas.graph.index.search.backend=solr5 atlas.graph.index.search.backend=${titan.index.backend}
# Solr cloud mode properties # Solr cloud mode properties
atlas.graph.index.search.solr.mode=cloud atlas.graph.index.search.solr.mode=cloud
...@@ -42,11 +42,10 @@ atlas.graph.index.search.solr.zookeeper-url=localhost:2181 ...@@ -42,11 +42,10 @@ atlas.graph.index.search.solr.zookeeper-url=localhost:2181
# Graph Search Index # Graph Search Index
#ElasticSearch #ElasticSearch
#atlas.graph.index.search.backend=elasticsearch atlas.graph.index.search.directory=${sys:atlas.home}/data/es
#atlas.graph.index.search.directory=${sys:atlas.home}/data/es atlas.graph.index.search.elasticsearch.client-only=false
#atlas.graph.index.search.elasticsearch.client-only=false atlas.graph.index.search.elasticsearch.local-mode=true
#atlas.graph.index.search.elasticsearch.local-mode=true atlas.graph.index.search.elasticsearch.create.sleep=2000
#atlas.graph.index.search.elasticsearch.create.sleep=2000
######### Notification Configs ######### ######### Notification Configs #########
......
distro/src/conf/atlas-env.sh
#!/usr/bin/env bash
# #
# Licensed to the Apache Software Foundation (ASF) under one # Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file # or more contributor license agreements. See the NOTICE file
...@@ -55,3 +56,6 @@ ...@@ -55,3 +56,6 @@
# Where do you want to expand the war file. By Default it is in /server/webapp dir under the base install dir. # Where do you want to expand the war file. By Default it is in /server/webapp dir under the base install dir.
#export ATLAS_EXPANDED_WEBAPP_DIR= #export ATLAS_EXPANDED_WEBAPP_DIR=
export MANAGE_LOCAL_HBASE=${hbase.embedded}
export MANAGE_LOCAL_SOLR=${solr.embedded}
distro/src/main/assemblies/standalone-package.xml
...@@ -28,7 +28,7 @@ ...@@ -28,7 +28,7 @@
<baseDirectory>apache-atlas-${project.version}</baseDirectory> <baseDirectory>apache-atlas-${project.version}</baseDirectory>
<fileSets> <fileSets>
<fileSet> <fileSet>
<directory>src/conf/</directory> <directory>target/conf/</directory>
<outputDirectory>conf</outputDirectory> <outputDirectory>conf</outputDirectory>
</fileSet> </fileSet>
......
docs/src/site/twiki/Configuration.twiki
...@@ -53,11 +53,11 @@ Without Ranger, HBase shell can be used to set the permissions. ...@@ -53,11 +53,11 @@ Without Ranger, HBase shell can be used to set the permissions.
echo "grant 'atlas', 'RWXCA', 'titan'" | hbase shell echo "grant 'atlas', 'RWXCA', 'titan'" | hbase shell
</verbatim> </verbatim>
Note that HBase is included in the distribution so that a standalone instance of HBase can be started as the default Note that if the embedded_services profile is used then HBase is included in the distribution so that a standalone
storage backend for the graph repository. By default, the standalone HBase instance will be started and stopped along instance of HBase can be started as the default storage backend for the graph repository. Using the embedded_services
with the Atlas server when the configuraton 'atlas.graph.storage.backend' is set to 'hbase' and profile will configure Atlas so that HBase instance will be started and stopped along with the Atlas server by default.
'atlas.graph.storage.hostname' is set to 'localhost'. Set the MANAGE_LOCAL_HBASE environment variable to 'False' to To use the embedded_services profile please see "Building Atlas" in the [[InstallationSteps][Installation Steps]]
override this behavior. section.
---+++ Graph Search Index ---+++ Graph Search Index
This section sets up the graph db - titan - to use an search indexing system. The example This section sets up the graph db - titan - to use an search indexing system. The example
...@@ -80,11 +80,11 @@ Please note that Solr installation in Cloud mode is a prerequisite before config ...@@ -80,11 +80,11 @@ Please note that Solr installation in Cloud mode is a prerequisite before config
atlas.graph.index.search.solr.zookeeper-url=<the ZK quorum setup for solr as comma separated value> eg: 10.1.6.4:2181,10.1.6.5:2181 atlas.graph.index.search.solr.zookeeper-url=<the ZK quorum setup for solr as comma separated value> eg: 10.1.6.4:2181,10.1.6.5:2181
</verbatim> </verbatim>
Also note that Solr is included in the distribution so that a standalone instance of Solr can be started as the default Also note that if the embedded_services profile is used then Solr is included in the distribution so that a standalone
search indexing backend. By default, the standalone Solr instance will be started and stopped along instance of Solr can be started as the default search indexing backend. Using the embedded_services profile will
with the Atlas server when the configuraton 'atlas.graph.index.search.backend' is set to 'solr5' and configure Atlas so that the standalone Solr instance will be started and stopped along with the Atlas server by default.
'atlas.graph.index.search.solr.zookeeper-url' is set to 'localhost'. Set the MANAGE_LOCAL_SOLR environment variable to To use the embedded_services profile please see "Building Atlas" in the [[InstallationSteps][Installation Steps]]
'False' to override this behavior. section.
---+++ Choosing between Persistence and Indexing Backends ---+++ Choosing between Persistence and Indexing Backends
......
docs/src/site/twiki/InstallationSteps.twiki
...@@ -18,6 +18,17 @@ mvn clean package -Pdist ...@@ -18,6 +18,17 @@ mvn clean package -Pdist
</verbatim> </verbatim>
To build a distribution that packages HBase and Solr, build with the embedded_services profile.
<verbatim>
mvn clean package -Pdist,embedded_services
</verbatim>
Using the embedded_services profile will configure Atlas so that an HBase instance and a Solr instance will be started
and stopped along with the Atlas server by default.
Tar can be found in atlas/distro/target/apache-atlas-${project.version}-bin.tar.gz Tar can be found in atlas/distro/target/apache-atlas-${project.version}-bin.tar.gz
Tar is structured as follows Tar is structured as follows
...@@ -64,15 +75,18 @@ Tar is structured as follows ...@@ -64,15 +75,18 @@ Tar is structured as follows
</verbatim> </verbatim>
Note that HBase is included in the distribution so that a standalone instance of HBase can be started as the default Note that if the embedded_services profile is specified for the build then HBase and Solr are included in the
storage backend for the graph repository. During Atlas installation the conf/hbase/hbase-site.xml.template gets distribution.
expanded and moved to hbase/conf/hbase-site.xml for the initial standalone HBase configuration. To configure ATLAS
In this case, a standalone instance of HBase can be started as the default storage backend for the graph repository.
During Atlas installation the conf/hbase/hbase-site.xml.template gets expanded and moved to hbase/conf/hbase-site.xml
for the initial standalone HBase configuration. To configure ATLAS
graph persistence for a different HBase instance, please see "Graph persistence engine - HBase" in the graph persistence for a different HBase instance, please see "Graph persistence engine - HBase" in the
[[Configuration][Configuration]] section. [[Configuration][Configuration]] section.
Also note that Solr is included in the distribution so that a standalone instance of Solr can be started as the default Also, a standalone instance of Solr can be started as the default search indexing backend. To configure ATLAS search
search indexing backend. To configure ATLAS search indexing for a different Solr instance, please see indexing for a different Solr instance please see "Graph Search Index - Solr" in the
"Graph Search Index - Solr" in the [[Configuration][Configuration]] section. [[Configuration][Configuration]] section.
---+++ Installing & Running Atlas ---+++ Installing & Running Atlas
...@@ -310,4 +324,4 @@ ensure the setup can run and delete the Zookeeper node at =/apache_atlas/setup_i ...@@ -310,4 +324,4 @@ ensure the setup can run and delete the Zookeeper node at =/apache_atlas/setup_i
run setup again. run setup again.
If the setup failed due to HBase Titan schema setup errors, it may be necessary to repair the HBase schema. If no If the setup failed due to HBase Titan schema setup errors, it may be necessary to repair the HBase schema. If no
data has been stored, one can also disable and drop the 'titan' schema in HBase to let setup run again. data has been stored, one can also disable and drop the 'titan' schema in HBase to let setup run again.
\ No newline at end of file
release-log.txt
...@@ -22,6 +22,7 @@ ATLAS-409 Atlas will not import avro tables with schema read from a file (dosset ...@@ -22,6 +22,7 @@ ATLAS-409 Atlas will not import avro tables with schema read from a file (dosset
ATLAS-379 Create sqoop and falcon metadata addons (venkatnrangan,bvellanki,sowmyaramesh via shwethags) ATLAS-379 Create sqoop and falcon metadata addons (venkatnrangan,bvellanki,sowmyaramesh via shwethags)
ALL CHANGES: ALL CHANGES:
ATLAS-823 Atlas should use external HBase and SOLR (tbeerbower via shwethags)
ATLAS-752 Column renames should retain traits/tags (svimal2106 via shwethags) ATLAS-752 Column renames should retain traits/tags (svimal2106 via shwethags)
ATLAS-821 Atlas UI - Add arrow to navigate to child term (kevalbhatt18 via yhemanth) ATLAS-821 Atlas UI - Add arrow to navigate to child term (kevalbhatt18 via yhemanth)
ATLAS-812 Atlas UI - Associate Terms with Assets (kevalbhatt18 via yhemanth) ATLAS-812 Atlas UI - Associate Terms with Assets (kevalbhatt18 via yhemanth)
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment