Skip to content

A
atlas
Commit 0143486a by Shwetha GS
Options

ATLAS-823 Atlas should use external HBase and SOLR (tbeerbower via shwethags)

parent 2e02ae62
Showing with 105 additions and 61 deletions
distro/pom.xml
......@@ -28,15 +28,11 @@
<artifactId>atlas-distro</artifactId>
<description>Apache Atlas Distribution</description>
<name>Apache Atlas Distribution</name>
<properties>
<hbase.dir>${project.build.directory}/hbase</hbase.dir>
<hbase.tar>http://archive.apache.org/dist/hbase/${hbase.version}/hbase-${hbase.version}-bin.tar.gz</hbase.tar>
<hbase.folder>hbase-${hbase.version}</hbase.folder>
<solr.dir>${project.build.directory}/solr</solr.dir>
<solr.tar>http://archive.apache.org/dist/lucene/solr/${solr.version}/solr-${solr.version}.tgz</solr.tar>
<solr.folder>solr-${solr.version}</solr.folder>
<titan.storage.backend>berkeleyje</titan.storage.backend>
<titan.index.backend>elasticsearch</titan.index.backend>
<hbase.embedded>false</hbase.embedded>
<solr.embedded>false</solr.embedded>
</properties>
<profiles>
......@@ -76,10 +72,55 @@
<build>
<plugins>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>single</goal>
</goals>
<phase>package</phase>
<configuration>
<descriptors>
<descriptor>src/main/assemblies/standalone-package.xml</descriptor>
<descriptor>src/main/assemblies/src-package.xml</descriptor>
</descriptors>
<finalName>apache-atlas-${project.version}</finalName>
<tarLongFileMode>gnu</tarLongFileMode>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
<!-- embedded_services profile to package and configure embedded hbase and solr with the distribution -->
<profile>
<id>embedded_services</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<properties>
<titan.storage.backend>hbase</titan.storage.backend>
<titan.index.backend>solr5</titan.index.backend>
<hbase.embedded>true</hbase.embedded>
<solr.embedded>true</solr.embedded>
<hbase.dir>${project.build.directory}/hbase</hbase.dir>
<hbase.tar>http://archive.apache.org/dist/hbase/${hbase.version}/hbase-${hbase.version}-bin.tar.gz</hbase.tar>
<hbase.folder>hbase-${hbase.version}</hbase.folder>
<solr.dir>${project.build.directory}/solr</solr.dir>
<solr.tar>http://archive.apache.org/dist/lucene/solr/${solr.version}/solr-${solr.version}.tgz</solr.tar>
<solr.folder>solr-${solr.version}</solr.folder>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-antrun-plugin</artifactId>
<version>1.7</version>
<executions>
<!-- package hbase -->
<execution>
<id>hbase</id>
<phase>generate-resources</phase>
......@@ -109,6 +150,7 @@
</target>
</configuration>
</execution>
<!-- package solr -->
<execution>
<id>solr</id>
<phase>generate-resources</phase>
......@@ -140,32 +182,13 @@
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>single</goal>
</goals>
<phase>package</phase>
<configuration>
<descriptors>
<descriptor>src/main/assemblies/standalone-package.xml</descriptor>
<descriptor>src/main/assemblies/src-package.xml</descriptor>
</descriptors>
<finalName>apache-atlas-${project.version}</finalName>
<tarLongFileMode>gnu</tarLongFileMode>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
<build>
<outputDirectory>target/bin</outputDirectory>
<outputDirectory>target</outputDirectory>
<resources>
<resource>
<directory>src/bin</directory>
......@@ -173,6 +196,12 @@
<includes>
<include>**/*.py</include>
</includes>
<targetPath>bin</targetPath>
</resource>
<resource>
<directory>src/conf</directory>
<filtering>true</filtering>
<targetPath>conf</targetPath>
</resource>
</resources>
<plugins>
......
distro/src/bin/atlas_config.py
......@@ -388,7 +388,7 @@ def is_hbase(confdir):
return grep(confdir, HBASE_STORAGE_CONF_ENTRY) is not None
def is_hbase_local(confdir):
if os.environ.get(MANAGE_LOCAL_HBASE, "True").lower() == 'false':
if os.environ.get(MANAGE_LOCAL_HBASE, "False").lower() == 'false':
return False
confdir = os.path.join(confdir, CONF_FILE)
......@@ -419,7 +419,7 @@ def is_solr(confdir):
return grep(confdir, SOLR_INDEX_CONF_ENTRY) is not None
def is_solr_local(confdir):
if os.environ.get(MANAGE_LOCAL_SOLR, "True").lower() == 'false':
if os.environ.get(MANAGE_LOCAL_SOLR, "False").lower() == 'false':
return False
confdir = os.path.join(confdir, CONF_FILE)
......
distro/src/bin/atlas_start.py
......@@ -118,8 +118,6 @@ def main():
mc.run_solr(mc.solrBinDir(atlas_home), "start", mc.get_solr_zk_url(confdir), mc.solrPort(), logdir)
print "solr started."
#solr indexes
if mc.is_solr(confdir):
print "setting up solr collections..."
mc.create_solr_collection(mc.solrBinDir(atlas_home), mc.solrConfDir(atlas_home), "vertex_index", logdir)
mc.create_solr_collection(mc.solrBinDir(atlas_home), mc.solrConfDir(atlas_home), "edge_index", logdir)
......
distro/src/conf/atlas-application.properties
......@@ -18,11 +18,11 @@
######### Graph Database Configs #########
# Graph Storage
#atlas.graph.storage.backend=berkeleyje
#atlas.graph.storage.directory=${sys:atlas.home}/data/berkley
atlas.graph.storage.backend=${titan.storage.backend}
atlas.graph.storage.directory=${sys:atlas.home}/data/berkley
#Hbase as stoarge backend
atlas.graph.storage.backend=hbase
#Hbase as storage backend
#atlas.graph.storage.backend=hbase
#For standalone mode , specify localhost
#for distributed mode, specify zookeeper quorum here - For more information refer http://s3.thinkaurelius.com/docs/titan/current/hbase.html#_remote_server_mode_2
atlas.graph.storage.hostname=localhost
......@@ -30,7 +30,7 @@ atlas.graph.storage.hbase.regions-per-server=1
atlas.graph.storage.lock.wait-time=10000
#Solr
atlas.graph.index.search.backend=solr5
atlas.graph.index.search.backend=${titan.index.backend}
# Solr cloud mode properties
atlas.graph.index.search.solr.mode=cloud
......@@ -42,11 +42,10 @@ atlas.graph.index.search.solr.zookeeper-url=localhost:2181
# Graph Search Index
#ElasticSearch
#atlas.graph.index.search.backend=elasticsearch
#atlas.graph.index.search.directory=${sys:atlas.home}/data/es
#atlas.graph.index.search.elasticsearch.client-only=false
#atlas.graph.index.search.elasticsearch.local-mode=true
#atlas.graph.index.search.elasticsearch.create.sleep=2000
atlas.graph.index.search.directory=${sys:atlas.home}/data/es
atlas.graph.index.search.elasticsearch.client-only=false
atlas.graph.index.search.elasticsearch.local-mode=true
atlas.graph.index.search.elasticsearch.create.sleep=2000
######### Notification Configs #########
......
distro/src/conf/atlas-env.sh
#!/usr/bin/env bash
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
......@@ -55,3 +56,6 @@
# Where do you want to expand the war file. By Default it is in /server/webapp dir under the base install dir.
#export ATLAS_EXPANDED_WEBAPP_DIR=
export MANAGE_LOCAL_HBASE=${hbase.embedded}
export MANAGE_LOCAL_SOLR=${solr.embedded}
distro/src/main/assemblies/standalone-package.xml
......@@ -28,7 +28,7 @@
<baseDirectory>apache-atlas-${project.version}</baseDirectory>
<fileSets>
<fileSet>
<directory>src/conf/</directory>
<directory>target/conf/</directory>
<outputDirectory>conf</outputDirectory>
</fileSet>
......
docs/src/site/twiki/Configuration.twiki
......@@ -53,11 +53,11 @@ Without Ranger, HBase shell can be used to set the permissions.
echo "grant 'atlas', 'RWXCA', 'titan'" | hbase shell
</verbatim>
Note that HBase is included in the distribution so that a standalone instance of HBase can be started as the default
storage backend for the graph repository. By default, the standalone HBase instance will be started and stopped along
with the Atlas server when the configuraton 'atlas.graph.storage.backend' is set to 'hbase' and
'atlas.graph.storage.hostname' is set to 'localhost'. Set the MANAGE_LOCAL_HBASE environment variable to 'False' to
override this behavior.
Note that if the embedded_services profile is used then HBase is included in the distribution so that a standalone
instance of HBase can be started as the default storage backend for the graph repository. Using the embedded_services
profile will configure Atlas so that HBase instance will be started and stopped along with the Atlas server by default.
To use the embedded_services profile please see "Building Atlas" in the [[InstallationSteps][Installation Steps]]
section.
---+++ Graph Search Index
This section sets up the graph db - titan - to use an search indexing system. The example
......@@ -80,11 +80,11 @@ Please note that Solr installation in Cloud mode is a prerequisite before config
atlas.graph.index.search.solr.zookeeper-url=<the ZK quorum setup for solr as comma separated value> eg: 10.1.6.4:2181,10.1.6.5:2181
</verbatim>
Also note that Solr is included in the distribution so that a standalone instance of Solr can be started as the default
search indexing backend. By default, the standalone Solr instance will be started and stopped along
with the Atlas server when the configuraton 'atlas.graph.index.search.backend' is set to 'solr5' and
'atlas.graph.index.search.solr.zookeeper-url' is set to 'localhost'. Set the MANAGE_LOCAL_SOLR environment variable to
'False' to override this behavior.
Also note that if the embedded_services profile is used then Solr is included in the distribution so that a standalone
instance of Solr can be started as the default search indexing backend. Using the embedded_services profile will
configure Atlas so that the standalone Solr instance will be started and stopped along with the Atlas server by default.
To use the embedded_services profile please see "Building Atlas" in the [[InstallationSteps][Installation Steps]]
section.
---+++ Choosing between Persistence and Indexing Backends
......
docs/src/site/twiki/InstallationSteps.twiki
......@@ -18,6 +18,17 @@ mvn clean package -Pdist
</verbatim>
To build a distribution that packages HBase and Solr, build with the embedded_services profile.
<verbatim>
mvn clean package -Pdist,embedded_services
</verbatim>
Using the embedded_services profile will configure Atlas so that an HBase instance and a Solr instance will be started
and stopped along with the Atlas server by default.
Tar can be found in atlas/distro/target/apache-atlas-${project.version}-bin.tar.gz
Tar is structured as follows
......@@ -64,15 +75,18 @@ Tar is structured as follows
</verbatim>
Note that HBase is included in the distribution so that a standalone instance of HBase can be started as the default
storage backend for the graph repository. During Atlas installation the conf/hbase/hbase-site.xml.template gets
expanded and moved to hbase/conf/hbase-site.xml for the initial standalone HBase configuration. To configure ATLAS
Note that if the embedded_services profile is specified for the build then HBase and Solr are included in the
distribution.
In this case, a standalone instance of HBase can be started as the default storage backend for the graph repository.
During Atlas installation the conf/hbase/hbase-site.xml.template gets expanded and moved to hbase/conf/hbase-site.xml
for the initial standalone HBase configuration. To configure ATLAS
graph persistence for a different HBase instance, please see "Graph persistence engine - HBase" in the
[[Configuration][Configuration]] section.
Also note that Solr is included in the distribution so that a standalone instance of Solr can be started as the default
search indexing backend. To configure ATLAS search indexing for a different Solr instance, please see
"Graph Search Index - Solr" in the [[Configuration][Configuration]] section.
Also, a standalone instance of Solr can be started as the default search indexing backend. To configure ATLAS search
indexing for a different Solr instance please see "Graph Search Index - Solr" in the
[[Configuration][Configuration]] section.
---+++ Installing & Running Atlas
......@@ -310,4 +324,4 @@ ensure the setup can run and delete the Zookeeper node at =/apache_atlas/setup_i
run setup again.
If the setup failed due to HBase Titan schema setup errors, it may be necessary to repair the HBase schema. If no
data has been stored, one can also disable and drop the 'titan' schema in HBase to let setup run again.
\ No newline at end of file
data has been stored, one can also disable and drop the 'titan' schema in HBase to let setup run again.
release-log.txt
......@@ -22,6 +22,7 @@ ATLAS-409 Atlas will not import avro tables with schema read from a file (dosset
ATLAS-379 Create sqoop and falcon metadata addons (venkatnrangan,bvellanki,sowmyaramesh via shwethags)
ALL CHANGES:
ATLAS-823 Atlas should use external HBase and SOLR (tbeerbower via shwethags)
ATLAS-752 Column renames should retain traits/tags (svimal2106 via shwethags)
ATLAS-821 Atlas UI - Add arrow to navigate to child term (kevalbhatt18 via yhemanth)
ATLAS-812 Atlas UI - Associate Terms with Assets (kevalbhatt18 via yhemanth)
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment