* @param limit number of rows to be returned in the result, used for pagination. maxlimit > limit > 0. -1 maps to atlas.search.defaultlimit property value
* @param offset offset to the results returned, used for pagination. offset >= 0. -1 maps to offset 0
@@ -1006,39 +1011,20 @@ public class AtlasClient {
...
@@ -1006,39 +1011,20 @@ public class AtlasClient {
/**
/**
* Search given query DSL
* Search given query DSL
* @param query DSL query
* @param query DSL query
* @param limit number of rows to be returned in the result, used for pagination. maxlimit > limit > 0. -1 maps to atlas.search.defaultlimit property value
* @param offset offset to the results returned, used for pagination. offset >= 0. -1 maps to offset 0
@@ -1052,15 +1038,20 @@ public class AtlasClient {
...
@@ -1052,15 +1038,20 @@ public class AtlasClient {
/**
/**
* Search given full text search
* Search given full text search
* @param query Query
* @param query Query
* @param limit number of rows to be returned in the result, used for pagination. maxlimit > limit > 0. -1 maps to atlas.search.defaultlimit property value
* @param offset offset to the results returned, used for pagination. offset >= 0. -1 maps to offset 0
* NOTE: Pagination is not implemented currently for full text search, so limit and offset are not used
@@ -134,6 +134,18 @@ atlas.lineage.hive.table.schema.query=hive_table where name=?, columns
...
@@ -134,6 +134,18 @@ atlas.lineage.hive.table.schema.query=hive_table where name=?, columns
</verbatim>
</verbatim>
---++ Search Configs
Search APIs (DSL and full text search) support pagination and have optional limit and offset arguments. Following configs are related to search pagination
<verbatim>
# Default limit used when limit is not specified in API
atlas.search.defaultlimit=100
# Maximum limit allowed in API. Limits maximum results that can be fetched to make sure the atlas server doesn't run out of memory
atlas.search.maxlimit=10000
</verbatim>
---++ Notification Configs
---++ Notification Configs
Refer http://kafka.apache.org/documentation.html#configuration for Kafka configuration. All Kafka configs should be prefixed with 'atlas.kafka.'
Refer http://kafka.apache.org/documentation.html#configuration for Kafka configuration. All Kafka configs should be prefixed with 'atlas.kafka.'
* The _!WithPath_ clause can be used with transitive closure queries to retrieve the Path that
* The _!WithPath_ clause can be used with transitive closure queries to retrieve the Path that
connects the two related Entities. (We also provide a higher level interface for Closure Queries
connects the two related Entities. (We also provide a higher level interface for Closure Queries
see scaladoc for 'org.apache.atlas.query.ClosureQuery')
see scaladoc for 'org.apache.atlas.query.ClosureQuery')
* ORDERBY is optional. Orderby clause should be specified in single quote ('). When order by clause is specified case insensitive sorting is done in ascending order.
* ORDERBY is optional. When order by clause is specified, case insensitive sorting is done based on the column specified.
For sorting in descending order specify 'DESC' after order by clause. If no order by is specified then no default sorting is applied.
For sorting in descending order specify 'DESC' after order by clause. If no order by is specified, then no default sorting is applied.
* LIMIT is optional. It limits the maximum number of objects to be fetched starting from specified optional offset. If no offset is specified count starts from beginning.
* LIMIT is optional. It limits the maximum number of objects to be fetched starting from specified optional offset. If no offset is specified count starts from beginning.
* There are couple of Predicate functions different from SQL:
* There are couple of Predicate functions different from SQL:
* _is_ or _isa_can be used to filter Entities that have a particular Trait.
* _is_ or _isa_can be used to filter Entities that have a particular Trait.
* _has_ can be used to filter Entities that have a value for a particular Attribute.
* _has_ can be used to filter Entities that have a value for a particular Attribute.
* When querying for a space delimited multiple-word identifier, it need to be enclosed within
* Any identifiers or constants with special characters(space,$,",{,}) should be enclosed within backquote (`)
backquote (`)
---+++ DSL Examples
---+++ DSL Examples
For the model,
* from DB
Asset - attributes name, owner, description
DB - supertype Asset - attributes clusterName, parameters, comment
* 1. SrcQuery [select] [orderby desc] [Limit x offset y] -> source query followed by optional select statement followed by optional order by followed by optional limit
* 1. SrcQuery [select] [orderby desc] [Limit x offset y] -> source query followed by optional select statement followed by optional order by followed by optional limit
* eg: Select "hive_db where hive_db has name orderby 'hive_db.owner' limit 2 offset 1"
* eg: Select "hive_db where hive_db has name orderby 'hive_db.owner' limit 2 offset 1"
{"hive_db where (name = \"Reporting\") orderby 'hive_db.name'",1,"name",isAscending},
{"hive_db where (name = \"Reporting\") orderby name",1,"name",isAscending},
{"hive_db where (name = \"Reporting\") orderby 'hive_db.name' limit 10",1,"name",isAscending},
{"hive_db where (name = \"Reporting\") orderby 'hive_db.name' limit 10",1,"name",isAscending},
{"hive_db where (name = \"Reporting\") select name as _col_0, owner as _col_1 orderby '_col_1'",1,"_col_1",isAscending},//will it work
{"hive_db where (name = \"Reporting\") select name as _col_0, owner as _col_1 orderby '_col_1'",1,"_col_1",isAscending},//will it work
{"hive_db where (name = \"Reporting\") select name as _col_0, owner as _col_1 orderby '_col_1' limit 10",1,"_col_1",isAscending},
{"hive_db where (name = \"Reporting\") select name as _col_0, owner as _col_1 orderby owner limit 10",1,"_col_1",isAscending},
{"hive_db where hive_db has name orderby 'hive_db.owner'",3,"owner",isAscending},
{"hive_db where hive_db has name orderby owner",3,"owner",isAscending},
{"hive_db where hive_db has name orderby 'hive_db.owner' limit 5",3,"owner",isAscending},
{"hive_db where hive_db has name orderby owner limit 5",3,"owner",isAscending},
{"hive_db where hive_db has name orderby 'hive_db.owner' limit 2 offset 0",2,"owner",isAscending},
{"hive_db where hive_db has name orderby owner limit 2 offset 0",2,"owner",isAscending},
{"hive_db where hive_db has name orderby 'hive_db.owner' limit 2 offset 1",2,"owner",isAscending},
{"hive_db where hive_db has name orderby 'hive_db.owner' limit 2 offset 1",2,"owner",isAscending},
{"hive_db where (name = \"Reporting\") select name as _col_0, (createTime + 1) as _col_1 orderby '_col_1'",1,"_col_1",isAscending},
{"hive_db where (name = \"Reporting\") select name as _col_0, (createTime + 1) as _col_1 orderby '_col_1'",1,"_col_1",isAscending},
...
@@ -604,15 +634,15 @@ public class GraphBackedDiscoveryServiceTest extends BaseRepositoryTest {
...
@@ -604,15 +634,15 @@ public class GraphBackedDiscoveryServiceTest extends BaseRepositoryTest {
{"hive_db where (name = \"Reporting\") select name as _col_0, (createTime + 1) as _col_1 orderby '_col_1' limit 10 offset 1",0,"_col_1",isAscending},
{"hive_db where (name = \"Reporting\") select name as _col_0, (createTime + 1) as _col_1 orderby '_col_1' limit 10 offset 1",0,"_col_1",isAscending},
{"hive_db where (name = \"Reporting\") select name as _col_0, (createTime + 1) as _col_1 orderby '_col_1' limit 10 offset 0",1,"_col_1",isAscending},
{"hive_db where (name = \"Reporting\") select name as _col_0, (createTime + 1) as _col_1 orderby '_col_1' limit 10 offset 0",1,"_col_1",isAscending},
{"hive_table where (name = \"sales_fact\" and createTime > \"2014-01-01\" ) select name as _col_0, createTime as _col_1 orderby '_col_1' ",1,"_col_1",isAscending},
{"hive_table where (name = \"sales_fact\" and createTime > \"2014-01-01\" ) select name as _col_0, createTime as _col_1 orderby createTime",1,"_col_1",isAscending},
{"hive_table where (name = \"sales_fact\" and createTime > \"2014-01-01\" ) select name as _col_0, createTime as _col_1 orderby '_col_1' limit 10 ",1,"_col_1",isAscending},
{"hive_table where (name = \"sales_fact\" and createTime > \"2014-01-01\" ) select name as _col_0, createTime as _col_1 orderby createTime limit 10 ",1,"_col_1",isAscending},
{"hive_table where (name = \"sales_fact\" and createTime > \"2014-01-01\" ) select name as _col_0, createTime as _col_1 orderby '_col_1' limit 10 offset 0",1,"_col_1",isAscending},
{"hive_table where (name = \"sales_fact\" and createTime > \"2014-01-01\" ) select name as _col_0, createTime as _col_1 orderby createTime limit 10 offset 0",1,"_col_1",isAscending},
{"hive_table where (name = \"sales_fact\" and createTime > \"2014-01-01\" ) select name as _col_0, createTime as _col_1 orderby '_col_1' limit 10 offset 5",0,"_col_1",isAscending},
{"hive_table where (name = \"sales_fact\" and createTime > \"2014-01-01\" ) select name as _col_0, createTime as _col_1 orderby '_col_1' limit 10 offset 5",0,"_col_1",isAscending},
{"hive_table where (name = \"sales_fact\" and createTime >= \"2014-12-11T02:35:58.440Z\" ) select name as _col_0, createTime as _col_1 orderby '_col_0' ",1,"_col_0",isAscending},
{"hive_table where (name = \"sales_fact\" and createTime >= \"2014-12-11T02:35:58.440Z\" ) select name as _col_0, createTime as _col_1 orderby name",1,"_col_0",isAscending},
{"hive_table where (name = \"sales_fact\" and createTime >= \"2014-12-11T02:35:58.440Z\" ) select name as _col_0, createTime as _col_1 orderby '_col_0' limit 10 offset 0",1,"_col_0",isAscending},
{"hive_table where (name = \"sales_fact\" and createTime >= \"2014-12-11T02:35:58.440Z\" ) select name as _col_0, createTime as _col_1 orderby name limit 10 offset 0",1,"_col_0",isAscending},
{"hive_table where (name = \"sales_fact\" and createTime >= \"2014-12-11T02:35:58.440Z\" ) select name as _col_0, createTime as _col_1 orderby '_col_0' limit 10 offset 1",0,"_col_0",isAscending},
{"hive_table where (name = \"sales_fact\" and createTime >= \"2014-12-11T02:35:58.440Z\" ) select name as _col_0, createTime as _col_1 orderby name limit 10 offset 1",0,"_col_0",isAscending},
{"hive_table where (name = \"sales_fact\" and createTime >= \"2014-12-11T02:35:58.440Z\" ) select name as _col_0, createTime as _col_1 orderby '_col_0' limit 10",1,"_col_0",isAscending},
{"hive_table where (name = \"sales_fact\" and createTime >= \"2014-12-11T02:35:58.440Z\" ) select name as _col_0, createTime as _col_1 orderby name limit 10",1,"_col_0",isAscending},
{"hive_table where (name = \"sales_fact\" and createTime >= \"2014-12-11T02:35:58.440Z\" ) select name as _col_0, createTime as _col_1 orderby '_col_0' limit 0 offset 1",0,"_col_0",isAscending},
{"hive_table where (name = \"sales_fact\" and createTime >= \"2014-12-11T02:35:58.440Z\" ) select name as _col_0, createTime as _col_1 orderby '_col_0' limit 0 offset 1",0,"_col_0",isAscending},
...
@@ -624,7 +654,7 @@ public class GraphBackedDiscoveryServiceTest extends BaseRepositoryTest {
...
@@ -624,7 +654,7 @@ public class GraphBackedDiscoveryServiceTest extends BaseRepositoryTest {
@@ -73,50 +77,49 @@ public class MetadataDiscoveryResource {
...
@@ -73,50 +77,49 @@ public class MetadataDiscoveryResource {
/**
/**
* Search using a given query.
* Search using a given query.
*
*
* @param query search query in raw gremlin or DSL format falling back to full text.
* @param query search query in DSL format falling back to full text.
* @param limit number of rows to be returned in the result, used for pagination. maxlimit > limit > 0. -1 maps to atlas.search.defaultlimit property value
* @param offset offset to the results returned, used for pagination. offset >= 0. -1 maps to offset 0
LOG.error("Unable to get entity list for query {} using dsl",query,throwable);
returnsearchUsingFullText(query);
}finally{
AtlasPerfTracer.log(perf);
}
}
AtlasPerfTracer.log(perf);
returnresponse;
}
}
/**
/**
* Search using query DSL format.
* Search using query DSL format.
*
*
* @param dslQuery search query in DSL format.
* @param dslQuery search query in DSL format.
* @param limit number of rows to be returned in the result, used for pagination. maxlimit > limit > 0. -1 maps to atlas.search.defaultlimit property value
* @param offset offset to the results returned, used for pagination. offset >= 0. -1 maps to offset 0
* Limit and offset in API are used in conjunction with limit and offset in DSL query
* Final limit = min(API limit, max(query limit - API offset, 0))
@@ -189,13 +215,18 @@ public class MetadataDiscoveryResource {
...
@@ -189,13 +215,18 @@ public class MetadataDiscoveryResource {
* Search using full text search.
* Search using full text search.
*
*
* @param query search query.
* @param query search query.
* @param limit number of rows to be returned in the result, used for pagination. maxlimit > limit > 0. -1 maps to atlas.search.defaultlimit property value
* @param offset offset to the results returned, used for pagination. offset >= 0. -1 maps to offset 0
* NOTE: Pagination is not implemented currently for full text search, so limit and offset are not used
