Herd-UI Search Issue

[tinshuksingh]

Hello Team,

I have created business object definition 'test_bus_def_1' from Swagger and now I am trying to search it from Herd-UI home page. It's giving NullPointerException.

Below are the logs -

• Mar-02-2018 10:29:04.186 [http-nio-8080-exec-10] INFO org.finra.herd.dao.JestClientFactory.getJestClient userId=TRUSTED_USER - Elasticsearch REST Client Settings: scheme=http, hostname=localhost, port=9200, serverUri=http://localhost:9200
  Mar-02-2018 10:29:04.837 [http-nio-8080-exec-10] ERROR finra.herd.service.helper.HerdErrorInformationExceptionHandler.logError userId=TRUSTED_USER - A general error occurred.
  java.lang.NullPointerException
  at io.searchbox.core.SearchResult.getHits(SearchResult.java:85) ~[jest-common-5.3.3.jar:?]
  at io.searchbox.core.SearchResult.getHits(SearchResult.java:73) ~[jest-common-5.3.3.jar:?]
  at io.searchbox.core.SearchResult.getHits(SearchResult.java:65) ~[jest-common-5.3.3.jar:?]
  at io.searchbox.core.SearchResult.getHits(SearchResult.java:61) ~[jest-common-5.3.3.jar:?]
  at org.finra.herd.dao.impl.IndexSearchDaoImpl.buildIndexSearchResults(IndexSearchDaoImpl.java:331) ~[herd-dao-0.59.0.jar:?]
  at org.finra.herd.dao.impl.IndexSearchDaoImpl.indexSearch(IndexSearchDaoImpl.java:299) ~[herd-dao-0.59.0.jar:?]
  at org.finra.herd.service.impl.IndexSearchServiceImpl.indexSearch(IndexSearchServiceImpl.java:124) ~[herd-service-0.59.0.jar:?]
  at org.finra.herd.service.impl.IndexSearchServiceImpl$$FastClassBySpringCGLIB$$b380baaf.invoke() ~[herd-service-0.59.0.jar:?]

• We added below entries in srch_idx table as it was giving error No active search index found for type BUS_OBJCT_DFNTN/TAG. -

    ```
    insert into srch_idx (srch_idx_nm, srch_idx_type_cd, srch_idx_stts_cd, creat_user_id, actv_fl) values ('test_bus_def_1', 'BUS_OBJCT_DFNTN', 'READY', 'TRUSTED_USER', 'Y');
    insert into srch_idx (srch_idx_nm, srch_idx_type_cd, srch_idx_stts_cd, creat_user_id, actv_fl) values ('test_bus_def_1_tag', 'TAG', 'READY', 'TRUSTED_USER', 'Y');
    ```
  

• Do we have to perform any steps to be able to search from Herd-UI home page?

[nateiam]

Tinshuk -

I said it before and will say it again. I love how you dig right in and get things going!

Here's the story on our search. You probably know from your knowledge of CloudFormation and install that we embed Elasticsearch behind the Herd search endpoints. We have this nicely integrated in our environments but have not yet put some settings and configurations in the open source release.

Unlike #370 with Uploader, this requires multiple configs to get it to work. I am mentioning @DavidBalash who is an expert at the Elasticsearch integration in Herd.

Here is an overview for your information -- David will share more details and good default configs and/or detailed instructions for each area:

• Instructions to build indexes
  • We created a couple Herd endpoints to build the search indexes. These endpoints read from the Herd repository and add documents to the Elasticsearch index.
  • There are also endpoints to activate a new index when it's build and to check the status of the indexes
  • You should delete the rows you manually created as the endpoints will manage that data
  • You might find the search index endpoints in the swagger docs. But also @DavidBalash, please take our internal documentation on how to create the indexes, sanitize it for OSS, and create a page on the Herd OSS wiki for Tinshuk and others
• Elasticsearch-related configs
  • Connectivity - it looks from your logs like you might already have this in place
  • Index mapping - there are a few config values that describe how Elasticsearch maps the Herd data when building an index...and related settings
  • @DavidBalash please provide good defaults to Tinshuk and also port all the elasticsearch.* entries from our internal config documentation to config doc in the the GitHub wiki

It's worth mentioning that we have another team working on including all this config and default settings in a CloudFormation so it's easy for open source. We're just not quite there yet -- but I am sure you and David can get your instance up and running!

[DavidBalash]

Hi Tinshuk,

I have added the following Elasticsearch setup page to the Herd wiki on GitHub:

https://github.com/FINRAOS/herd/wiki/Elasticsearch-setup

I have also added all of the Elasticsearch configuration values to the existing configuration values page on the wiki:

https://github.com/FINRAOS/herd/wiki/configuration-values

Please let us know if this helps to solve your search issue on the Herd-UI.

[zubair-nbx]

Hi,

I am facing the same issue, after activating the indexes I can see 4 documents in an index see image below:

Here is the curl request which I copied from the "Dev Tools" in chrome:

curl -X POST \ --header 'Content-Type: application/json' \ --header 'Accept: application/json' \ -d '{"searchTerm":"data_object_a","facetFields":["Tag","ResultType"],"indexSearchFilters":null,"enableHitHighlighting":true}' \ http://localhost:8080/herd-app/rest//indexSearch?fields=displayName,shortDescription&match

I am using the same version of ES as in pom file which is 5.1.1.

Here is the screen shot of Herd-UI app:

Please suggest/guide how to fix this.
@tinshuksingh Did Search worked for you ?

[nateiam]

Hi @zubair-nbx, can you please send the contents of your catalina.out from this time period? This should help us troubleshoot the issue.

Thank you!

[zubair-nbx]

Hi @nateiam , Thanks for your response. Please see my findings Below:

• Need to Enable Inline scripts in /config/elasticsearch.yml

script.inline: true script.stored: true script.file: true

The query generated by libs in Herd uses inline groovy script see the portion of query using script:

"script_score" : { "script" : { "inline" : "_score * (doc['_index'].value == 'bdef_1522162310906' ? doc['tagSearchScoreMultiplier']: 1)", "lang" : "groovy" }

This makes the Global search with filter 'All' works fine, but selecting filter 'Column' doesn't work still, see details below:

- Issue with ‘Column’ filter on Herd-UI homepage page global search:
When filter for ‘Column’ is enabled on Herd-UI homepage global search the query generated has 3 “multi_match” subqueries with includes the “fields”: [ ] and ES is not able to execute that query:

multi_match" : { "query" : "Price", "fields" : [], "type" : "phrase_prefix", "operator" : "OR", "slop" : 0, "prefix_length" : 0, "max_expansions" : 50, "lenient" : false, "zero_terms_query" : "NONE", "boost" : 1.0 }

Also note that, this setting is read from class “ConfigurationValue.java”

'ELASTICSEARCH_HIGHLIGHT_FIELDS("elasticsearch.highlight.fields", "{"fields": ["*"]}")'

It also causes exception while creating json object. The lib fasterXml / JSON is not able to parse the object {\"fields\": ["*"]}` into json and throws exception on method:

private HighlightBuilder buildHighlightQuery(String preTag, String postTag, Set<String> match)
in package:

herd-dao/src/main/java/org/finra/herd/dao/impl/IndexSearchDaoImpl.java

Code snippet below:

...try { @SuppressWarnings("unchecked") //Code crashes here IndexSearchHighlightFields highlightFieldsConfig = jsonHelper.unmarshallJsonToObject(IndexSearchHighlightFields.class, highlightFieldsValue); highlightFieldsConfig.getHighlightFields().forEach(highlightFieldConfig -> {.....

Hope it helps, please comment if more details is needed.
Thank you :)