ATLAS_4988: BusinessMetadata with attribute of data type Array takes time to Delete.

[UmeshPatil-1]

ATLAS-4988
What changes were proposed in this pull request?
Background: In Apache Atlas, deleting a Business Metadata (BM) definition requires a pre-check to ensure no existing entities are assigned to those attributes. For attributes with a data type of Array (multi-valued), the deletion process was taking an exceptionally long time (e.g., over 5 minutes for -38k entities), even when the BM was not assigned to any entity.

Root Cause:

1. Missing Solr Index: Attributes of type Array are generally not indexed in Solr.
2. Inefficient Search: The old logic forced a Solr-based search (entityDiscoveryService) for all attributes. When an attribute is not indexed, Solr performs a full collection scan, leading to severe performance degradation.
3. Data Payload: The search request was fetching full attribute values (setAttributes), adding unnecessary I/O overhead for a simple existence check.

Changes Proposed:

1. Hybrid Validation Strategy: Introduced a conditional check to determine if an attribute is indexable before deciding the search path.
2. Optimized Indexed Search: For indexable attributes, the Solr search was optimized by using the NOT_NULL operator and clearing requested attributes (Collections.emptySet()) to return only the GUID, significantly speeding up the response.
3. Direct Graph Fallback: Introduced isBusinessAttributePresentInGraph() for non-indexable attributes. This method queries the Graph database (JanusGraph) directly.
4. Targeted Lookups: The graph fallback uses Constants.ENTITY_TYPE_PROPERTY_KEY to narrow the scope to only relevant entity types, avoiding full system scans.
5. Integrity Maintenance: Ensured that even unindexed attributes are checked before deletion to prevent "dangling" metadata, while maintaining sub-second performance.

Impact:

Performance Gain: Deletion time for Business Metadata with Array attributes reduced from -341 seconds to -1.5 second.
Reliability: Maintains strict data integrity by ensuring no "in-use" Business Metadata can be deleted.
Scalability: The fix ensures that as the number of entities grows, the deletion of metadata remains performant.

How was this patch tested?
Maven Build:
Build Successful.

Manual Testing:

1. Creation & Deletion (No Assignment): Created Business Metadata with String and Array types. Verified deletion is nearly instantaneous (-0.8s - 1.5s).
2. Deletion Blocked (With Assignment): * Assigned a Business Metadata attribute to a hive_table entity.
3. Attempted to delete the BM definition.
4. Verified the system correctly throws ATLAS-409-00-002: Given type has references.
5. Large Dataset Validation: Tested against a repository containing -38,000 hive_table entities to confirm the Solr timeout and latency issue is resolved.

[sheetalshah1007]

Extract the attribute iteration logic in checkBusinessMetadataRef to a private method (e.g., validateAttributeReferences) for better readability.

[UmeshPatil-1]

Done

[sheetalshah1007]

1. Replace manual iteration with Graph Query for better performance:
The method isBusinessAttributePresentInGraph manually iterates over all vertices of each applicable type using graph.getVertices(Constants.ENTITY_TYPE_PROPERTY_KEY, typeName), then checks each vertex for the property and state. This can be slow for large graphs, as it loads and inspects every vertex in memory. This needs to be improved

2. Add error handling and logging:
No exception handling in the graph operations.
If the graph is unavailable or queries fail, it could lead to unhandled exceptions. Wrap the query in a try-catch and log warnings/errors.
Also, add optional performance logging to monitor query times in production.

3. Add JUnit tests for the new method covering:
-Non-indexable attributes with/without assignments.
-Inactive entities (should not count).

[mneethiraj]

Is it necessary to add ENTITY_TYPE_PROPERTY_KEY filter here? If yes, note that all subTypes of types inapplicableTypes should be checked as well; consider a business attribute associated with a type like DataSet.

[UmeshPatil-1]

You are correct. The ENTITY_TYPE_PROPERTY_KEY filter is not necessary and will remove these.

[mneethiraj]

Is it necessary to check only for ACTIVE entities? The business metadata can't be deleted even when a deleted entity has reference to it, right?

[UmeshPatil-1]

Thank's for pointing these out, To ensure full data integrity, we should block the deletion of Business Metadata if any entity—Active or Deleted—still holds a reference to it.