Search queries on multiple-node setup with Elasticsearch + dynamic sharding may ignore some indexes

Description

See https://hibernate.atlassian.net/browse/HSEARCH-2674#icft=HSEARCH-2674 : it's approximately the same problem.
When you have multiple Hibernate Search nodes, all of them targeting the same indexes in a single Elasticsearch cluster, you may end up having some "dynamic shards" (i.e. ES indexes) ignored when querying, simply because each Hibernate Search node is only aware of the shards it created itself.

One solution would be to not ever mention the index names when querying (except when using explicit shard filtering), and only filter by type.
We'd have to check how well this performs, though: ideally Elasticsearch would first narrow down the list of indexes to query based on all the indexes that mention the targeted types in their metadata, but I'm not sure it does.
=> Actually no, we cannot do this, because the list of targeted indexes may or may not have been filtered by the ShardIdentifierProvider depending on filters, and we have absolutely no clue as to whether there has been some filtering. So we cannot arbitrarily decide to make the query target every index... I'll mention this issue in HSEARCH-2674, which I'm afraid will have to be fixed first.
=> Also, I tried a bit to execute queries on all indexes, but on a specific set of types, and it seems Elasticsearch actually executes the request against all indexes, so the performance gain (which was the point of dynamic sharding in the first place) is lost, and we may even have worse performance than without sharding... Maybe we should consider using Elasticsearch shards + custom routing instead of creating multiple indices when doing dynamic sharding in the Elasticsearch case? (see HSEARCH-2634)

Activity

Show:

Yoann Rodière September 26, 2023 at 9:37 AM

Hello,

This issue was reported against Hibernate Search 5.x or earlier, and doesn't seem to be relevant to newer Hibernate Search versions (6.x+).

In order to focus on newer versions, we are going to close this issue.

If you are still affected by this issue on Hibernate Search 6.0 or later, and have a reproducer, please comment here or reach out to us so we can reopen the issue.

If you are still affected by this issue on Hibernate Search 5.x, and want to provide a fix, please comment here or reach out to us so we can work out the next steps with you.

Cheers,
Yoann

Yoann Rodière May 24, 2017 at 7:59 AM

Moving to 6.0: we're blocked by HSEARCH-2674.
I added a (failing) test case on my branch at https://github.com/yrodiere/hibernate-search/tree/HSEARCH-2725

Out of Date

Details

Assignee

Reporter

Fix versions

Priority

Created May 15, 2017 at 2:05 PM
Updated September 26, 2023 at 9:37 AM
Resolved September 26, 2023 at 9:37 AM