Avoid creating huge arrays for collectors when a query does not have an upper bound limit

Description

Instead of allocating such arrays, we could introduce an incremental allocation behind the hood.
E.g.: we'd execute the query once to retrieve the top 100, and if there is more, we'd do it again with 200, then 400, then 800, and so on.

Environment

None

Activity

Show:
Yoann Rodière
June 11, 2020, 10:18 AM

See org.hibernate.search.query.engine.impl.QueryHits#scoreDoc in Hibernate Search 5.
I'm not entirely sure this is how it works for queries without a limit, but I think it is. Let's check?

Yoann Rodière
June 11, 2020, 10:35 AM

By the way we'd have to check this is still relevant in Search 6, too. Since we split the collection in two phases (one phase to collect topDocs + aggregations, the second to retrieve projections from top docs), this will only be a problem if the topDocs or aggregation collectors size their internal structures according to the limit. If they don't, and use automatically increasing container (List, ...), we're fine.

Assignee

Fabio Massimo Ercoli

Reporter

Fabio Massimo Ercoli

Labels

None

Suitable for new contributors

None

Feedback Requested

None

Components

Fix versions

Affects versions

Priority

Major
Configure