Restore support for concurrent Lucene work execution - at least during mass indexing

Description

In Search 5, we used to execute calls to the IndexWriter from multiple threads when mass indexing.

As explained here, Lucene does perform better when using multiple threads to index, at least if the hardware supports concurrent writes, thanks to concurrent flushing.

Indexing is also faster when using multiple threads because analysis is relatively CPU-intensive.

In Search 6, we execute all calls to the IndexWriter from the same thread, with one thread assigned to each index manager. As a result, mass indexing performance is likely to degrade compared to Search 5.

We should explore the following solutions, and pick whatever works best.

All solutions below rely on the same concept of work stealing: a thread that has nothing to do will try to help out the thread of another index. The different lies only in which thread will try to help out.

All solutions will require some changes in the LuceneWriteWorkProcessor, which is currently thread-unsafe.

Solution 1 (not recommended): mass indexing thread helping with indexing

Restore the old behavior of executing writes from the mass indexing thread directly. This may prove difficult given the current architecture.

One possible solution: return custom objects instead of CompletableFuture from org.hibernate.search.mapper.pojo.work.spi.PojoIndexer#add, and instead of calling future.join() from the massIndexer thread, call theReturnedObject.executeIfNotAlreadyExecuted(), which will execute the work unless it's already been executed by the per-index thread.

Solution 2 (better): other per-index threads helping with indexing

Allow threads assigned to one index to steal the work from another index when they become idle.

Ordering

As a result, even "traditional" (non-massindexer) worksets will no longer be executed in the order they were submitted.
We could give up on ordering completely, because we usually only submit one workset per transaction, and the order worksets are submitted in is not guaranteed to match the order of transactions anyway. That's probably a bad idea, because we would definitely increase the likelyhood of works being executed in the wrong order.
Maybe we should implement sharding by document ID in the queues, so that multiple threads can actually process works on the same index, but from different queues?

Benefits

This solution could get us decent performance improvement for all writes, not just mass indexer writes, as soon as there are more than one index and only some of the indexes are being written to. Which is typically the case when mass indexing: by default we only mass index one type (= one index) at a time, but there are generally multiple types (= multiple indexes) to mass index.

We may even introduce settings to define a minimum and maximum number of threads, so that a Hibernate Search instance with 100 indexes still only uses 10 threads (max = 10), for example, with each thread sequentially working on each index, and a Hibernate Search instance with a single index still uses 2 threads (min = 2) to benefit from concurrent flushing.

For Infinispan, where we usually have only one index per cache (so one index per Hibernate Search instance), we could provide an SPI that allows Infinispan to define its own thread pool shared across all caches. That way, Infinispan would benefit from concurrent flushing.

Environment

None

Assignee

Yoann Rodière

Reporter

Yoann Rodière

Labels

None

Suitable for new contributors

None

Feedback Requested

None

Components

Fix versions

Priority

Major
Configure