We're updating the issue view to help you get more done. 

Add configuration options for the size of workset queues, workset batches and Elasticsearch bulks

Description

These sizes are currently hardcoded:

  1. The maximum bulk size in the Elasticsearch backend

  2. The maximum number of worksets per batch in the serial orchestrator of the Elasticsearch backend

  3. The maximum number of worksets per batch in the parallel orchestrator of the Elasticsearch backend

  4. The maximum capacity of the workset queue in the serial orchestrator of the Elasticsearch backend

  5. The maximum capacity of the workset queue in the parallel orchestrator of the Elasticsearch backend

  6. The maximum number of worksets per batch in the write orchestrator of the Lucene indexes

  7. The maximum capacity of the workset queue in the write orchestrator of the Lucene indexes

We should address three problems:

  1. The hardcoded sizes may not be very good. For example we allow 5000 worksets to queue up for execution in the parallel orchestrator of the Elasticsearch backend. Each workset might contain several works. A tad too much, maybe?

  2. Even if we make the default values better, they'll never fit every use case. Users should be able to change them through configuration properties.

  3. We should document how to pick a sensible size for queues.

For workset queues, keep in mind the queue should be at least equal to (estimated number of user threads)*(estimated number of worksets created by each thread).

For workset queues, we might want to allow to set the capacity to "unlimited" for people who'd rather get an OOM error than block because the queue is full. For infinite capacity, a linked list as implemented in org.hibernate.search.backend.impl.lucene.MultiWriteDrainableLinkedList might help.

Note there was a configuration option for the maximum work queue length of the async executor in Search 5: see org.hibernate.search.indexes.impl.PropertiesParseHelper#extractMaxQueueSize. The sync executor, however, had a queue of unlimited capacity (a linked list).

Environment

None

Status

Assignee

Unassigned

Reporter

Yoann Rodière

Fix versions

Priority

Major