Always enable delayed commits in the Lucene backend

Description

Let's start by saying that delayed commits are nothing to be afraid of. When they are enabled, write operations that require an immediate commit (e.g. automatic indexing by default) are still committed immediately. The delayed commits are only effective when some write operations do not perform an immediate commit. In that case, the user already decided to take the chance of losing data in the event of a crash, so waiting 1 second before the commit will not change much to the data safety.
It does change much when it comes to performance, though. Instead of committing e.g. every 100 operations, we commit e.g. every second: maybe no additional operation will be performed by then, in which case which case performance is the same, but maybe thousands of write operations will have be performed by then, in which case we just reduced commits by a factor 10... and probably increased throughput by the same factor.

With that in mind... not enabling delayed commits is becoming a problem. Early numbers from the performance tests on show that any improvement to the performance when delayed commits are enabled will probably lead to massive performance regressions when delayed commits are not enabled. We're talking about doubling the throughput when delayed commit are enabled, while dividing it by a factor 10 when delayed commits are disabled.

The reason for the regressions is that we have to commit after each batch of writes when delayed commits are disabled, and the improvements are about performing multiple, but smaller batche of writes in parallel. So when delayed commits are disabled, the "improvements" actually increase the amount of commits. When delayed commits are enabled, they do not.

Environment

None

Assignee

Yoann Rodière

Reporter

Yoann Rodière

Labels

None

Suitable for new contributors

None

Feedback Requested

None

Components

Fix versions

Priority

Major
Configure