SPI for "hot" restart that allows compatible schema changes with Lucene but doesn't interrupt search/indexing availability
Description
follows up on
Activity
Yoann RodièreMay 6, 2022 at 8:10 AM
Created to implement the first step (keep localheap Directory data, but don’t allow using the old Hibernate Search instance during the restart).
We’ll get back to this ticket as a follow-up if we discover we really need to allow using the old Hibernate Search instance during the restart.
Yoann RodièreApril 21, 2022 at 1:56 PM
I’m thinking that a first step would be to allow the new instance to just reuse the DirectoryHolders from the old instance, and to restart everything else (thread pools, queues, …).
In practice, that would mean:
Infinispan would have a per-cache read/write lock on Hibernate Search: it would acquire the read lock before any use of Hibernate Search (query or indexing).
Before a restart, Infinispan would acquire write lock, ensuring that concurrent queries/indexing will wait a little bit.
The restart would assume nobody will be sending requests anymore, would flush all queues and perform commits, then close everything except the DirectoryHolders.
The DirectoryHolders would be passed (somehow) to the new instance of Hibernate Search, and reused as-is (so that
ByteBuffersDirectorycontent won’t be lost).The new instance of Hibernate Search would finish starting, and be returned to Infinispan.
Infinispan would replace the old SearchMapping instance with the new one.
Infinispan would release the write lock, allowing queries/indexing to proceed.
This is the safest approach in my opinion.
It will introduce a little latency (the amount of time Hibernate Search needs to flush, stop, and restart), but:
We will need some locks in Infinispan whatever happens, or we would run the risk of creating a query before the Hibernate Search restart and running it after the restart, which would fail miserably.
I think some latency is to be expected for an operation that involves changing the schema of your cache.
The latency may be small or at least acceptable in the use cases of the people requesting this feature; we’ll have no idea until we run performance tests.
I think we can improve performance incrementally afterwards. At least the feature and restart SPI will be there. We just need to design that SPI in such a way that it allows an atomic switch of the SearchMapping, even if initially it would be useless.
If we end up looking at improving the resulting restart latency:
The first step would be, IMO, to rely on Infinispan’s “common” thread pools instead of creating our own (see ), so that we don’t have to create threads during a restart.
Then we could have a look at how to reuse everything below the search/indexing features in the new SearchMapping, so that operations initiated on the old SearchMapping would continue working (at least for a short time) after we switched to the new SearchMapping.
Yoann RodièreApril 19, 2022 at 4:20 PM
Hey , if you start working on this, let’s discuss SPIs first
Details
Assignee
UnassignedUnassignedReporter
Yoann RodièreYoann RodièreComponents
Priority
Major
Details
Details
Assignee
Reporter
Yoann RodièreComponents
Priority
Major
Follows up on .
Infinispan needs to be able to add fields to a schema, without disrupting the execution of queries
One solution Hibernate Search could provide is the ability to restart, with a different mapping (provided by Infinispan), while not ever stopping some key underlying resources: index writers, thread pools, REST clients, etc. The new instance would reuse the same writers, pools, etc. That way:
While Hibernate Search is restarting, the old Hibernate Search instance can be used.
At some point, the new instance of Hibernate Search, relying on the same underlying resources (writers, pools, …), becomes usable.
The caller makes the switch to the new instance.
The old Hibernate Search instance gets closed, freeing up resources that are not reused in the new instance (but not those that are reused, obviously).
Note that, in order for this to work, there must be a small period of time where both the old and new instance of Hibernate Search are usable, and we must allow the caller to make the switch from the old instance to the new one before the old one becomes unusable (closed).
Also note, in case of failure to restart (e.g. invalid mapping), the old instance should remain usable.
NON-GOALS;
Hibernate Search will not check the compatibility of the new schema with the old one. We assume the caller knows what it’s doing (i.e. they only add new fields).
Hibernate Search will not trigger mass indexing automatically during the restart. We assume previously indexed data does not need reindexing, or if it does, the caller will take care of that.
Hibernate Search will not provide “hot” reindexing as part of this ticket. This is addressed by other tickets.