When using an Infinispan Directory the deletion of old segments is one of the slowest operations, as it requires several network RPCs to make sure that each file is not being used anymore.
But also commits are a significantly sensitive operation on Infinispan as it scales reads but not writes to the index, and each such commit must be applied sequentially.
Clearly the RPCs being triggered for deletion purposes of stale segments stay in the way of other sequential operations which are actually useful for indexing progress, so we expect it to be effective to delegate the operations related to stale segments cleanup to a background executor.
Such an IndexDeletionPolicy is hence expected to be particularly suited for storage on Infinispan, but isn't necessarily coupled to Infinispan usage only.
I tried it, and was wondering why perf was not changing.
CommitPoint.delete() ultimatelly does
which will be handled later by the IndexFileDeleter
Ok I was wrong. Can we get to delete asynchronously in some other way, or do we need to embed this logic in the Directory?
No, I hadn't seen that PR. Will look to that as well now.. I guess it makes this issue redundant.
This won't be necessary if Infinispan embeds the same functionality in the Lucene Directory.