Include an IndexDeletionPolicy which deletes older commits asynchronously

Description

When using an Infinispan Directory the deletion of old segments is one of the slowest operations, as it requires several network RPCs to make sure that each file is not being used anymore.
But also commits are a significantly sensitive operation on Infinispan as it scales reads but not writes to the index, and each such commit must be applied sequentially.

Clearly the RPCs being triggered for deletion purposes of stale segments stay in the way of other sequential operations which are actually useful for indexing progress, so we expect it to be effective to delegate the operations related to stale segments cleanup to a background executor.

Such an IndexDeletionPolicy is hence expected to be particularly suited for storage on Infinispan, but isn't necessarily coupled to Infinispan usage only.

Environment

None

Activity

Show:
Gustavo Fernandes
October 8, 2014, 6:24 PM
Edited

I tried it, and was wondering why perf was not changing.
CommitPoint.delete() ultimatelly does

which will be handled later by the IndexFileDeleter

Sanne Grinovero
October 10, 2014, 3:36 PM

Ok I was wrong. Can we get to delete asynchronously in some other way, or do we need to embed this logic in the Directory?

Gustavo Fernandes
October 10, 2014, 3:45 PM
Sanne Grinovero
October 10, 2014, 4:44 PM

No, I hadn't seen that PR. Will look to that as well now.. I guess it makes this issue redundant.

Sanne Grinovero
October 10, 2014, 4:44 PM

This won't be necessary if Infinispan embeds the same functionality in the Lucene Directory.

Assignee

Gustavo Fernandes

Reporter

Sanne Grinovero

Labels

Suitable for new contributors

None

Pull Request

None

Feedback Requested

None

Components

Priority

Critical
Configure