Uploaded image for project: 'Hibernate ORM'
  1. HHH-12036

Inefficient Infinispan cache invalidation for bulk operations

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 5.0.10, 5.1.10, 5.2.11
    • Fix Version/s: 5.2.13
    • Component/s: hibernate-infinispan
    • Labels:
      None
    • Environment:
      WildFly 10.1.0, 11.0.0.CR1, 11.0.0 master
    • Bug Testcase Reminder (view):

      Bug reports should generally be accompanied by a test case!

    • Last commented by a user?:
      true
    • Sprint:

      Description

      Hibernate-infinispan uses a very inefficient way to perform cache invalidation for bulk operations (JPA CriteriaUpdate/CriteriaDelete). Rather than broadcasting a clear to all nodes in the cluster, the cache is cleared entry by entry. As entity caches are invalidating caches by default, this requires a query to all remote nodes first to collect all keys. These keys are then bundled in a very large message and sent out to all nodes again. During this entire procedure, it seems the cache region is locked on all nodes, causing the entire cluster to stall (I presume this is needed to prevent inserts into the cache between the query and the invalidation phase).

      We are seeing this behavior on WildFly 10.1.0, 11.0.0.CR1 and 11 master. The correspoding code in Hibernate is:
      https://github.com/hibernate/hibernate-orm/blob/master/hibernate-infinispan/src/main/java/org/hibernate/cache/infinispan/access/InvalidationCacheAccessDelegate.java#L144 and
      https://github.com/hibernate/hibernate-orm/blob/master/hibernate-infinispan/src/main/java/org/hibernate/cache/infinispan/util/Caches.java#L280

      The current implementation makes it impossible to perform batch operations on large cache regions with tens of thousands of entries spanning multiple nodes without blocking the entire cluster for many seconds, even up to a minute. On some places we can change the code to update the entries one by one. However, in other places this will result in thousands of queries to the database in stead of 1, making it far from ideal.

      It seems Infinispan lacks a cluster-wide clear command. Therefore, I'll be filing a bug report at Infinispan as well. Note that the documentation of Cache.entrySet contains the following sentence: "Use involving execution of this method on a production system is not recommended as they can be quite expensive operations".

        Attachments

          Issue links

            Activity

              People

              • Votes:
                1 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: