Inconsistent cache in a cluster

Description

Seems that cache operations are not properly propagated from one node (1) in the cluster to another (2). Looks like node (2) executes cache operation concurrently so they can be executed in the wrong order.

Here is the test case:

There is a cluster with two nodes.
1) Parent (Customer) and child (Contact) objects are persisted on node (1).
2) Parent and child are loaded on node (2).
3) New child is persisted on node (1).
4) New child is deleted on node (1).
5) Parent and child are loaded on node (2), which results with error:

org.hibernate.ObjectNotFoundException: No row with the given identifier exists: [org.hibernate.test.Contact#2]

According to production logs it seems that on node (2) PutKeyValueCommand and RemoveCommand are executed in the wrong order and Contact#2 object, that was deleted, stays in the cache. Reason for this could be that Infinispan's "putForExternalRead()" method is used, "...where, concurrent modification and transactions are not a consideration..." (https://issues.jboss.org/browse/JBCACHE-848).

In the test log (attached) it's hard to tell which thread belongs to which "node" in order to see what really happens (I was not able to make "threadNamePrefix" work).

I also had problems setting up interceptors (another Infinispan bug?) in order to synchronize cache operations and make the problem 100% reproducible so you'll have to run test case several times until you reproduce it.

Environment

Hibernate 4.2.16.Final, Infinispan 5.3.0.Final

Status

Assignee

Unassigned

Reporter

Miljenko Brkic

Fix versions

None

Labels

None

backPortable

None

Suitable for new contributors

None

Requires Release Note

None

Pull Request

None

backportDecision

None

Components

Affects versions

4.2.16

Priority

Major