Two concurrent element shifts in a list damage database integrity.

Description

Problem description: Assume you have an entity Foo having a list of Bars. It's a unidirectional OneToMany mapping with a @JoinColumn annotation. When shifting element in the lists from within two different transactions starting concurrently the database gets inconsistent and elements disappear when the range of affected elements overlaps.

Example: when you have a list with three elements and shift element 0 to index 1 in one and element 2 to 1 in another transaction then you'll end up with a table with two indices 1 and one index 2. Reloading the collection in hibernate in a new transaction the element at index 0 is then 0. More generally speaking arbitrary list elements may vanish in this scenario when working with bigger lists and other elements to be shifted. There is also no way to reinsert the vanished element into the collection unless you have it cached somewhere else. Here is the relevant code from the attached reproducer.

The output of the logging statements I inserted is

Analysis: This behavior is caused by the way hibernate rewrites the list's indices. Only affected indices will be updated. When the shift affects n list elements only the indices of the n affected elements will be updated. Hibernate generates 2*n UPDATE statements for this. The first n to set all the affected indices to null, the second n statements to write the new index to the rows. This is a perfectly fine and clever optimization in a single transactioned / threaded environment but fails when multiple transactions are involved. That is because the transaction manager does not know about lists and applies both changes consecutively. It does not have a chance to detect the resulting conflict because the changes are incomplete from a transactional point of view.

Suggestion: Hibernate should update all list indices when it detects that elements of a collection were moved. Regardless of the transaction interleaving this would guarantee that database remains in a consistent state. The transaction that is closed last would then define the state without damaging database consistency.

I have a attached a unit test exposing the undesired behavior. Sorry for the two attachments. Despite the name they are the same.

Environment

JDK 11, Hibernate Core 5.4.16

Activity

Show:
Björn Zurmaar
June 19, 2020, 7:11 AM

Thank you very much for taking the time to look at my code. I really appreciate that!

I was under the impression that flushing is done automatically. The flushmode should only determine when this flush happens as far as i know. I also ran the code you provided with the exact same result as mine. Index 1 occurs two times and index 0 has vanished in the database. Consequently bar1 disappears from the list.

It’s not that the data expected is not written to the database. The problem is that hibernate only rewrites the affected indices. This is not sufficient to create transactional safety (in the sense of keeping the collection sane) from my point of view.

Christian Beikov
June 19, 2020, 7:15 AM

I don’t know what code you ran, but the code I attached blows up with an {{OptimisticLockException}} just like you would expect.

Björn Zurmaar
June 19, 2020, 8:04 AM

Ok, as you wrote about the missing flushes I assumed you just added the flushes and only copied JPAUnitTestCase.java. Sorry for the confusion. After some experiments I found out that the flush() call does not change anything. The key here is that you modified Foo.java and added:

Thank you so much for showing me a way to fix the problem.

While the problem can be fixed this way I’m even more puzzled than before. Hibernate uses optimistic locking by default. But in order to make this work you need a version field in your entity. If this is correct:

  1. Is there any scenario where you can safely or actually should omit a version field?

  2. If this is a requirement for optimistic locking to work properly, why does hibernate not complain about the missing field or automatically generate such a field on the DB level?

Christian Beikov
June 19, 2020, 8:23 AM

You are asking very basic JPA questions and I don’t think this is the right place to ask these question.

I don’t want to be rude but maybe you should buy a book about JPA/Hibernate and read about the core concepts to understand this better. In general, optimistic locking requires a version field, that’s just how optimistic locking works. You could also use pessimistic locking which uses a database lock, but that only holds for the transaction lifetime.

Björn Zurmaar
June 19, 2020, 8:49 AM

Thanks for the feedback and your help.

Assignee

Christian Beikov

Reporter

Björn Zurmaar

Fix versions

None

Labels

None

backPortable

None

Suitable for new contributors

None

Requires Release Note

None

Pull Request

None

backportDecision

None

Components

Affects versions

Priority

Major
Configure