Persistent sets are marked dirty on session.merge() even if there have been no changes to the collection.
This is especially painful when the collection is immutable and results in an "changed an immutable collection instance" exception on flush.
I tracked the behaviour down a bit and believe the problem to be in CollectionType.replace().
Here the passed in orginal PersistentSet is replaced by a plain HashSet in this line:
Object result = target == null || target == original ? instantiateResult( original ) : target;
The "result" object (HashSet) is then passed to the CollectionType.replaceElements() method (instead of the original PersistentSet).
In CollectionType.replaceElements() the code to clear the dirty state of the collection does not execute anymore, because the passed-in "original" collection is the described HashSet and not the original PersistentSet.
This way the PersistentSet remains marked dirty.
A workaround is to manually clear the dirty state of an immutable collection after merge but before flush.