The change to #close() will most definitively fix the issue. Thank you for fixing the problem!

We completely understand and in fact I'll be making a change so that when #close() is called, those delayed AfterTransactionCompletion callbacks fire prior to closing the Session and/or EntityManager.

Thank you for the thorough investigation into this problem!

The problem is that we are not in fact using the EntityManager in this way. I merely tried to reproduce the problem with as little code as possible using an example from Jboss/Narayana. We are in fact using Spring in combination with Narayana and Hibernate. We have an environment which is, for the most part, message driven through Spring's DefaultMessageListenerContainer. So it is actually this Spring code that is doing the calls to the JtaTransactionManager. Our EntityManager is (of course) injected through the @PersistenceContext annotation. So the code to close the EntityManager is not actually in our own codebase.

But I cannot post the sources of our application publicly and it is also too massive (thousands of classes) to be used as an 'example'. I will start working on an example to recreate the memory leak in a manner that more resembles our environment.

After reviewing your test case, the first thing which stood out to me was how you handled closing the EntityManager. Your code does this:

One could argue that potentially the EntityManager was never constructed or opened in the try-block for some reason.
With that in mind, I would say a better finally-block would be this:

This has a positive impact on your situation because it guarantees that the AfterTransactionCompletion callbacks fire, thus avoiding the memory leak you describe and illustrate in your test case. The reason is because the #isOpen() method performs some transaction management checks that notice that the callbacks were delayed by the reaper thread and must execute on the original calling thread. Its this call which effectively forces them to fire thus clearing the AuditProcess queue for you.

With that said, this will throw a HibernateException because the underlying SynchronizationCallbackCoordinatorTrackingImpl#processAnyDelayedAfterCompletion will be called and presently informs the original calling thread of the following after executing the AfterTransactionCompletion callbacks.

What you could then do is handle that exception yourself if you anticipate it so it doesn't present an issue as follows:

I could discuss this with the team to see if there are any concerns with us adding the same type of transaction management checks from #isOpen() to #close(). If there is no issue, that would atleast allow those after-completion callbacks to fire to avoid the memory leak you described. But that will still imply a change in your finally block to handle the PersistenceException for your use case.

I have also added the testcase as an attachment. You can run it by just running main() in QuickStartApplication. I added some logging and through Reflection retrieve the size of the auditProcesses at the end. You can see it returns 1 whereas it should be 0.