We found possible memory leak in BoundedConcurrentHashMap.LIRS implementation.
I think that problem is in a org.hibernate.internal.util.collections.BoundedConcurrentHashMap.LIRS.clear() method.
This method clear an accessQueue variable but LIRS stack (represented by a header variable) is not cleared.
I attached an image from heap analysis to demonstrate this problem.
In this example SessionFactoryImpl is properly closed (method close() is called on every instance of LIRS).
could you go into the detail of the segment content?
It would be interesting to see the size of a typical cache entry and where do all these bytes go inside this cache entry. We really need the details to see if we can improve the size of a cache entry or if it's not possible.
@Sanne Grinovero You made a good point about 2048 Entries. I have more than one war caching data into one heap that is why it is more than 2048. I guess these entries are large enough that I can not store 10K(across multiple war files) of them on my heap. I can only take one dump per server so i really can not separate them but I have shared screenshots of individual Potential leak as reported by MAT tool, those are still relevant.
@Guillaume Smet Please find the attached screen shot. Looks each query cache is taking 25MB ish. That seems a lot to me. I am sure i have large number(200 or so) of parameters but still 25MB is a lot.
A segment contains more than one entry. I think the relevant information would be in table. What I need is the detail of the HQLQueryPlan objects (and probably the QueryTransformer objects beneath it).
Let's summarize this issue:
the query plan cache is bounded (via hibernate.query.plan_cache_max_size set by default to 2048);
given the LIRS cache algorithm using by the query plan cache, calling clean up does not really remove the elements from the cache: it only invalidates the elements and they will be replaced by others when new objects will be added to the cache.
So the important point here is that we don't have any memory leak. I will add a comment in the source code explaining what you can expect from the query plan cache cleanup method.
As for Ranjith issue, I think it's a matter of sizing the cache as he has a lot of HQL queries from different applications.
Note that I suspect the HQLQueryPlan objects are significantly bigger than what we would have liked and I posted something to the hibernate-dev mailing list to discuss this subject.
Thank you Guillaume Smet and Sanne Grinovero . I may not have enough memory to keep 2048 cache plans for all applications. Each cache plan roughly takes 0.6MB.
I will limit cache the size for each application and proceed.