I think it could be a good thing to allow an experienced user to have something like an referential cache (cache the actual POJO) in some situations.
In some applications it's okay to have not so up-to-date versions of objects in a query result list (i.e. an application that checks the database for changes every hour or so). For these applications the object/hydratation creation process of the Hibernate Cache could be overkill and a POJO cache would suffice.
So why shouldn't Hibernate Search enable users to have their own cache management of these POJOs? I have looked into the code of 4.5 (as this is the version I am using) and have seen that the whole Object retrieval stuff is centralized in one class in HSEARCH (ObjectLoaderHelper) and I wrote a quick and dirty javaagent/javassist hack that wraps the loading into my own caching mechanism (https://github.com/Hotware/HibernateSearchReferenceCache/blob/master/src/main/java/de/hotware/hibernate/search/extension/reference/bytecode/ObjectLoaderTransformer.java) as a proof of concept.
I understand that Referential caching can be harmful, but it could be a neat feature for the experienced user.