Hibernate 5.x JPA isn't working with WebSphere Application Server Liberty because Hibernate isn't aware of the transaction manager API, com.ibm.tx.jta.TransactionManagerFactory, and thus fails when attempting operations for which it needs a transaction manager, such as suspend/resume/setRollbackOnly, and so on. A simple way to reproduce is to specify a persistence unit with a jta-data-source which specifies auto-create of tables, for example, via javax.persistence.schema-generation.database.action=create. Hibernate wants to use the transaction manager to issue a suspend and is unable to find a JtaPlatform. If you point it at a built-in ones for WebSphere, it tries to use interfaces from old versions of WebSphere Application Server that aren't available in Liberty.
A customer can work around this by providing their own JtaPlatform implementation that redirects to com.ibm.tx.jta.TransactionManagerFactory. In fact, we have suggested this exact approach to customers on stack overflow (see the following link which has a an example, http://stackoverflow.com/questions/41872159/is-it-possible-to-use-hibernate-5-as-jpa-engine-in-was-liberty-profile), but it is too burdensome for many customers, and so we would like to officially request that Hibernate add a built-in JtaPlatform implementation for Liberty, such as the one listed in the stack overflow post.
Hibernate 5.x, WebSphere Application Server Liberty 188.8.131.52