The current OrionTransactionManagerLookup, shipped with Hibernate, indicates that the transaction manager is published at JNDI namespace 'java:comp/UserTransaction'. This was the case for OC4J 903 - 904 but Oracle changed the location in OC4J 10G. The new (and hidden) location is available in the uploaded OC4JTransactionManagerLookup class which have been succesfully tested on OC4J 903, 904 and 10G.
PS: please email me if I can contribute to your project. My expertise is EJB3 (EntityManager) and Oracle App Server.
Oracle App Server (Orion based)