Custom Scanner throws ClassCast when loaded from TempClassLoader

We have built an internal Scanner to add classes from external dependencies based on a meta file.

This used to work on Hibernate 4.3.8.Final but we are updating to the latest version (5.1.0.Final) and it throws an exception in WildFly because of TempClassLoader.

Caused by: java.lang.ClassCastException: pkg.org.hibernate.jpa.test.packaging.temp_classloader.CustomAutoScanner cannot be cast to org.hibernate.boot.archive.scan.spi.Scanner
at org.hibernate.boot.model.process.internal.ScanningCoordinator.buildScanner(ScanningCoordinator.java:165)
... 20 more

I've create the scenario at my fork https://github.com/Janario/hibernate-orm/commit/6dda1fd07ddd5346b19c7182d991b8e36a11ff86
Custom scanner https://github.com/Janario/hibernate-orm/commit/6dda1fd07ddd5346b19c7182d991b8e36a11ff86#diff-ce09751ef9dbaa28b0acffe22efd376fR20

TempClassLoader (similar to wildfly-jpa https://github.com/wildfly/wildfly/blob/master/jpa/subsystem/src/main/java/org/jboss/as/jpa/classloader/TempClassLoader.java#L64)
https://github.com/Janario/hibernate-orm/commit/6dda1fd07ddd5346b19c7182d991b8e36a11ff86#diff-5e427fcaa86edb0e68fc03e4cfadff1fR10

As solution I have removed the tempClassLoader once it would fail with any instance returned.
Also the Hibernate Scanner doesn't load the entities classes, it reads as binary with javassist.
https://github.com/Janario/hibernate-orm/commit/67732206b25ba2ededf9466820e1db6b9193e53d#diff-2f4dcf938d3a977a4f6d23232a16d7e7L68