Infinispan listener implementations need to load entities and keys using application classloader.

Description

While testing hibernate-infinispan, I see ClassNotFoundExceptions when the jgroups receiver thread attempts to deserialize an entity PK (in this case within the BaseRegion cache listener). Deserialization throw a CNFE since the custom key is not known to the context classloader of the jgroups up handler. It would appear that lazy deserialization is not sufficient to prevent this. Consequently, I think we need to reproduce the classloader handling from the jbosscache provider. That is, store a reference to the application classloader and swap thread context classloaders any time a key or an entity needs to be deserialized.

2010-09-10 13:46:53,339 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (Incoming-2,null) Execution error: : org.infinispan.CacheException: Caught exception invoking method public void org.hibernate.cache.infinispan.impl.BaseRegion.entryModified(org.infinispan.notifications.cachelistener.event.CacheEntryModifiedEvent) on listener instance org.hibernate.cache.infinispan.query.QueryResultsRegionImpl@70099d73
at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocation$1.run(AbstractListenerImpl.java:179) [:4.1.1.CR0]
at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:47) [:4.1.1.CR0]
at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocation.invoke(AbstractListenerImpl.java:189) [:4.1.1.CR0]
at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyCacheEntryModified(CacheNotifierImpl.java:146) [:4.1.1.CR0]
at org.infinispan.commands.write.PutKeyValueCommand.perform(PutKeyValueCommand.java:108) [:4.1.1.CR0]
at org.infinispan.interceptors.CallInterceptor.handleDefault(CallInterceptor.java:72) [:4.1.1.CR0]
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57) [:4.1.1.CR0]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76) [:4.1.1.CR0]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [:4.1.1.CR0]
at org.infinispan.interceptors.ReplicationInterceptor.handleCrudMethod(ReplicationInterceptor.java:107) [:4.1.1.CR0]
at org.infinispan.interceptors.ReplicationInterceptor.visitPutKeyValueCommand(ReplicationInterceptor.java:78) [:4.1.1.CR0]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76) [:4.1.1.CR0]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [:4.1.1.CR0]
at org.infinispan.interceptors.LockingInterceptor.visitPutKeyValueCommand(LockingInterceptor.java:198) [:4.1.1.CR0]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76) [:4.1.1.CR0]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [:4.1.1.CR0]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [:4.1.1.CR0]
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57) [:4.1.1.CR0]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76) [:4.1.1.CR0]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [:4.1.1.CR0]
at org.infinispan.interceptors.MarshalledValueInterceptor.visitPutKeyValueCommand(MarshalledValueInterceptor.java:116) [:4.1.1.CR0]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76) [:4.1.1.CR0]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [:4.1.1.CR0]
at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:82) [:4.1.1.CR0]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:120) [:4.1.1.CR0]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [:4.1.1.CR0]
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:57) [:4.1.1.CR0]
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:38) [:4.1.1.CR0]
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:105) [:4.1.1.CR0]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:120) [:4.1.1.CR0]
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:273) [:4.1.1.CR0]
at org.infinispan.commands.tx.PrepareCommand.perform(PrepareCommand.java:111) [:4.1.1.CR0]
at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:81) [:4.1.1.CR0]
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:176) [:4.1.1.CR0]
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:148) [:4.1.1.CR0]
at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:575) [:2.10.0.GA]
at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:486) [:2.10.0.GA]
at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:362) [:2.10.0.GA]
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:771) [:2.10.0.GA]
at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:136) [:2.10.0.GA]
at org.jgroups.JChannel.up(JChannel.java:1453) [:2.10.0.GA]
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:887) [:2.10.0.GA]
at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:483) [:2.10.0.GA]
at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:265) [:2.10.0.GA]
at org.jgroups.protocols.FRAG2.up(FRAG2.java:188) [:2.10.0.GA]
at org.jgroups.protocols.FC.up(FC.java:474) [:2.10.0.GA]
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:888) [:2.10.0.GA]
at org.jgroups.protocols.VIEW_SYNC.up(VIEW_SYNC.java:171) [:2.10.0.GA]
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234) [:2.10.0.GA]
at org.jgroups.protocols.UNICAST.up(UNICAST.java:309) [:2.10.0.GA]
at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:877) [:2.10.0.GA]
at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:671) [:2.10.0.GA]
at org.jgroups.protocols.BARRIER.up(BARRIER.java:120) [:2.10.0.GA]
at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:132) [:2.10.0.GA]
at org.jgroups.protocols.FD.up(FD.java:266) [:2.10.0.GA]
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:270) [:2.10.0.GA]
at org.jgroups.protocols.MERGE2.up(MERGE2.java:210) [:2.10.0.GA]
at org.jgroups.protocols.Discovery.up(Discovery.java:281) [:2.10.0.GA]
at org.jgroups.protocols.PING.up(PING.java:67) [:2.10.0.GA]
at org.jgroups.stack.Protocol.up(Protocol.java:371) [:2.10.0.GA]
at org.jgroups.protocols.TP.passMessageUp(TP.java:1009) [:2.10.0.GA]
at org.jgroups.protocols.TP.access$100(TP.java:56) [:2.10.0.GA]
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1549) [:2.10.0.GA]
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1531) [:2.10.0.GA]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [:1.6.0_20]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [:1.6.0_20]
at java.lang.Thread.run(Thread.java:636) [:1.6.0_20]
Caused by: org.infinispan.CacheException: Unable to unmarshall value
at org.infinispan.marshall.MarshalledValue.deserialize(MarshalledValue.java:114) [:4.1.1.CR0]
at org.infinispan.marshall.MarshalledValue.get(MarshalledValue.java:167) [:4.1.1.CR0]
at org.infinispan.notifications.cachelistener.event.EventImpl.getKey(EventImpl.java:66) [:4.1.1.CR0]
at org.hibernate.cache.infinispan.impl.BaseRegion.handleEvictAllModification(BaseRegion.java:264) [:3.5.6-SNAPSHOT]
at org.hibernate.cache.infinispan.impl.BaseRegion.entryModified(BaseRegion.java:260) [:3.5.6-SNAPSHOT]
at sun.reflect.GeneratedMethodAccessor271.invoke(Unknown Source) [:1.6.0_20]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [:1.6.0_20]
at java.lang.reflect.Method.invoke(Method.java:616) [:1.6.0_20]
at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocation$1.run(AbstractListenerImpl.java:175) [:4.1.1.CR0]
... 66 more
Caused by: java.lang.ClassNotFoundException: org.jboss.test.cluster.clusteredentity.classloader.AccountHolderPK from BaseClassLoader@620968f9{vfs:///home/paul/jboss/as/infinispan-int/build/target/jboss-6.0.0-SNAPSHOT/server/cluster-udp-1/conf/jboss-service.xml}
at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:480) [jboss-classloader.jar:2.2.0.Alpha7]
at java.lang.ClassLoader.loadClass(ClassLoader.java:266) [:1.6.0_20]
at java.lang.Class.forName0(Native Method) [:1.6.0_20]
at java.lang.Class.forName(Class.java:264) [:1.6.0_20]
at org.jboss.marshalling.AbstractClassResolver.loadClass(AbstractClassResolver.java:135) [:]
at org.jboss.marshalling.AbstractClassResolver.resolveClass(AbstractClassResolver.java:116) [:]
at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:845) [:]
at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1247) [:]
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272) [:]
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [:]
at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1610) [:]
at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1526) [:]
at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1175) [:]
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272) [:]
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [:]
at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1610) [:]
at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1526) [:]
at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1175) [:]
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272) [:]
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [:]
at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:36) [:]
at org.infinispan.marshall.MarshallUtil.unmarshallMap(MarshallUtil.java:65) [:4.1.1.CR0]
at org.infinispan.marshall.exts.MapExternalizer.readObject(MapExternalizer.java:78) [:4.1.1.CR0]
at org.infinispan.marshall.jboss.ConstantObjectTable$ExternalizerAdapter.readObject(ConstantObjectTable.java:266) [:4.1.1.CR0]
at org.infinispan.marshall.jboss.ConstantObjectTable.readObject(ConstantObjectTable.java:253) [:4.1.1.CR0]
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351) [:]
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [:]
at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1610) [:]
at org.jboss.marshalling.river.RiverObjectInputStream.defaultReadObject(RiverObjectInputStream.java:73) [:]
at org.hibernate.cache.QueryKey.readObject(QueryKey.java:192) [:3.5.6-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [:1.6.0_20]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [:1.6.0_20]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [:1.6.0_20]
at java.lang.reflect.Method.invoke(Method.java:616) [:1.6.0_20]
at org.jboss.marshalling.reflect.SerializableClass.callReadObject(SerializableClass.java:224) [:]
at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1513) [:]
at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1175) [:]
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272) [:]
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [:]
at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:36) [:]
at org.infinispan.marshall.jboss.GenericJBossMarshaller.objectFromObjectStream(GenericJBossMarshaller.java:162) [:4.1.1.CR0]
at org.infinispan.marshall.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:111) [:4.1.1.CR0]
at org.infinispan.marshall.AbstractMarshaller.objectFromByteBuffer(AbstractMarshaller.java:49) [:4.1.1.CR0]
at org.infinispan.marshall.MarshalledValue.deserialize(MarshalledValue.java:111) [:4.1.1.CR0]
... 74 more

Environment

None

Activity

Show:
Paul Ferraro
September 15, 2010, 4:20 PM

Does the interceptor approach still make sense (i.e. the 1st patch)? Something like this will still needed in order to properly deserialize cache entries during RPC handling.

Galder Zamarreno
September 16, 2010, 8:18 AM

I'm not sure whether it's needed or not. For the general case not at all cos the idea is that cache entries are deserialized when client thread requests it and such thread is generally the app thread that has the correct classloader. However, once I built a UT for it yday, I found out that the update timestamps keeps a local cache out of listener event key/values, so we might need your patch at least temporarily.

Galder Zamarreno
September 16, 2010, 10:05 AM

Paul, I don't think ClassLoaderAwareCommandInterceptor is needed here. You only need listener calls to have the right classloader. In any other situation, the cache is accessed from the app classloader. Can you remove it and see if your tests work fine? If they don't, please attach any stacktrace you get.

Paul Ferraro
September 16, 2010, 5:31 PM

Updated patch that drops unnecessary interceptor, and with more complete event handling.

Steve Ebersole
March 21, 2011, 7:09 PM

Bulk closing stale resolved issues

Assignee

Galder Zamarreno

Reporter

Paul Ferraro

Fix versions

Labels

None

backPortable

None

Suitable for new contributors

None

Requires Release Note

None

Pull Request

None

backportDecision

None

Affects versions

Priority

Critical
Configure