Orbeon 3.9 lock() leak in XFormsStateManager?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Orbeon 3.9 lock() leak in XFormsStateManager?

bwallis42
We have a problem under high load conditions (performance testing) that is causing a number of threads to build up in the Orbeon server. The stack trace below shows where all of these threads are sitting (we have around 40 of them at the moment).

These threads seem to be permanently stuck on a lock.

I was wondering if this has been seen before.

I think the problem is occurring due to a small window in the code. The lock is taken here in the call to beforeUpdate() and then released in the call to afterUpdate() in the finally block.

XFormsServer.java: 154
        final XFormsContainingDocument containingDocument = XFormsStateManager.instance().beforeUpdate(parameters);
        boolean keepDocument = false;
        Callable<SubmissionResult> replaceAllCallable = null;
        try {
...
        } finally {
            // Make sure to call this to release the lock
            XFormsStateManager.instance().afterUpdate(containingDocument, keepDocument);
        }

If we have a look at the beforeUpdate() method in XFormsStateManager we see that after the lock is taken there is a call to findOrRestoreDocument() and I think this is where the problem occurs. If the call to this throws an exception then that will cause the calling code to fail and since the call to beforeUpdate() is not inside the try block the finally block is not used and the lock is not released. I have also seen other errors in our server persistence implementation where under high load it fails to return the xform document and instead returns a 500 Server Error response. The number of these errors seems to be similar in number to the number of stuck threads that we are seeing (but not quite the same which may be due to having lost some of the logging messages).

    public XFormsContainingDocument beforeUpdate(RequestParameters parameters) {

        assert parameters.getUUID() != null;

        // Check that the session is associated with the requested UUID. This enforces the rule that an incoming request
        // for a given UUID must belong to the same session that created the document. If the session expires, the
        // key goes away as well, and the key won't be present. If we don't do this check, the XForms server might
        // handle requests for a given UUID within a separate session, therefore providing access to other sessions,
        // which is not desirable. Further, we now have a lock stored in the session.
        final Lock lock = getDocumentLock(parameters.getUUID());
        if (lock == null)
            throw new OXFException("Session has expired. Unable to process incoming request.");

        // Lock document
        lock.lock();

        // We got the lock, return the document

        return findOrRestoreDocument(parameters, false);
    }

"ajp-0.0.0.0-8209-93" daemon prio=10 tid=0x00007fb6ca0e8000 nid=0x203e waiting on condition [0x00007fb6c74f0000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x0000000781ed02f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
   at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
   at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
   at org.orbeon.oxf.xforms.state.XFormsStateManager.beforeUpdate(XFormsStateManager.java:308)
   at org.orbeon.oxf.xforms.processor.XFormsServer.doIt(XFormsServer.java:154)
   at org.orbeon.oxf.xforms.processor.XFormsServer.start(XFormsServer.java:100)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor$5.run(PipelineProcessor.java:661)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor.executeChildren(PipelineProcessor.java:726)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor.start(PipelineProcessor.java:658)
   at org.orbeon.oxf.processor.pipeline.choose.ConcreteChooseProcessor.start(ConcreteChooseProcessor.java:247)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor$5.run(PipelineProcessor.java:661)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor.executeChildren(PipelineProcessor.java:726)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor.start(PipelineProcessor.java:658)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor$5.run(PipelineProcessor.java:661)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor.executeChildren(PipelineProcessor.java:726)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor.start(PipelineProcessor.java:658)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor$1.getInput(PipelineProcessor.java:142)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor$1.readImpl(PipelineProcessor.java:85)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl$TopLevelOutputFilter.read(ProcessorOutputImpl.java:263)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl.read(ProcessorOutputImpl.java:406)
   at org.orbeon.oxf.processor.pipeline.choose.ConcreteChooseProcessor$1.readImpl(ConcreteChooseProcessor.java:126)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl$TopLevelOutputFilter.read(ProcessorOutputImpl.java:263)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl.read(ProcessorOutputImpl.java:406)
   at org.orbeon.oxf.processor.ProcessorImpl.readInputAsSAX(ProcessorImpl.java:260)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor$1$1.run(PipelineProcessor.java:94)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor.executeChildren(PipelineProcessor.java:726)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor.access$000(PipelineProcessor.java:61)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor$1.readImpl(PipelineProcessor.java:92)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl$TopLevelOutputFilter.read(ProcessorOutputImpl.java:263)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl.read(ProcessorOutputImpl.java:406)
   at org.orbeon.oxf.processor.ProcessorImpl.readInputAsSAX(ProcessorImpl.java:260)
   at org.orbeon.oxf.processor.pipeline.TeeProcessor$TeeProcessorOutputImpl.readImpl(TeeProcessor.java:89)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl$TopLevelOutputFilter.read(ProcessorOutputImpl.java:263)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl.read(ProcessorOutputImpl.java:406)
   at org.orbeon.oxf.processor.ProcessorImpl.readInputAsSAX(ProcessorImpl.java:260)
   at org.orbeon.oxf.processor.ProcessorImpl.readInputAsTinyTree(ProcessorImpl.java:286)
   at org.orbeon.oxf.processor.ProcessorImpl$3.read(ProcessorImpl.java:315)
   at org.orbeon.oxf.processor.ProcessorImpl.readCacheInputAsObject(ProcessorImpl.java:365)
   at org.orbeon.oxf.processor.ProcessorImpl.readCacheInputAsObject(ProcessorImpl.java:330)
   at org.orbeon.oxf.processor.ProcessorImpl.readCacheInputAsTinyTree(ProcessorImpl.java:313)
   at org.orbeon.oxf.processor.pipeline.choose.ConcreteChooseProcessor.start(ConcreteChooseProcessor.java:185)
   at org.orbeon.oxf.processor.pipeline.choose.ConcreteChooseProcessor$1.readImpl(ConcreteChooseProcessor.java:124)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl$TopLevelOutputFilter.read(ProcessorOutputImpl.java:263)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl.read(ProcessorOutputImpl.java:406)
   at org.orbeon.oxf.processor.ProcessorImpl.readInputAsSAX(ProcessorImpl.java:260)
   at org.orbeon.oxf.processor.ProcessorImpl.readInputAsTinyTree(ProcessorImpl.java:286)
   at org.orbeon.oxf.processor.ProcessorImpl$3.read(ProcessorImpl.java:315)
   at org.orbeon.oxf.processor.ProcessorImpl.readCacheInputAsObject(ProcessorImpl.java:365)
   at org.orbeon.oxf.processor.ProcessorImpl.readCacheInputAsObject(ProcessorImpl.java:330)
   at org.orbeon.oxf.processor.ProcessorImpl.readCacheInputAsTinyTree(ProcessorImpl.java:313)
   at org.orbeon.oxf.processor.pipeline.choose.ConcreteChooseProcessor.start(ConcreteChooseProcessor.java:185)
   at org.orbeon.oxf.processor.pipeline.choose.ConcreteChooseProcessor$1.readImpl(ConcreteChooseProcessor.java:124)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl$TopLevelOutputFilter.read(ProcessorOutputImpl.java:263)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl.read(ProcessorOutputImpl.java:406)
   at org.orbeon.oxf.processor.ProcessorImpl.readInputAsSAX(ProcessorImpl.java:260)
   at org.orbeon.oxf.processor.ProcessorImpl.readInputAsSAX(ProcessorImpl.java:264)
   at org.orbeon.oxf.processor.IdentityProcessor$1.readImpl(IdentityProcessor.java:34)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl$TopLevelOutputFilter.read(ProcessorOutputImpl.java:263)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl.read(ProcessorOutputImpl.java:406)
   at org.orbeon.oxf.processor.ProcessorImpl.readInputAsSAX(ProcessorImpl.java:260)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor$1$1.run(PipelineProcessor.java:94)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor.executeChildren(PipelineProcessor.java:726)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor.access$000(PipelineProcessor.java:61)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor$1.readImpl(PipelineProcessor.java:92)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl$TopLevelOutputFilter.read(ProcessorOutputImpl.java:263)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl.read(ProcessorOutputImpl.java:406)
   at org.orbeon.oxf.processor.pipeline.choose.ConcreteChooseProcessor$1.readImpl(ConcreteChooseProcessor.java:126)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl$TopLevelOutputFilter.read(ProcessorOutputImpl.java:263)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl.read(ProcessorOutputImpl.java:406)
   at org.orbeon.oxf.processor.ProcessorImpl.readInputAsSAX(ProcessorImpl.java:260)
   at org.orbeon.oxf.processor.ProcessorImpl.readInputAsSAX(ProcessorImpl.java:264)
   at org.orbeon.oxf.processor.IdentityProcessor$1.readImpl(IdentityProcessor.java:34)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl$TopLevelOutputFilter.read(ProcessorOutputImpl.java:263)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl.read(ProcessorOutputImpl.java:406)
   at org.orbeon.oxf.processor.ProcessorImpl.readInputAsSAX(ProcessorImpl.java:260)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor$1$1.run(PipelineProcessor.java:94)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor.executeChildren(PipelineProcessor.java:726)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor.access$000(PipelineProcessor.java:61)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor$1.readImpl(PipelineProcessor.java:92)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl$TopLevelOutputFilter.read(ProcessorOutputImpl.java:263)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl.read(ProcessorOutputImpl.java:406)
   at org.orbeon.oxf.processor.pipeline.choose.ConcreteChooseProcessor$1.readImpl(ConcreteChooseProcessor.java:126)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl$TopLevelOutputFilter.read(ProcessorOutputImpl.java:263)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl.read(ProcessorOutputImpl.java:406)
   at org.orbeon.oxf.processor.ProcessorImpl.readInputAsSAX(ProcessorImpl.java:260)
   at org.orbeon.oxf.processor.ProcessorImpl.readInputAsSAX(ProcessorImpl.java:264)
   at org.orbeon.oxf.processor.IdentityProcessor$1.readImpl(IdentityProcessor.java:34)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl$TopLevelOutputFilter.read(ProcessorOutputImpl.java:263)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl.read(ProcessorOutputImpl.java:406)
   at org.orbeon.oxf.processor.ProcessorImpl.readInputAsSAX(ProcessorImpl.java:260)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor$1$1.run(PipelineProcessor.java:94)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor.executeChildren(PipelineProcessor.java:726)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor.access$000(PipelineProcessor.java:61)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor$1.readImpl(PipelineProcessor.java:92)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl$TopLevelOutputFilter.read(ProcessorOutputImpl.java:263)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl.read(ProcessorOutputImpl.java:406)
   at org.orbeon.oxf.processor.pipeline.choose.ConcreteChooseProcessor$1.readImpl(ConcreteChooseProcessor.java:126)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl$TopLevelOutputFilter.read(ProcessorOutputImpl.java:263)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl.read(ProcessorOutputImpl.java:406)
   at org.orbeon.oxf.processor.ProcessorImpl.readInputAsSAX(ProcessorImpl.java:260)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor$1$1.run(PipelineProcessor.java:94)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor.executeChildren(PipelineProcessor.java:726)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor.access$000(PipelineProcessor.java:61)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor$1.readImpl(PipelineProcessor.java:92)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl$TopLevelOutputFilter.read(ProcessorOutputImpl.java:263)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl.read(ProcessorOutputImpl.java:406)
   at org.orbeon.oxf.processor.pipeline.choose.ConcreteChooseProcessor$1.readImpl(ConcreteChooseProcessor.java:126)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl$TopLevelOutputFilter.read(ProcessorOutputImpl.java:263)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl.read(ProcessorOutputImpl.java:406)
   at org.orbeon.oxf.processor.ProcessorImpl.readInputAsSAX(ProcessorImpl.java:260)
   at org.orbeon.oxf.processor.ProcessorImpl.readInputAsSAX(ProcessorImpl.java:264)
   at org.orbeon.oxf.processor.IdentityProcessor$1.readImpl(IdentityProcessor.java:34)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl$TopLevelOutputFilter.read(ProcessorOutputImpl.java:263)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl.read(ProcessorOutputImpl.java:406)
   at org.orbeon.oxf.processor.ProcessorImpl.readInputAsSAX(ProcessorImpl.java:260)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor$1$1.run(PipelineProcessor.java:94)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor.executeChildren(PipelineProcessor.java:726)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor.access$000(PipelineProcessor.java:61)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor$1.readImpl(PipelineProcessor.java:92)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl$TopLevelOutputFilter.read(ProcessorOutputImpl.java:263)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl.read(ProcessorOutputImpl.java:406)
   at org.orbeon.oxf.processor.pipeline.choose.ConcreteChooseProcessor$1.readImpl(ConcreteChooseProcessor.java:126)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl$TopLevelOutputFilter.read(ProcessorOutputImpl.java:263)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl.read(ProcessorOutputImpl.java:406)
   at org.orbeon.oxf.processor.ProcessorImpl.readInputAsSAX(ProcessorImpl.java:260)
   at org.orbeon.oxf.processor.pipeline.TeeProcessor$TeeProcessorOutputImpl.readImpl(TeeProcessor.java:89)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl$TopLevelOutputFilter.read(ProcessorOutputImpl.java:263)
   at org.orbeon.oxf.processor.impl.ProcessorOutputImpl.read(ProcessorOutputImpl.java:406)
   at org.orbeon.oxf.processor.ProcessorImpl.readInputAsSAX(ProcessorImpl.java:260)
   at org.orbeon.oxf.processor.ProcessorImpl.readInputAsTinyTree(ProcessorImpl.java:286)
   at org.orbeon.oxf.processor.ProcessorImpl$3.read(ProcessorImpl.java:315)
   at org.orbeon.oxf.processor.ProcessorImpl.readCacheInputAsObject(ProcessorImpl.java:365)
   at org.orbeon.oxf.processor.ProcessorImpl.readCacheInputAsObject(ProcessorImpl.java:330)
   at org.orbeon.oxf.processor.ProcessorImpl.readCacheInputAsTinyTree(ProcessorImpl.java:313)
   at org.orbeon.oxf.processor.pipeline.choose.ConcreteChooseProcessor.start(ConcreteChooseProcessor.java:185)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor$5.run(PipelineProcessor.java:661)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor.executeChildren(PipelineProcessor.java:726)
   at org.orbeon.oxf.processor.pipeline.PipelineProcessor.start(PipelineProcessor.java:658)
   at org.orbeon.oxf.processor.PageFlowControllerProcessor.start(PageFlowControllerProcessor.java:476)
   at org.orbeon.oxf.pipeline.InitUtils.runProcessor(InitUtils.java:99)
   at org.orbeon.oxf.webapp.ProcessorService.service(ProcessorService.java:97)
   at org.orbeon.oxf.servlet.OrbeonServletDelegate.service(OrbeonServletDelegate.java:133)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:324)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242)
   at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:734)
   at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:541)
   at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:479)
   at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:407)
   at org.orbeon.oxf.servlet.OrbeonXFormsFilter.doFilter(OrbeonXFormsFilter.java:83)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:274)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242)
   at au.com.infomedix.cpflogin.CpfSSOSessionIdFilter.doFilter(CpfSSOSessionIdFilter.java:68)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:274)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242)
   at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
   at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
   at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:181)
   at org.jboss.modcluster.catalina.CatalinaContext$RequestListenerValve.event(CatalinaContext.java:285)
   at org.jboss.modcluster.catalina.CatalinaContext$RequestListenerValve.invoke(CatalinaContext.java:261)
   at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:593)
   at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:88)
   at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:100)
   at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:159)
   at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
   at org.jboss.web.tomcat.service.sso.ClusteredSingleSignOn.invoke(ClusteredSingleSignOn.java:735)
   at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
   at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
   at org.jboss.web.tomcat.service.request.ActiveRequestResponseCacheValve.invoke(ActiveRequestResponseCacheValve.java:53)
   at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:362)
   at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:504)
   at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:437)
   at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:951)
   at java.lang.Thread.run(Thread.java:662)
Reply | Threaded
Open this post in threaded view
|

Re: Orbeon 3.9 lock() leak in XFormsStateManager?

bwallis42
I've done some more investigation on this and the error is not caused by our server error in the persistence it is caused by entries in the forms state cache expiring.

beforeUpdate()
  calls lock()
  calls findOrRestoreDocument()
    // if the document isn't cached then
    calls createDocumentFromStore()
      // if this then doesn't find the state in the statestore
        throws OXFException "Unable to retrieve XForms engine state. Please reload the current page. Note that you will lose any unsaved changes."
        OR
        throws OXFException "Your session has expired. Please reload the current page. Note that you will lose any unsaved changes."

and it is this exception that is causing the unlock to be bypassed. We are seeing these exceptions in the log.
Reply | Threaded
Open this post in threaded view
|

Re: Orbeon 3.9 lock() leak in XFormsStateManager?

bwallis42
In reply to this post by bwallis42
So, the patch we are going with into testing today is as follows. This makes sure that if there is an error in the call to findOrRestoreDocument() then we unlock the document again before re-throwing the exception. This is in XFormsStateManager.java.

Does this fix make sense? Seems right to me.

thanks.

    public XFormsContainingDocument beforeUpdate(RequestParameters parameters) {

        assert parameters.getUUID() != null;

        // Check that the session is associated with the requested UUID. This enforces the rule that an incoming request
        // for a given UUID must belong to the same session that created the document. If the session expires, the
        // key goes away as well, and the key won't be present. If we don't do this check, the XForms server might
        // handle requests for a given UUID within a separate session, therefore providing access to other sessions,
        // which is not desirable. Further, we now have a lock stored in the session.
        final Lock lock = getDocumentLock(parameters.getUUID());
        if (lock == null)
            throw new OXFException("Session has expired. Unable to process incoming request.");

        // Lock document
        lock.lock();

        // We got the lock, return the document
        try
        {
            return findOrRestoreDocument(parameters, false);
        }
        catch(RuntimeException e)
        {
            // don't be caught out with a permanent lock! We can occasionally get an OXFException thrown
            // from XFormsStateManager.findOrRestoreDocument() courtesy of XFormsStateManager.createDocumentFromStore()
            lock.unlock();
            throw e;
        }
    }
Reply | Threaded
Open this post in threaded view
|

Re: Orbeon 3.9 lock() leak in XFormsStateManager?

bwallis42
I see that this code has changed in version 4 (looking at 4.9 source) where the lock acquisition has been separated from the call to findOrRestoreDocument() which should prevent this issue happening in that version.

On some further searching I see this is fix #534 (https://github.com/orbeon/orbeon-forms/issues/534) in version 4.
Reply | Threaded
Open this post in threaded view
|

Re: Orbeon 3.9 lock() leak in XFormsStateManager?

Erik Bruchez
Administrator
Brian,

Yes I think you found the issue. I am glad you are reaching similar conclusions.

There is a bit of subtlety in deciding in which case the lock should be released. There are at least 2 options:

1. using `finally`, assuming even in the case of a fatal VM error that we can try to release the lock (but is it worth it?)
2. checking on specific exceptions, as I think you are doing here by checking `RuntimeException`

I think `RuntimeException` is not great, because there can be non-runtime exceptions thrown, as well as errors. Scala has a `NonFatal` extractor to filter out really bad errors:

    https://github.com/scala/scala/blob/v2.11.5/src/library/scala/util/control/NonFatal.scala

-Erik