Hi,
We have an urgent issue in production. When the application is under load requests to generate the xform sometimes do not come back and the threads associated with those request keep on waiting for a lock monitor to become available. I am still digging but it appears there may be an issue with MemoryCacheImpl. I looked at the source code and no changes have been made to this class in a while and hence I am not sure why we would be the only ones with the issue. Looking at the class it does look like a couple of methods are not threadsafe (findAny, getMax etc....) but I am really not sure if this is really the issue. Any thoughts out there? Here is a thread that seems to hang on to a lock: "http-172.17.0.66-1443-46" daemon prio=10 tid=0x00002aab4c80d800 nid=0x643d waiting for monitor entry [0x000000004a9a7000] java.lang.Thread.State: BLOCKED (on object monitor) at org.orbeon.oxf.cache.MemoryCacheImpl.findValid(MemoryCacheImpl.java:124) - locked <0x00002aaabea27c88> (a org.orbeon.oxf.cache.MemoryCacheImpl) at org.orbeon.oxf.util.XPathCache.getXPathExpression(XPathCache.java:367) at org.orbeon.oxf.util.XPathCache.getXPathExpression(XPathCache.java:276) at org.orbeon.oxf.processor.pipeline.choose.ConcreteChooseProcessor.start(ConcreteChooseProcessor.java:193) at org.orbeon.oxf.processor.pipeline.PipelineProcessor$5.run(PipelineProcessor.java:647) at org.orbeon.oxf.processor.ProcessorImpl.executeChildren(ProcessorImpl.java:526) at org.orbeon.oxf.processor.pipeline.PipelineProcessor.start(PipelineProcessor.java:644) at org.orbeon.oxf.pipeline.InitUtils.runProcessor(InitUtils.java:90) at org.orbeon.oxf.webapp.ProcessorService.service(ProcessorService.java:100) at org.orbeon.oxf.servlet.OrbeonServletDelegate.service(OrbeonServletDelegate.java:138) at javax.servlet.http.HttpServlet.service(HttpServlet.java:810) ....... Here is one of 50 typical threads that is waiting forever to get that lock: "http-172.17.0.66-1443-258" daemon prio=10 tid=0x000000005125b000 nid=0x66d0 waiting for monitor entry [0x0000000068985000] java.lang.Thread.State: BLOCKED (on object monitor) at org.orbeon.oxf.cache.MemoryCacheImpl.findValid(MemoryCacheImpl.java:124) - waiting to lock <0x00002aaabea27c88> (a org.orbeon.oxf.cache.MemoryCacheImpl) at org.orbeon.oxf.util.XPathCache.getXPathExpression(XPathCache.java:367) at org.orbeon.oxf.util.XPathCache.getXPathExpression(XPathCache.java:276) at org.orbeon.oxf.processor.pipeline.choose.ConcreteChooseProcessor.start(ConcreteChooseProcessor.java:193) at org.orbeon.oxf.processor.pipeline.PipelineProcessor$5.run(PipelineProcessor.java:647) at org.orbeon.oxf.processor.ProcessorImpl.executeChildren(ProcessorImpl.java:526) at org.orbeon.oxf.processor.pipeline.PipelineProcessor.start(PipelineProcessor.java:644) at org.orbeon.oxf.pipeline.InitUtils.runProcessor(InitUtils.java:90) at org.orbeon.oxf.webapp.ProcessorService.service(ProcessorService.java:100) at org.orbeon.oxf.servlet.OrbeonServletDelegate.service(OrbeonServletDelegate.java:138) at javax.servlet.http.HttpServlet.service(HttpServlet.java:810) |
Administrator
|
(FYI we are taking this issue offline.)
-Erik On Wed, Aug 25, 2010 at 1:25 PM, npujol73 <[hidden email]> wrote: > > Hi, > > We have an urgent issue in production. When the application is under load > requests to generate the xform sometimes do not come back and the threads > associated with those request keep on waiting for a lock monitor to become > available. I am still digging but it appears there may be an issue with > MemoryCacheImpl. I looked at the source code and no changes have been made > to this class in a while and hence I am not sure why we would be the only > ones with the issue. Looking at the class it does look like a couple of > methods are not threadsafe (findAny, getMax etc....) but I am really not > sure if this is really the issue. Any thoughts out there? > > Here is a thread that seems to hang on to a lock: > > "http-172.17.0.66-1443-46" daemon prio=10 tid=0x00002aab4c80d800 nid=0x643d > waiting for monitor entry [0x000000004a9a7000] > java.lang.Thread.State: BLOCKED (on object monitor) > at org.orbeon.oxf.cache.MemoryCacheImpl.findValid(MemoryCacheImpl.java:124) > - locked <0x00002aaabea27c88> (a org.orbeon.oxf.cache.MemoryCacheImpl) > at org.orbeon.oxf.util.XPathCache.getXPathExpression(XPathCache.java:367) > at org.orbeon.oxf.util.XPathCache.getXPathExpression(XPathCache.java:276) > at > org.orbeon.oxf.processor.pipeline.choose.ConcreteChooseProcessor.start(ConcreteChooseProcessor.java:193) > at > org.orbeon.oxf.processor.pipeline.PipelineProcessor$5.run(PipelineProcessor.java:647) > at > org.orbeon.oxf.processor.ProcessorImpl.executeChildren(ProcessorImpl.java:526) > at > org.orbeon.oxf.processor.pipeline.PipelineProcessor.start(PipelineProcessor.java:644) > at org.orbeon.oxf.pipeline.InitUtils.runProcessor(InitUtils.java:90) > at > org.orbeon.oxf.webapp.ProcessorService.service(ProcessorService.java:100) > at > org.orbeon.oxf.servlet.OrbeonServletDelegate.service(OrbeonServletDelegate.java:138) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:810) > ....... > Here is one of 50 typical threads that is waiting forever to get that lock: > > "http-172.17.0.66-1443-258" daemon prio=10 tid=0x000000005125b000 nid=0x66d0 > waiting for monitor entry [0x0000000068985000] > java.lang.Thread.State: BLOCKED (on object monitor) > at org.orbeon.oxf.cache.MemoryCacheImpl.findValid(MemoryCacheImpl.java:124) > - waiting to lock <0x00002aaabea27c88> (a > org.orbeon.oxf.cache.MemoryCacheImpl) > at org.orbeon.oxf.util.XPathCache.getXPathExpression(XPathCache.java:367) > at org.orbeon.oxf.util.XPathCache.getXPathExpression(XPathCache.java:276) > at > org.orbeon.oxf.processor.pipeline.choose.ConcreteChooseProcessor.start(ConcreteChooseProcessor.java:193) > at > org.orbeon.oxf.processor.pipeline.PipelineProcessor$5.run(PipelineProcessor.java:647) > at > org.orbeon.oxf.processor.ProcessorImpl.executeChildren(ProcessorImpl.java:526) > at > org.orbeon.oxf.processor.pipeline.PipelineProcessor.start(PipelineProcessor.java:644) > at org.orbeon.oxf.pipeline.InitUtils.runProcessor(InitUtils.java:90) > at > org.orbeon.oxf.webapp.ProcessorService.service(ProcessorService.java:100) > at > org.orbeon.oxf.servlet.OrbeonServletDelegate.service(OrbeonServletDelegate.java:138) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:810) > > -- > View this message in context: http://orbeon-forms-ops-users.24843.n4.nabble.com/Unusually-large-number-of-blocking-threads-tp2338846p2338846.html > Sent from the Orbeon Forms (ops-users) mailing list archive at Nabble.com. > > > -- > You receive this message as a subscriber of the [hidden email] mailing list. > To unsubscribe: mailto:[hidden email] > For general help: mailto:[hidden email]?subject=help > OW2 mailing lists service home page: http://www.ow2.org/wws > > -- You receive this message as a subscriber of the [hidden email] mailing list. To unsubscribe: mailto:[hidden email] For general help: mailto:[hidden email]?subject=help OW2 mailing lists service home page: http://www.ow2.org/wws |
Free forum by Nabble | Edit this page |