Lock contention in Saxon's NamePool

> We're having some odd problems with deadlocks in Saxon's NamePool, and I
> was wondering if anyone here may have seen this behavior.
>
> Background: We're running OPS (er, OF) 3.0.1, which includes Saxon
> 8.1.1, on Linux. Our application is NOT a "conventional" OF/XForms app;
> rather, we're using OF's pipeline engine with our own pipelines and XSLT
> to generate XHTML and WML for mobile web browsers. You can see the app
> at http://xhtml.weather.com/xhtml/ (for XHTML) or
> http://wml.weather.com/xhtml/ (for WML).
>
> We've generally been very happy with the performance of OF; we've seen
> days with a million pageviews or more, with good stability and response
> time.
>
> However, in the last week or so, we've seen our response time spike
> under high loads. When we get a thread dump of the JVM (via "kill -3"),
> we see that many of our listener threads are getting hung up in Saxon's
> NamePool. Specifically, our stack traces look like this:
>
> "TP-Processor55" daemon prio=1 tid=0x088fbe68 nid=0x7572 waiting for
> monitor entry [0x7ec30000..0x7ec31700]
>     at org.orbeon.saxon.om.NamePool.allocate(NamePool.java:384)
>     - waiting to lock <0x4cde0260> (a org.orbeon.saxon.om.NamePool)
>     at org.orbeon.saxon.tree.NamespaceImpl.<init>(NamespaceImpl.java:35)
>     at
> org.orbeon.saxon.tree.ElementWithAttributes.addNamespaceNodes(ElementWithAttributes.java:191)
>
>     at
> org.orbeon.saxon.tree.ElementWithAttributes.getNamespaceCodes(ElementWithAttributes.java:227)
>
>     at
> org.orbeon.saxon.style.LiteralResultElement.validate(LiteralResultElement.java:214)
>
> (etc...)
>
> The odd thing is that all the threads take a different route to get to
> this point; they all hang on NamePool.allocate(), but the rest of the
> stack trace is different, showing that the threads were all at different
> points of generating their responses when they hit the deadlock.
>
> We can't associate the first appearance of these problems with any
> particular change in our application code or server config; they seem to
> have cropped up out of nowhere.
>
> Can anyone shed any light on this? We'd be eternally grateful.
>
> (I'll also be posting this question to the Saxon forum, but since we're
> dealing with Orbeon's rewritten Saxon processor here, it seems
> reasonable to ask here.)
>
> Thanks,
>
> Jeff Jones
> Sr Internet App Devel
> The Weather Channel Interactive
>

> Jeff,
>
> I do seem to remember that Saxon at some point became much slower due to
> synchronization in NamePool with Java 1.5, to the point where it was
> several times slower than with 1.4. But that was a long time ago, and
> when we finally migrated to Saxon 8.x, that issue seemed to have
> disappeared.
>
> Also note that we are using Saxon 8.8 in the latest versions of Orbeon
> Forms.
>
> What JDK are you using? Is it an option to try another JDK, or a more
> recent version of Orbeon Forms?
>
> -Erik
>
> Jeff Jones wrote:
>> We're having some odd problems with deadlocks in Saxon's NamePool, and
>> I was wondering if anyone here may have seen this behavior.
>>
>> Background: We're running OPS (er, OF) 3.0.1, which includes Saxon
>> 8.1.1, on Linux. Our application is NOT a "conventional" OF/XForms
>> app; rather, we're using OF's pipeline engine with our own pipelines
>> and XSLT to generate XHTML and WML for mobile web browsers. You can
>> see the app at http://xhtml.weather.com/xhtml/ (for XHTML) or
>> http://wml.weather.com/xhtml/ (for WML).
>>
>> We've generally been very happy with the performance of OF; we've seen
>> days with a million pageviews or more, with good stability and
>> response time.
>>
>> However, in the last week or so, we've seen our response time spike
>> under high loads. When we get a thread dump of the JVM (via "kill
>> -3"), we see that many of our listener threads are getting hung up in
>> Saxon's NamePool. Specifically, our stack traces look like this:
>>
>> "TP-Processor55" daemon prio=1 tid=0x088fbe68 nid=0x7572 waiting for
>> monitor entry [0x7ec30000..0x7ec31700]
>>     at org.orbeon.saxon.om.NamePool.allocate(NamePool.java:384)
>>     - waiting to lock <0x4cde0260> (a org.orbeon.saxon.om.NamePool)
>>     at org.orbeon.saxon.tree.NamespaceImpl.<init>(NamespaceImpl.java:35)
>>     at
>> org.orbeon.saxon.tree.ElementWithAttributes.addNamespaceNodes(ElementWithAttributes.java:191)
>>
>>     at
>> org.orbeon.saxon.tree.ElementWithAttributes.getNamespaceCodes(ElementWithAttributes.java:227)
>>
>>     at
>> org.orbeon.saxon.style.LiteralResultElement.validate(LiteralResultElement.java:214)
>>
>> (etc...)
>>
>> The odd thing is that all the threads take a different route to get to
>> this point; they all hang on NamePool.allocate(), but the rest of the
>> stack trace is different, showing that the threads were all at
>> different points of generating their responses when they hit the
>> deadlock.
>>
>> We can't associate the first appearance of these problems with any
>> particular change in our application code or server config; they seem
>> to have cropped up out of nowhere.
>>
>> Can anyone shed any light on this? We'd be eternally grateful.
>>
>> (I'll also be posting this question to the Saxon forum, but since
>> we're dealing with Orbeon's rewritten Saxon processor here, it seems
>> reasonable to ask here.)
>>
>> Thanks,
>>
>> Jeff Jones
>> Sr Internet App Devel
>> The Weather Channel Interactive
>>
>
>
>
> ------------------------------------------------------------------------
>
>
> --
> You receive this message as a subscriber of the [hidden email] mailing list.
> To unsubscribe: mailto:[hidden email]
> For general help: mailto:[hidden email]?subject=help
> ObjectWeb mailing lists service home page: http://www.objectweb.org/wws

> Erik,
>
> I saw some comments in the Saxon forums about NamePool problems in Saxon
> 6, but the threads all ended with "upgrade to Saxon 7.5 and you'll be
> fine."
>
> We're running Sun's Java 6 in production - we kept hitting an
> OutOfMemory error due to a classloader/memory-mapping bug in the 1.5
> JVM. Our upgrade to Java 6 happened before the onset of the current
> problems, though.
>
> We are about to try out a recent nightly build of OF 3.5, for the
> express purpose of getting the Saxon 8.8 upgrade. It appears that a
> little bit of tweaking was done to NamePool between 8.1 and 8.8, but
> it's just a shot in the dark. I'll definitely let you know what our
> results are.
>
> I'll look into the notion of JVM-dependent synchronization problems -
> thanks for the suggestion.
>
> And thanks for all your work on Orbeon Forms!