Orbeon server performance

We've been running some load tests on our Orbeon-based web app and
finding that the application's response times quickly degrade with
multiple users.

Overall on a Pentium 4 3.0GHz machine (4GB RAM, about 1.5GB for the VM)
we can't mange to serve more than about 35 requests a minute on a LAN
connection. So even with just 30 concurrent users performance is
unacceptable. Profiling suggests about 85-90% of the time is spent in
Orbeon.

The form we've been testing with is reasonably big, but not complex: the
XHTML+XForms definition is 325KB. Orbeon is being used for the
XForms->HTML epilogue only: the XHTML+XForms definition form is passed
in as as w3c DOM, with most of the instance data already embedded inline.

Fiddling with caching settings seems to have little impact: since we're
testing with just a single form, the default cache settings are probably
big enough. There are some cache misses, but debugging these they seem
to be related to limitations in Orbeons caching:
- XPathCache.getXPathExpression doesn't cache across requests, because
XFormsFunctionLibary.hashCode() is different for each request.
- the RequestGenerator seems to cause some cache misses also

The profiler reports <1% in XPathCache, so I doubt these cache misses
are significant. The request generator I don't know about, but suspect
it's not significant either.

Overall the epilogue is extremely CPU intensive. Once form is displayed,
performance is ok for interactions with the forms: initial display is
the problem.

This is running on the OPS_3_5_M1_200609021808 build. We're in the
process of upgrading to the final release: I know since this build the
read only instances have been added, but on this form there isn't a
great deal of reference data. Other than this I don't believe there has
been any major performance work has there?

What have been others experiences with running Orbeon under large user
loads?

Is further performance work in the pipeline (so to speak)? One part of
the epilogue which I wonder can be avoided is the execution of
theme.xsl, which appears to be fairly expensive.

Adrian




--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
ObjectWeb mailing lists service home page: http://www.objectweb.org/wws

Adrian,

 > - XPathCache.getXPathExpression doesn't cache across requests, because
 > XFormsFunctionLibary.hashCode() is different for each request.

This is a good catch. I suspect this could be fixed.

http://forge.objectweb.org/tracker/index.php?func=detail&aid=306695&group_id=168&atid=350207

It probably *does* cache between requests acting on a same XForms
document, if the document stays in cache, but it probably does *not*
cache between requests that regard different documents.

 > The profiler reports <1% in XPathCache, so I doubt these cache misses
 > are significant.

What would be more important would be the time spent in the Saxon
XPath engine! I don't think the cache itself would take much time.

 > Overall the epilogue is extremely CPU intensive. Once form is
 > displayed, performance is ok for interactions with the forms:
 > initial display is the problem.

At least that is good news ;-)

 > This is running on the OPS_3_5_M1_200609021808 build. We're in the
 > process of upgrading to the final release: I know since this build
 > the read only instances have been added, but on this form there
 > isn't a great deal of reference data. Other than this I don't
 > believe there has been any major performance work has there?

I wouldn't think so, but it's better to give it a try. Also make sure
that your properties.xml use the property names of 3.5. I think that
one or two properties related to caching may have changed names.

 > Is further performance work in the pipeline (so to speak)? One part
 > of the epilogue which I wonder can be avoided is the execution of
 > theme.xsl, which appears to be fairly expensive.

Performance is definitely in our pipeline. There are tons and tons of
things that can be done to improve Orbeon Forms performance. We have
barely scratched the surface of it.

I assume that you had a look at this page:

   http://www.orbeon.com/ops/doc/integration-tuning

If you don't need theme.xsl, then by all means don't include it. You
can also remove the handling of the legacy XForms engine in
xforms-epilogue.xpl, and simplify epilogue-servlet.xpl.

The epilogue also runs the XForms engine, and that should sure take
some time.

It's nice to see that you used a profiler, but I think that Java
profiling should tell us even more. We actually have a tool to profile
hprof stack traces in Orbeon Forms. It's not included in 3.5 builds,
but it's in the CVS, under the old location
examples/web/examples/profiler. This little XForms app allows you to
upload an hprof file and look at useful aggregated times. Something
you may want to try.

I hope this helps,

-Erik

--
Orbeon Forms - Web Forms for the Enterprise Done the Right Way
http://www.orbeon.com/




--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
ObjectWeb mailing lists service home page: http://www.objectweb.org/wws

Hi Erik,

Erik Bruchez wrote: elsewhere (some of which were specific to us). The output of an <img>
tag for the xforms-help xhtml:label had to be done with a patch to
HandlerBase:

    // This was originally done in theme.xsl
    if(type.equals("help")) {
        final String imgQName =
XMLUtils.buildQName(handlerContext.findXHTMLPrefix(), "img");
        final AttributesImpl imgAttributes = new AttributesImpl();
        imgAttributes.addAttribute("", "class", "class",
ContentHandlerHelper.CDATA, "xforms-help-image");
        imgAttributes.addAttribute("", "alt", "alt",
ContentHandlerHelper.CDATA, "Help");
        imgAttributes.addAttribute("", "src", "src",
ContentHandlerHelper.CDATA, "/renderer/images/help.gif");
       
handlerContext.getController().getOutput().startElement(XMLConstants.XHTML_NAMESPACE_URI,
"img", imgQName, imgAttributes);
       
handlerContext.getController().getOutput().endElement(XMLConstants.XHTML_NAMESPACE_URI,
"img", imgQName);
    }

Removing the theme.xsl step made a fairly significant difference: the
time spent running the epilogue dropped by about 20%. I'd suggest that
it's worth modifying the default epilogue to remove this expensive step,
given that the logic in theme.xsl can be achieved elsewhere.

It does act as a convenient mechanism for customising/post-processing
the HTML output - but the performance impact should be advertised for
those using it. Alternatively some sort of Java plugin/extension
mechanism could be added to the XFormsToHtml processor (SAX based so
harder to write but little impact on performance).

Adrian




--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
ObjectWeb mailing lists service home page: http://www.objectweb.org/wws

Adrian Baker wrote:
> ...
> - XPathCache.getXPathExpression doesn't cache across requests, because
> XFormsFunctionLibary.hashCode() is different for each request.
> ...
>
> The profiler reports <1% in XPathCache, so I doubt these cache misses
> are significant.

Under a multi-user load these misses seem to become more significant.
JProfiler is reporting a lot of contention on
org.orbeon.saxon.om.NamePool, which gets used from
org.orbeon.oxf.util.XPathCache$CachedPoolableObjetFactory.makeObject().

Of course the profiler has lied to me before. But in general there does
seem to be some contention slowing Orbeon when loading it with multiple
users: throughput (requests served per min) is higher with a single user
than with multiple.


Adrian




--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
ObjectWeb mailing lists service home page: http://www.objectweb.org/wws

Adrian Baker wrote: Ah, I see this has already been raised:
http://mail-archive.objectweb.org/ops-users/2007-02/msg00103.html

If the XPathCache problem
(http://forge.objectweb.org/tracker/index.php?func=detail&aid=306695&group_id=168&atid=350207)
is fixed, this would reduce the number of calls to NamePool, and perhaps
reduce the amount of deadlocking?

Note that I'm running on Saxon 8.7, so 8.8 may have fixed/improved upon
this problem anyway.

Adrian




--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
ObjectWeb mailing lists service home page: http://www.objectweb.org/wws

Adrian,

We've observed the same thing here - in extreme cases, we've actually
had servers brought down by contention for NamePool.

At that point, we were running OF 3.0.1, which includes Orbeon's
repackaging of Saxon 8.1.1B. We upgraded to OF 3.5 (actually a post-RC1
nightly build from just a month or two ago), which contains Saxon 8.8B,
and the NamePool contention decreased significantly; I don't have a
specific reference, but I'm pretty sure I found something in Saxon's
changelogs about improvements to the NamePool architecture.

So the moral of the story, I guess, is that you're not the only one to
see the problems you're reporting here. If you're not running OF 3.5,
though, this might be a good reason to migrate, because it should make a
big difference.

Jeff Jones
Sr Internet Appl Dev
The Weather Channel Interactive


Adrian Baker wrote:

--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
ObjectWeb mailing lists service home page: http://www.objectweb.org/wws

Adrian Baker wrote:

> If the XPathCache problem
> (http://forge.objectweb.org/tracker/index.php?func=detail&aid=306695&group_id=168&atid=350207)
> is fixed, this would reduce the number of calls to NamePool, and perhaps
> reduce the amount of deadlocking?

FWIW I think we'll fix the bug above fairly soon (we have a tentative
fix already). We have also investigated another issue related to the
XPath cache. I think we could get by with a single, global
XFormsFunctionLibrary object, and incidently single IndependentContext
and Configuration objects as well.

-Erik

--
Orbeon Forms - Web Forms for the Enterprise Done the Right Way
http://www.orbeon.com/



--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
ObjectWeb mailing lists service home page: http://www.objectweb.org/wws

I just wanted to mention that this bug is fixed, and that not only
should the XPathCache cache across pages, but also we got rid of memory
leaks at the same time. :-)

-Erik

--
Orbeon Forms - Web Forms for the Enterprise Done the Right Way
http://www.orbeon.com/



--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
ObjectWeb mailing lists service home page: http://www.objectweb.org/wws