Orbeon Forms community mailing list

Decreasing Orbeon RAM Usage

Classic

List

Threaded

9 messages Options

Ralf Jung

Decreasing Orbeon RAM Usage

Hi,

I am looking for advise how to decrease RAM usage of Orbeon 3.9. I've got a
fairly simply pipeline that fetches a relatively large XML file (50 MiB) via
HTTP (using the URL generator), applies an XSLT transformation to it and sends
the reply back to the user. The xpl file can be found attached.
However, even if I grant the tomcat process (which hosts Orbeon as the only
servlet) 1000M of max. heap space, an out of memory exception is thrown. I
have to go up to 1200M. Is it really necessary to grant Orbeon so much RAM?
If I add an XML converter at the end of the pipeline to get proper indention
in the returned XML, even that is not enough.

This is even more surprising as I am using another pipeline that reads the
same amount of data from a file on disc, applies a fairly complex XSLT
transformation, then iterates over the children of the root node of the result
and stores those as individual files on disc. That transformation is run with
the Orbeon CLI module and it works fine with as little as 400M of heap size.
It's almost as if just doing the same within tomcat already triples RAM usage?
My bet would be that some kind of caching is performed, which would of course
be a horribly bad idea here.

Any help would be appreciated,
Ralf Jung

--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
OW2 mailing lists service home page: http://www.ow2.org/wws

export.xpl (1K) Download Attachment

Hank Ratzesberger-3

Re: Decreasing Orbeon RAM Usage

Hi Ralf,

I removed the eXist and several other libraries and disabled the XForms

servlet in the web.xml for a site that was simply to provide REST

services (query an Oracle database, perform XSTL, etc) . I am sure

the memory footprint is much smaller.

On the other hand, isn't it the Tomcat/servlet configuration that

specifies the memory heap? Set the memory options to a minimum?

http://wiki.orbeon.com/forms/doc/developer-guide/admin/performance-tuning#TOC-Set--Xms-and--Xmx-to-the-same-value

Cheers,

Hank

On Sat, Jul 9, 2011 at 1:28 PM, Ralf Jung <[hidden email]> wrote:

Hi,

I am looking for advise how to decrease RAM usage of Orbeon 3.9. I've got a
fairly simply pipeline that fetches a relatively large XML file (50 MiB) via
HTTP (using the URL generator), applies an XSLT transformation to it and sends
the reply back to the user. The xpl file can be found attached.
However, even if I grant the tomcat process (which hosts Orbeon as the only
servlet) 1000M of max. heap space, an out of memory exception is thrown. I
have to go up to 1200M. Is it really necessary to grant Orbeon so much RAM?
If I add an XML converter at the end of the pipeline to get proper indention
in the returned XML, even that is not enough.

This is even more surprising as I am using another pipeline that reads the
same amount of data from a file on disc, applies a fairly complex XSLT
transformation, then iterates over the children of the root node of the result
and stores those as individual files on disc. That transformation is run with
the Orbeon CLI module and it works fine with as little as 400M of heap size.
It's almost as if just doing the same within tomcat already triples RAM usage?
My bet would be that some kind of caching is performed, which would of course
be a horribly bad idea here.

Any help would be appreciated,
Ralf Jung

--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
OW2 mailing lists service home page: http://www.ow2.org/wws

--
Hank Ratzesberger

XMLWerks.com

--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
OW2 mailing lists service home page: http://www.ow2.org/wws

Ralf Jung

Re: Re: Decreasing Orbeon RAM Usage

Hi Hank,

thanks for your reply!

> I removed the eXist and several other libraries and disabled the XForms
> servlet in the web.xml for a site that was simply to provide REST
> services (query an Oracle database, perform XSTL, etc) . I am sure
> the memory footprint is much smaller.
eXist is already disabled (and the jar files completely removed, just to be
sure) as well as Orbeon Form Runner & Builder, and XForms is required for
another part of the website.

> On the other hand, isn't it the Tomcat/servlet configuration that
> specifies the memory heap? Set the memory options to a minimum?
>
> http://wiki.orbeon.com/forms/doc/developer-guide/admin/performance-tuning#T
> OC-Set--Xms-and--Xmx-to-the-same-value
Yes, I have to specify heap size in the tomcat config, and I had to set it to
1.2G for the script to work. Which really surprised me, seeing that Tomcat
plus Orbeon can do the XForms app just fine with a 200M heap, and the large
transformation I am doing for the import with the standalone CLI Orbeon works
with 400M. Of course RAM usage will go up by Tomcat hanging around, but I'd
expect 600M to be enough, from my previous observations...

Kind regards,
Ralf

--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
OW2 mailing lists service home page: http://www.ow2.org/wws

Erik Bruchez

Re: Re: Re: Decreasing Orbeon RAM Usage

Administrator

Ralf,

By default some caching takes place with the URL generator and serialializers.

Try explicitly using oxf:url-generator and disabling the local cache:

http://wiki.orbeon.com/forms/doc/developer-guide/processors-url-generator#TOC-Cache-control

A similar setting exists for oxf:http-serializer:

http://wiki.orbeon.com/forms/doc/developer-guide/processors-http-serializer#TOC-Configuration

-Erik

On Wed, Jul 13, 2011 at 2:00 AM, Ralf Jung <[hidden email]> wrote:

> Hi Hank,
>
> thanks for your reply!
>
>> I removed the eXist and several other libraries and disabled the XForms
>> servlet in the web.xml for a site that was simply to provide REST
>> services (query an Oracle database, perform XSTL, etc) . I am sure
>> the memory footprint is much smaller.
> eXist is already disabled (and the jar files completely removed, just to be
> sure) as well as Orbeon Form Runner & Builder, and XForms is required for
> another part of the website.
>
>> On the other hand, isn't it the Tomcat/servlet configuration that
>> specifies the memory heap? Set the memory options to a minimum?
>>
>> http://wiki.orbeon.com/forms/doc/developer-guide/admin/performance-tuning#T
>> OC-Set--Xms-and--Xmx-to-the-same-value
> Yes, I have to specify heap size in the tomcat config, and I had to set it to
> 1.2G for the script to work. Which really surprised me, seeing that Tomcat
> plus Orbeon can do the XForms app just fine with a 200M heap, and the large
> transformation I am doing for the import with the standalone CLI Orbeon works
> with 400M. Of course RAM usage will go up by Tomcat hanging around, but I'd
> expect 600M to be enough, from my previous observations...
>
> Kind regards,
> Ralf
>
>
> --
> You receive this message as a subscriber of the [hidden email] mailing list.
> To unsubscribe: mailto:[hidden email]
> For general help: mailto:[hidden email]?subject=help
> OW2 mailing lists service home page: http://www.ow2.org/wws
>
>

Ralf Jung

Re: Re: Re: Re: Decreasing Orbeon RAM Usage

Hi Erik,

I already had disabled caching for the generator, but not for the serializer.
Doing so reduced RAM usage drastically, the full pipeline now works with a
500M heap. Thanks a lot!

Some hint in the wiki saying that that pipelines without "data" output (like
those using explicit serializers) need to be specified as "model" in the page
flow would be useful. Setting the same pipeline as view (which is what I
usually do) does not work, it complains about the view not having a data
output.

Is caching performed for intermediate results as well, like the output on an
XSLT processor that is then used as input for another processor, or is just
the beginning and end (the files read and the serialized output) cached?

Kind regards,
Ralf

Am Freitag 15 Juli 2011, um 03:46:35 schrieb Erik Bruchez:

> Ralf,
>
> By default some caching takes place with the URL generator and
> serialializers.
>
> Try explicitly using oxf:url-generator and disabling the local cache:
>
> http://wiki.orbeon.com/forms/doc/developer-guide/processors-url-generator#T
> OC-Cache-control
>
> A similar setting exists for oxf:http-serializer:
>
> http://wiki.orbeon.com/forms/doc/developer-guide/processors-http-serializer
> #TOC-Configuration
>
> -Erik
>
> On Wed, Jul 13, 2011 at 2:00 AM, Ralf Jung <[hidden email]> wrote:
> > Hi Hank,
> >
> > thanks for your reply!
> >
> >> I removed the eXist and several other libraries and disabled the XForms
> >> servlet in the web.xml for a site that was simply to provide REST
> >> services (query an Oracle database, perform XSTL, etc) . I am sure
> >> the memory footprint is much smaller.
> >
> > eXist is already disabled (and the jar files completely removed, just to
> > be sure) as well as Orbeon Form Runner & Builder, and XForms is required
> > for another part of the website.
> >
> >> On the other hand, isn't it the Tomcat/servlet configuration that
> >> specifies the memory heap? Set the memory options to a minimum?
> >>
> >> http://wiki.orbeon.com/forms/doc/developer-guide/admin/performance-tunin
> >> g#T OC-Set--Xms-and--Xmx-to-the-same-value
> >
> > Yes, I have to specify heap size in the tomcat config, and I had to set
> > it to 1.2G for the script to work. Which really surprised me, seeing
> > that Tomcat plus Orbeon can do the XForms app just fine with a 200M
> > heap, and the large transformation I am doing for the import with the
> > standalone CLI Orbeon works with 400M. Of course RAM usage will go up by
> > Tomcat hanging around, but I'd expect 600M to be enough, from my
> > previous observations...
> >
> > Kind regards,
> > Ralf
> >
> >
> > --
> > You receive this message as a subscriber of the [hidden email] mailing
> > list. To unsubscribe: mailto:[hidden email]
> > For general help: mailto:[hidden email]?subject=help
> > OW2 mailing lists service home page: http://www.ow2.org/wws

Erik Bruchez

Re: Re: Re: Re: Re: Decreasing Orbeon RAM Usage

Administrator

Ralf,

> I already had disabled caching for the generator, but not for the serializer.
> Doing so reduced RAM usage drastically, the full pipeline now works with a
> 500M heap. Thanks a lot!

Great!

> Some hint in the wiki saying that that pipelines without "data" output (like
> those using explicit serializers) need to be specified as "model" in the page
> flow would be useful. Setting the same pipeline as view (which is what I
> usually do) does not work, it complains about the view not having a data
> output.

We kind of have something along those lines buried here:

http://wiki.orbeon.com/forms/doc/developer-guide/page-flow-controller#TOC-Model-only

If you have a more explicit piece of text to put there (or somewhere
else), I will gladly do it ;)

> Is caching performed for intermediate results as well, like the output on an
> XSLT processor that is then used as input for another processor, or is just
> the beginning and end (the files read and the serialized output) cached?

Inputs and outputs in general are not cached. Only things like XSLT
stylesheets or small configs are meant to be cached.

However, if you have "tees" (i.e. branches in the pipeline), then
temporarily large amounts of data might be kept in memory.

-Erik

>
> Kind regards,
> Ralf
>
> Am Freitag 15 Juli 2011, um 03:46:35 schrieb Erik Bruchez:
>> Ralf,
>>
>> By default some caching takes place with the URL generator and
>> serialializers.
>>
>> Try explicitly using oxf:url-generator and disabling the local cache:
>>
>> http://wiki.orbeon.com/forms/doc/developer-guide/processors-url-generator#T
>> OC-Cache-control
>>
>> A similar setting exists for oxf:http-serializer:
>>
>> http://wiki.orbeon.com/forms/doc/developer-guide/processors-http-serializer
>> #TOC-Configuration
>>
>> -Erik
>>
>> On Wed, Jul 13, 2011 at 2:00 AM, Ralf Jung <[hidden email]> wrote:
>> > Hi Hank,
>> >
>> > thanks for your reply!
>> >
>> >> I removed the eXist and several other libraries and disabled the XForms
>> >> servlet in the web.xml for a site that was simply to provide REST
>> >> services (query an Oracle database, perform XSTL, etc) . I am sure
>> >> the memory footprint is much smaller.
>> >
>> > eXist is already disabled (and the jar files completely removed, just to
>> > be sure) as well as Orbeon Form Runner & Builder, and XForms is required
>> > for another part of the website.
>> >
>> >> On the other hand, isn't it the Tomcat/servlet configuration that
>> >> specifies the memory heap? Set the memory options to a minimum?
>> >>
>> >> http://wiki.orbeon.com/forms/doc/developer-guide/admin/performance-tunin
>> >> g#T OC-Set--Xms-and--Xmx-to-the-same-value
>> >
>> > Yes, I have to specify heap size in the tomcat config, and I had to set
>> > it to 1.2G for the script to work. Which really surprised me, seeing
>> > that Tomcat plus Orbeon can do the XForms app just fine with a 200M
>> > heap, and the large transformation I am doing for the import with the
>> > standalone CLI Orbeon works with 400M. Of course RAM usage will go up by
>> > Tomcat hanging around, but I'd expect 600M to be enough, from my
>> > previous observations...
>> >
>> > Kind regards,
>> > Ralf
>> >
>> >
>> > --
>> > You receive this message as a subscriber of the [hidden email] mailing
>> > list. To unsubscribe: mailto:[hidden email]
>> > For general help: mailto:[hidden email]?subject=help
>> > OW2 mailing lists service home page: http://www.ow2.org/wws
>
>
> --
> You receive this message as a subscriber of the [hidden email] mailing list.
> To unsubscribe: mailto:[hidden email]
> For general help: mailto:[hidden email]?subject=help
> OW2 mailing lists service home page: http://www.ow2.org/wws
>
>

Ralf Jung

Re: Re: Re: Re: Re: Re: Decreasing Orbeon RAM Usage

Hi Erik,

> We kind of have something along those lines buried here:
>
> http://wiki.orbeon.com/forms/doc/developer-guide/page-flow-controller#TOC-M
> odel-only
>
> If you have a more explicit piece of text to put there (or somewhere
> else), I will gladly do it ;)
I would add something along these lines to
http://wiki.orbeon.com/forms/doc/developer-guide/processors-http-serializer,
maybe as a fourth "Note":

If your pipeline does not have any output besides the HTTP serializer, you
have to remove the output declaration from the pipeline and reference it as
"model" in the pageflow controller. Otherwise, Orbeon will complain about the
missing data output.

However I'm not a native speaker, so take the formulation with a grain of salt
;-)

> Inputs and outputs in general are not cached. Only things like XSLT
> stylesheets or small configs are meant to be cached.
>
> However, if you have "tees" (i.e. branches in the pipeline), then
> temporarily large amounts of data might be kept in memory.
I assume that "large amount of data" is a SAX-event representation of the
document passed through that pipeline?

Kind regards,
Ralf

--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
OW2 mailing lists service home page: http://www.ow2.org/wws

Erik Bruchez

Re: Re: Re: Re: Re: Re: Re: Decreasing Orbeon RAM Usage

Administrator

Ralf,

Thanks. I have placed a note to this effect at the end of this page:

http://wiki.orbeon.com/forms/doc/developer-guide/processors-http-serializer

-Erik

On Thu, Jul 28, 2011 at 5:54 AM, Ralf Jung <[hidden email]> wrote:

> Hi Erik,
>
>> We kind of have something along those lines buried here:
>>
>> http://wiki.orbeon.com/forms/doc/developer-guide/page-flow-controller#TOC-M
>> odel-only
>>
>> If you have a more explicit piece of text to put there (or somewhere
>> else), I will gladly do it ;)
> I would add something along these lines to
> http://wiki.orbeon.com/forms/doc/developer-guide/processors-http-serializer,
> maybe as a fourth "Note":
>
> If your pipeline does not have any output besides the HTTP serializer, you
> have to remove the output declaration from the pipeline and reference it as
> "model" in the pageflow controller. Otherwise, Orbeon will complain about the
> missing data output.
>
> However I'm not a native speaker, so take the formulation with a grain of salt
> ;-)
>
>
>> Inputs and outputs in general are not cached. Only things like XSLT
>> stylesheets or small configs are meant to be cached.
>>
>> However, if you have "tees" (i.e. branches in the pipeline), then
>> temporarily large amounts of data might be kept in memory.
> I assume that "large amount of data" is a SAX-event representation of the
> document passed through that pipeline?
>
> Kind regards,
> Ralf
>
>
> --
> You receive this message as a subscriber of the [hidden email] mailing list.
> To unsubscribe: mailto:[hidden email]
> For general help: mailto:[hidden email]?subject=help
> OW2 mailing lists service home page: http://www.ow2.org/wws
>
>

Ralf Jung

Re: Re: Re: Re: Re: Re: Re: Re: Decreasing Orbeon RAM Usage

Hi Erik,

great, thanks :)

Kind regards,
Ralf

On Wednesday 03 August 2011 06:42:40 Erik Bruchez wrote:

> Ralf,
>
> Thanks. I have placed a note to this effect at the end of this page:
>
> http://wiki.orbeon.com/forms/doc/developer-guide/processors-http-serializer
>
> -Erik
>
> On Thu, Jul 28, 2011 at 5:54 AM, Ralf Jung <[hidden email]> wrote:
> > Hi Erik,
> >
> >> We kind of have something along those lines buried here:
> >>
> >> http://wiki.orbeon.com/forms/doc/developer-guide/page-flow-controller#TO
> >> C-M odel-only
> >>
> >> If you have a more explicit piece of text to put there (or somewhere
> >> else), I will gladly do it ;)
> >
> > I would add something along these lines to
> > http://wiki.orbeon.com/forms/doc/developer-guide/processors-http-serializ
> > er, maybe as a fourth "Note":
> >
> > If your pipeline does not have any output besides the HTTP serializer,
> > you have to remove the output declaration from the pipeline and
> > reference it as "model" in the pageflow controller. Otherwise, Orbeon
> > will complain about the missing data output.
> >
> > However I'm not a native speaker, so take the formulation with a grain of
> > salt ;-)
> >
> >> Inputs and outputs in general are not cached. Only things like XSLT
> >> stylesheets or small configs are meant to be cached.
> >>
> >> However, if you have "tees" (i.e. branches in the pipeline), then
> >> temporarily large amounts of data might be kept in memory.
> >
> > I assume that "large amount of data" is a SAX-event representation of the
> > document passed through that pipeline?
> >
> > Kind regards,
> > Ralf
> >
> >
> > --
> > You receive this message as a subscriber of the [hidden email] mailing
> > list. To unsubscribe: mailto:[hidden email]
> > For general help: mailto:[hidden email]?subject=help
> > OW2 mailing lists service home page: http://www.ow2.org/wws