Orbeon Forms community mailing list

Re: Re: Re: Slow page load

Posted by Erik Bruchez on Dec 21, 2010; 2:21am
URL: https://discuss.orbeon.com/Slow-page-load-tp2768407p3117204.html

>> It uses the XForms engine in the background, which
>> stores a DOM in RAM, in particular.
> Good to know, that explains a lot.
> I think it would be a good idea to document such details in the processor
> documentation - i.e. whether the processor can work on a SAX-like "stream" of
> XML events or whether the whole file is loaded at once into a DOM.

Good point. I have used the opportunity to move the XForms submission
processor doc to the wiki:

http://wiki.orbeon.com/forms/doc/developer-guide/processors-xforms-submission

and added a note about (non-)streaming here.

Haven't updated other processors' documentation on streaming at this
point, but we should do that in the future.

> I need to send the data to a site using HTTP POST (the receiver is an eXist
> XQuery script). From what I gather, URL generator does the opposite. URL
> serializer says it can work with the http: protocol, but I can find no
> documentation about how the data is submitted (i.e. PUT, POST, whatever).

A bit of history: oxf:xforms-submission was created from the
requirement of doing PUT/POST of documents. We contemplated the idea
of upgrading the URL serializer (which is very limited now), and then
decided instead to leverage xforms:submission by simply wrapping the
XForms engine into a processor.

This had benefits, but as you have seen one drawback is that at this
point, this is not optimal especially for large documents.

-Erik

>
> Kind regards,
> Ralf
>
> Am Samstag 18 Dezember 2010, um 02:46:28 schrieb Erik Bruchez:
>> Ralf,
>>
>> Mmh, I fear that oxf:xforms-submission does not deal very well with
>> very large files.
>>
>> Can you use oxf:url-generator instead here?
>>
>> -Erik
>>
>> On Thu, Dec 16, 2010 at 7:11 AM, Ralf Jung <[hidden email]> wrote:
>> > Hi Erik,
>> >
>> > thanks, that was helpful!
>> > However, the data is sent to the DB using the xforms-submission
>> > processor. I assume the reason for the large RAM usage is that the
>> > stream it creates is cached, too, but how can I disable this? I could
>> > not find an option in the processor nor in the submission doc.
>> >
>> > Kind regards,
>> > Ralf Jung
>> >
>> > Am Donnerstag 16 Dezember 2010, um 06:45:42 schrieb Erik Bruchez:
>> >> I would expect a 45 MB file to take at least 10x in RAM, so 450 MB. If
>> >> that is just copied once or twice, that would explain why you run out
>> >> of memory.
>> >>
>> >> Now that doesn't mean things can't be improved.
>> >>
>> >> Both the URL generator and the HTTP serializer have configurations to
>> >> disable caching:
>> >>
>> >> *
>> >> http://wiki.orbeon.com/forms/doc/developer-guide/processors-url-generato
>> >> r# TOC-Cache-control *
>> >> http://www.orbeon.com/orbeon/doc/processors-serializers-http#d6e26
>> >>
>> >> I would try to load and serialize large documents by setting
>> >> use-local-cache to false in both cases.
>> >>
>> >> -Erik
>> >>
>> >> On Tue, Dec 14, 2010 at 1:45 AM, Ralf Jung <[hidden email]> wrote:
>> >> > Hi,
>> >> >
>> >> > not sure if this is related, but I am also under the impression that
>> >> > there is some memory issue somewhere: I am using an XPL pipe to load a
>> >> > document, using XSLT to split the nodes from the root into segments of
>> >> > max. 500 nodes, and then sending each of those segments to the DB. The
>> >> > XML file in 45MiB in size (>9k children of the root, i.e. 19
>> >> > segments), and if I give less than 1200MiB RAM to Orbeon, it crashes
>> >> > with "GC overhead limit exceeded" around the 10th segment. It feels
>> >> > like it's keeping much more data during that loop than it needs.
>> >> > (I previously had it send the whole file to the DB in a batch, which
>> >> > would behave even worse).
>> >> > This is with Orbeon 3.8 CE.
>> >> >
>> >> > Kind regards,
>> >> > Ralf Jung
>> >> >
>> >> > Am Dienstag 14 Dezember 2010, um 10:03:25 schrieb ncrofts:
>> >> >> Hi Erik,
>> >> >>
>> >> >> The tests were done with the official 3.8 CE release,
>> >> >> orbeon-3.8.0.rc1.201005041622-CE.
>> >> >>
>> >> >> I've not tried with a recent nightly build, but with respect all
>> >> >> previous builds I've tried have been similarly memory intensive. I'm
>> >> >> not sure yet what the specific bottleneck(s) are and there is
>> >> >> obviously quite a lot of processing going on just to read from a db
>> >> >> and process XML, but it is curious that an XPL based application
>> >> >> (that also processes db/XML) seems lighter on the memory footprint.
>> >> >> It does feel rather a disproportionate overhead just to generate a
>> >> >> few K's worth of HTML and I can't help but feel something is not
>> >> >> quite right.
>> >> >>
>> >> >> Regards,
>> >> >> Neil
>> >> >
>> >> > --
>> >> > You receive this message as a subscriber of the [hidden email]
>> >> > mailing list. To unsubscribe: mailto:[hidden email]
>> >> > For general help: mailto:[hidden email]?subject=help
>> >> > OW2 mailing lists service home page: http://www.ow2.org/wws
>> >
>> > --
>> > You receive this message as a subscriber of the [hidden email] mailing
>> > list. To unsubscribe: mailto:[hidden email]
>> > For general help: mailto:[hidden email]?subject=help
>> > OW2 mailing lists service home page: http://www.ow2.org/wws
>
>
> --
> You receive this message as a subscriber of the [hidden email] mailing list.
> To unsubscribe: mailto:[hidden email]
> For general help: mailto:[hidden email]?subject=help
> OW2 mailing lists service home page: http://www.ow2.org/wws
>
>

--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
OW2 mailing lists service home page: http://www.ow2.org/wws