I have a page that loads fast (about 1-2 seconds) on one server, and extremely slowly on another (about 10 minutes). I verified that my data is loading in about 5 seconds on the slow server, so the majority of the time is not spent loading the data from the web service. I turned on debug logging and got the following logged for each XML element loaded:
2010-09-30 13:54:37,578 DEBUG XFormsServer - XForms - controls - start building 2010-09-30 13:54:51,797 DEBUG XFormsServer - XForms - controls - end building {time (ms): "14219"} As you can see, building the controls is taking about 15 seconds, and it is doing that for every element being mapped to a table cell, adding up to about 10 minutes. Any idea why this logic is taking so long? Thanks, Bill |
Administrator
|
Bill,
Any other differences between these servers, like servlet container, Java heap, etc.? -Erik On Thu, Sep 30, 2010 at 3:22 PM, wreeder <[hidden email]> wrote: > > I have a page that loads fast (about 1-2 seconds) on one server, and > extremely slowly on another (about 10 minutes). I verified that my data is > loading in about 5 seconds on the slow server, so the majority of the time > is not spent loading the data from the web service. I turned on debug > logging and got the following logged for each XML element loaded: > > 2010-09-30 13:54:37,578 DEBUG XFormsServer - XForms - controls - start > building > 2010-09-30 13:54:51,797 DEBUG XFormsServer - XForms - controls - end > building {time (ms): "14219"} > > As you can see, building the controls is taking about 15 seconds, and it is > doing that for every element being mapped to a table cell, adding up to > about 10 minutes. > > Any idea why this logic is taking so long? > > Thanks, > Bill > -- > View this message in context: http://orbeon-forms-ops-users.24843.n4.nabble.com/Slow-page-load-tp2768407p2768407.html > Sent from the Orbeon Forms (ops-users) mailing list archive at Nabble.com. > > > -- > You receive this message as a subscriber of the [hidden email] mailing list. > To unsubscribe: mailto:[hidden email] > For general help: mailto:[hidden email]?subject=help > OW2 mailing lists service home page: http://www.ow2.org/wws > > -- You receive this message as a subscriber of the [hidden email] mailing list. To unsubscribe: mailto:[hidden email] For general help: mailto:[hidden email]?subject=help OW2 mailing lists service home page: http://www.ow2.org/wws |
Erik,
The 2 environments are running on identical WebLogic domains on the same machine. It is likely that the slow server has more activity than the faster server. After more investigation, we found the following: 1. Every reload of this page caused the heap memory to reduce by about 20-30mb. After reloading this page several times, we ran out of heap memory. Do you know why memory is not being garbage collected before the JVM crashes with an out of heap memory exception? We are using WebLogic 11g with the Sun JDK 1.6.0_18 runtime. 2. This page has logic to dynamically add elements to an instance used to display a grid, and from logging, we we a bulk of the time being spent here. We will change our logic to fully prepare the instance in an XPL instead of in the XForm page and hope this improves the performance. 3. This page has 2 dialog boxes, requiring different data than the main page. We will initialize the instances required for the dialogs with an empty template, and will load it on demand when the dialog is loaded first. We hope the initial page load will be quicker with this technique. I'll let you know how this works for us. If you have any other suggestions of best practices regarding instance manipulation on a XForm page, I'd appreciate your insight. Thanks, Bill |
Administrator
|
Since 3.8.0, we implemented some performance enhancements which might
work for you. One issue with 3.8.0 was that at each page load for a given page, the so-called static state of the page was likely to be rebuild, which was costly. Recent fixes help quite a bit with that with better caching. Can you give a nightly build a try? We are also going to release some of them in a PE build soon: http://wiki.orbeon.com/forms/doc/developer-guide/release-orbeon-forms-381-perf-pe This said, 20-30 MB per page load is huge and I don't know if this can be accounted by this issue alone. -Erik On Tue, Oct 5, 2010 at 7:00 AM, wreeder <[hidden email]> wrote: > > Erik, > > The 2 environments are running on identical WebLogic domains on the same > machine. It is likely that the slow server has more activity than the faster > server. After more investigation, we found the following: > > 1. Every reload of this page caused the heap memory to reduce by about > 20-30mb. After reloading this page several times, we ran out of heap memory. > Do you know why memory is not being garbage collected before the JVM crashes > with an out of heap memory exception? We are using WebLogic 11g with the Sun > JDK 1.6.0_18 runtime. > > 2. This page has logic to dynamically add elements to an instance used to > display a grid, and from logging, we we a bulk of the time being spent here. > We will change our logic to fully prepare the instance in an XPL instead of > in the XForm page and hope this improves the performance. > > 3. This page has 2 dialog boxes, requiring different data than the main > page. We will initialize the instances required for the dialogs with an > empty template, and will load it on demand when the dialog is loaded first. > We hope the initial page load will be quicker with this technique. > > I'll let you know how this works for us. If you have any other suggestions > of best practices regarding instance manipulation on a XForm page, I'd > appreciate your insight. > > Thanks, > Bill > -- > View this message in context: http://orbeon-forms-ops-users.24843.n4.nabble.com/Slow-page-load-tp2768407p2956066.html > Sent from the Orbeon Forms (ops-users) mailing list archive at Nabble.com. > > > -- > You receive this message as a subscriber of the [hidden email] mailing list. > To unsubscribe: mailto:[hidden email] > For general help: mailto:[hidden email]?subject=help > OW2 mailing lists service home page: http://www.ow2.org/wws > > -- You receive this message as a subscriber of the [hidden email] mailing list. To unsubscribe: mailto:[hidden email] For general help: mailto:[hidden email]?subject=help OW2 mailing lists service home page: http://www.ow2.org/wws |
Erik, We re-factored the page to reduce the page size dramatically, and that helped a lot with performance. Next we upgraded to the 12/1/2010 nightly build, and found the performance to be about the same. Some pages even load a bit slower. Are there any configuration
parameters or xforms options we need to use to realize the improved performance? Thanks, Bill From: Erik Bruchez [via Orbeon Forms (ops-users)] [mailto:ml-node+[hidden email]]
Since 3.8.0, we implemented some performance enhancements which might
View message @
http://orbeon-forms-ops-users.24843.n4.nabble.com/Slow-page-load-tp2768407p2967825.html
CONFIDENTIALITY NOTICE This e-mail message from Delphi Technology, Inc. is intended only for the individual or entity to which it is addressed. This e-mail may contain information that is privileged, confidential and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you received this e-mail by accident, please notify the sender immediately and destroy this e-mail and all copies of it. |
Administrator
|
Bill,
Performance should be better by default. You will mostly notice it if you run multiple forms and with several users, as one of the main issue was caused by information being not cached or cached not aggressively enough. What kind of forms do you have, and what kind of performance do you observe? -Erik On Tue, Dec 7, 2010 at 6:40 AM, wreeder <[hidden email]> wrote: > Erik, > > > > We re-factored the page to reduce the page size dramatically, and that > helped a lot with performance. > > Next we upgraded to the 12/1/2010 nightly build, and found the performance > to be about the same. Some pages even load a bit slower. Are there any > configuration parameters or xforms options we need to use to realize the > improved performance? > > > > Thanks, > > Bill > > > > From: Erik Bruchez [via Orbeon Forms (ops-users)] [mailto:ml-node+[hidden > email]] > Sent: Thursday, October 07, 2010 11:21 PM > To: Bill Reeder > Subject: Re: Re: Slow page load > > > > Since 3.8.0, we implemented some performance enhancements which might > work for you. > > One issue with 3.8.0 was that at each page load for a given page, the > so-called static state of the page was likely to be rebuild, which was > costly. Recent fixes help quite a bit with that with better caching. > > Can you give a nightly build a try? We are also going to release some > of them in a PE build soon: > > http://wiki.orbeon.com/forms/doc/developer-guide/release-orbeon-forms-381-perf-pe > > This said, 20-30 MB per page load is huge and I don't know if this can > be accounted by this issue alone. > > -Erik > > On Tue, Oct 5, 2010 at 7:00 AM, wreeder <[hidden email]> wrote: > >> >> Erik, >> >> The 2 environments are running on identical WebLogic domains on the same >> machine. It is likely that the slow server has more activity than the >> faster >> server. After more investigation, we found the following: >> >> 1. Every reload of this page caused the heap memory to reduce by about >> 20-30mb. After reloading this page several times, we ran out of heap >> memory. >> Do you know why memory is not being garbage collected before the JVM >> crashes >> with an out of heap memory exception? We are using WebLogic 11g with the >> Sun >> JDK 1.6.0_18 runtime. >> >> 2. This page has logic to dynamically add elements to an instance used to >> display a grid, and from logging, we we a bulk of the time being spent >> here. >> We will change our logic to fully prepare the instance in an XPL instead >> of >> in the XForm page and hope this improves the performance. >> >> 3. This page has 2 dialog boxes, requiring different data than the main >> page. We will initialize the instances required for the dialogs with an >> empty template, and will load it on demand when the dialog is loaded >> first. >> We hope the initial page load will be quicker with this technique. >> >> I'll let you know how this works for us. If you have any other suggestions >> of best practices regarding instance manipulation on a XForm page, I'd >> appreciate your insight. >> >> Thanks, >> Bill >> -- >> View this message in context: >> http://orbeon-forms-ops-users.24843.n4.nabble.com/Slow-page-load-tp2768407p2956066.html >> Sent from the Orbeon Forms (ops-users) mailing list archive at Nabble.com. >> >> >> -- >> You receive this message as a subscriber of the [hidden email] mailing >> list. >> To unsubscribe: mailto:[hidden email] >> For general help: mailto:[hidden email]?subject=help >> OW2 mailing lists service home page: http://www.ow2.org/wws >> >> > > -- > You receive this message as a subscriber of the [hidden email] mailing list. > To unsubscribe: mailto:[hidden email] > For general help: mailto:[hidden email]?subject=help > > OW2 mailing lists service home page: http://www.ow2.org/wws > > ________________________________ > > View message @ > http://orbeon-forms-ops-users.24843.n4.nabble.com/Slow-page-load-tp2768407p2967825.html > To unsubscribe from Slow page load, click here. > > > > ________________________________ > CONFIDENTIALITY NOTICE > This e-mail message from Delphi Technology, Inc. is intended only for the > individual or entity to which it is addressed. This e-mail may contain > information that is privileged, confidential and exempt from disclosure > under applicable law. If you are not the intended recipient, you are hereby > notified that any dissemination, distribution or copying of this > communication is strictly prohibited. If you received this e-mail by > accident, please notify the sender immediately and destroy this e-mail and > all copies of it. > > ________________________________ > View this message in context: RE: Re: Slow page load > Sent from the Orbeon Forms (ops-users) mailing list archive at Nabble.com. > > > -- > You receive this message as a subscriber of the [hidden email] mailing > list. > To unsubscribe: mailto:[hidden email] > For general help: mailto:[hidden email]?subject=help > OW2 mailing lists service home page: http://www.ow2.org/wws > > -- You receive this message as a subscriber of the [hidden email] mailing list. To unsubscribe: mailto:[hidden email] For general help: mailto:[hidden email]?subject=help OW2 mailing lists service home page: http://www.ow2.org/wws |
Hello Erik,
Sorry to interrupt this thread, but the 30-40MB figure mentioned in the OP's message triggered a thought here. For what it is worth, we also see significant amounts of memory being consumed every request and the figure of 30-40MB per request is about what I see too when using a pretty basic XForms. We have two main applications. One which does not use Xforms, but does make use of XPL pipelines to process and serve up XML data. The XPL-based application also uses more memory per request than I would like, around 5-10MB each request. The XPL typically perform a database transaction out to MySQL and then apply some XSLT transforms to the data before serializing the data via HTTP. The second app we have is using Xforms, but not currently using Forms Builder, just hand-coded XForms. Again, these get their data by querying a db, transforming and then displaying in Xforms. The forms themselves. Every page request seems to cause about 30-40MB of memory to be consumed. Both applications are running in Tomcat 6.x with Java 6.x and also connect to MySQL 5.x The consequence of the above is that we are having significant problems scaling up the applications. I appreciate these figures are very anecdotal, but the main point being that there is seems to be a huge memory overhead when it comes to serving up the XForms pages and to be honest the XPL based app is not great either. I may well be being rather naive about the memory management, but it does seem really excessive that such huge amounts of memory are being consumed to serve up really quite small and simple pages. I'm hopefully going to be able to do some more profiling soon, as I really need to make this more scalable than it is at the moment. However any thoughts / experiences about what the most likely causes of the large memory use would be helpful and may help tackle this issue which I fear is a major disadvantage of the current platform. Thanks in advance for any help. Regards, Neil |
Administrator
|
Neil,
Are these numbers with a recent build? Just curious. -Erik On Sun, Dec 12, 2010 at 3:41 AM, ncrofts <[hidden email]> wrote: > > Hello Erik, > > Sorry to interrupt this thread, but the 30-40MB figure mentioned in the OP's > message triggered a thought here. For what it is worth, we also see > significant amounts of memory being consumed every request and the figure of > 30-40MB per request is about what I see too when using a pretty basic > XForms. > > We have two main applications. One which does not use Xforms, but does make > use of XPL pipelines to process and serve up XML data. The XPL-based > application also uses more memory per request than I would like, around > 5-10MB each request. The XPL typically perform a database transaction out to > MySQL and then apply some XSLT transforms to the data before serializing the > data via HTTP. > > The second app we have is using Xforms, but not currently using Forms > Builder, just hand-coded XForms. Again, these get their data by querying a > db, transforming and then displaying in Xforms. The forms themselves. Every > page request seems to cause about 30-40MB of memory to be consumed. > > Both applications are running in Tomcat 6.x with Java 6.x and also connect > to MySQL 5.x > > The consequence of the above is that we are having significant problems > scaling up the applications. > > I appreciate these figures are very anecdotal, but the main point being that > there is seems to be a huge memory overhead when it comes to serving up the > XForms pages and to be honest the XPL based app is not great either. I may > well be being rather naive about the memory management, but it does seem > really excessive that such huge amounts of memory are being consumed to > serve up really quite small and simple pages. > > I'm hopefully going to be able to do some more profiling soon, as I really > need to make this more scalable than it is at the moment. However any > thoughts / experiences about what the most likely causes of the large memory > use would be helpful and may help tackle this issue which I fear is a major > disadvantage of the current platform. > > Thanks in advance for any help. > > Regards, > Neil > > > -- > View this message in context: http://orbeon-forms-ops-users.24843.n4.nabble.com/Slow-page-load-tp2768407p3084214.html > Sent from the Orbeon Forms (ops-users) mailing list archive at Nabble.com. > > > -- > You receive this message as a subscriber of the [hidden email] mailing list. > To unsubscribe: mailto:[hidden email] > For general help: mailto:[hidden email]?subject=help > OW2 mailing lists service home page: http://www.ow2.org/wws > > -- You receive this message as a subscriber of the [hidden email] mailing list. To unsubscribe: mailto:[hidden email] For general help: mailto:[hidden email]?subject=help OW2 mailing lists service home page: http://www.ow2.org/wws |
Hi Erik,
The tests were done with the official 3.8 CE release, orbeon-3.8.0.rc1.201005041622-CE. I've not tried with a recent nightly build, but with respect all previous builds I've tried have been similarly memory intensive. I'm not sure yet what the specific bottleneck(s) are and there is obviously quite a lot of processing going on just to read from a db and process XML, but it is curious that an XPL based application (that also processes db/XML) seems lighter on the memory footprint. It does feel rather a disproportionate overhead just to generate a few K's worth of HTML and I can't help but feel something is not quite right. Regards, Neil |
Hi,
not sure if this is related, but I am also under the impression that there is some memory issue somewhere: I am using an XPL pipe to load a document, using XSLT to split the nodes from the root into segments of max. 500 nodes, and then sending each of those segments to the DB. The XML file in 45MiB in size (>9k children of the root, i.e. 19 segments), and if I give less than 1200MiB RAM to Orbeon, it crashes with "GC overhead limit exceeded" around the 10th segment. It feels like it's keeping much more data during that loop than it needs. (I previously had it send the whole file to the DB in a batch, which would behave even worse). This is with Orbeon 3.8 CE. Kind regards, Ralf Jung Am Dienstag 14 Dezember 2010, um 10:03:25 schrieb ncrofts: > Hi Erik, > > The tests were done with the official 3.8 CE release, > orbeon-3.8.0.rc1.201005041622-CE. > > I've not tried with a recent nightly build, but with respect all previous > builds I've tried have been similarly memory intensive. I'm not sure yet > what the specific bottleneck(s) are and there is obviously quite a lot of > processing going on just to read from a db and process XML, but it is > curious that an XPL based application (that also processes db/XML) seems > lighter on the memory footprint. It does feel rather a disproportionate > overhead just to generate a few K's worth of HTML and I can't help but feel > something is not quite right. > > Regards, > Neil -- You receive this message as a subscriber of the [hidden email] mailing list. To unsubscribe: mailto:[hidden email] For general help: mailto:[hidden email]?subject=help OW2 mailing lists service home page: http://www.ow2.org/wws |
Administrator
|
I would expect a 45 MB file to take at least 10x in RAM, so 450 MB. If
that is just copied once or twice, that would explain why you run out of memory. Now that doesn't mean things can't be improved. Both the URL generator and the HTTP serializer have configurations to disable caching: * http://wiki.orbeon.com/forms/doc/developer-guide/processors-url-generator#TOC-Cache-control * http://www.orbeon.com/orbeon/doc/processors-serializers-http#d6e26 I would try to load and serialize large documents by setting use-local-cache to false in both cases. -Erik On Tue, Dec 14, 2010 at 1:45 AM, Ralf Jung <[hidden email]> wrote: > Hi, > > not sure if this is related, but I am also under the impression that there is > some memory issue somewhere: I am using an XPL pipe to load a document, using > XSLT to split the nodes from the root into segments of max. 500 nodes, and > then sending each of those segments to the DB. The XML file in 45MiB in size > (>9k children of the root, i.e. 19 segments), and if I give less than 1200MiB > RAM to Orbeon, it crashes with "GC overhead limit exceeded" around the 10th > segment. It feels like it's keeping much more data during that loop than it > needs. > (I previously had it send the whole file to the DB in a batch, which would > behave even worse). > This is with Orbeon 3.8 CE. > > Kind regards, > Ralf Jung > > Am Dienstag 14 Dezember 2010, um 10:03:25 schrieb ncrofts: >> Hi Erik, >> >> The tests were done with the official 3.8 CE release, >> orbeon-3.8.0.rc1.201005041622-CE. >> >> I've not tried with a recent nightly build, but with respect all previous >> builds I've tried have been similarly memory intensive. I'm not sure yet >> what the specific bottleneck(s) are and there is obviously quite a lot of >> processing going on just to read from a db and process XML, but it is >> curious that an XPL based application (that also processes db/XML) seems >> lighter on the memory footprint. It does feel rather a disproportionate >> overhead just to generate a few K's worth of HTML and I can't help but feel >> something is not quite right. >> >> Regards, >> Neil > > > -- > You receive this message as a subscriber of the [hidden email] mailing list. > To unsubscribe: mailto:[hidden email] > For general help: mailto:[hidden email]?subject=help > OW2 mailing lists service home page: http://www.ow2.org/wws > > -- You receive this message as a subscriber of the [hidden email] mailing list. To unsubscribe: mailto:[hidden email] For general help: mailto:[hidden email]?subject=help OW2 mailing lists service home page: http://www.ow2.org/wws |
Hi Erik,
thanks, that was helpful! However, the data is sent to the DB using the xforms-submission processor. I assume the reason for the large RAM usage is that the stream it creates is cached, too, but how can I disable this? I could not find an option in the processor nor in the submission doc. Kind regards, Ralf Jung Am Donnerstag 16 Dezember 2010, um 06:45:42 schrieb Erik Bruchez: > I would expect a 45 MB file to take at least 10x in RAM, so 450 MB. If > that is just copied once or twice, that would explain why you run out > of memory. > > Now that doesn't mean things can't be improved. > > Both the URL generator and the HTTP serializer have configurations to > disable caching: > > * > http://wiki.orbeon.com/forms/doc/developer-guide/processors-url-generator# > TOC-Cache-control * > http://www.orbeon.com/orbeon/doc/processors-serializers-http#d6e26 > > I would try to load and serialize large documents by setting > use-local-cache to false in both cases. > > -Erik > > On Tue, Dec 14, 2010 at 1:45 AM, Ralf Jung <[hidden email]> wrote: > > Hi, > > > > not sure if this is related, but I am also under the impression that > > there is some memory issue somewhere: I am using an XPL pipe to load a > > document, using XSLT to split the nodes from the root into segments of > > max. 500 nodes, and then sending each of those segments to the DB. The > > XML file in 45MiB in size (>9k children of the root, i.e. 19 segments), > > and if I give less than 1200MiB RAM to Orbeon, it crashes with "GC > > overhead limit exceeded" around the 10th segment. It feels like it's > > keeping much more data during that loop than it needs. > > (I previously had it send the whole file to the DB in a batch, which > > would behave even worse). > > This is with Orbeon 3.8 CE. > > > > Kind regards, > > Ralf Jung > > > > Am Dienstag 14 Dezember 2010, um 10:03:25 schrieb ncrofts: > >> Hi Erik, > >> > >> The tests were done with the official 3.8 CE release, > >> orbeon-3.8.0.rc1.201005041622-CE. > >> > >> I've not tried with a recent nightly build, but with respect all > >> previous builds I've tried have been similarly memory intensive. I'm > >> not sure yet what the specific bottleneck(s) are and there is obviously > >> quite a lot of processing going on just to read from a db and process > >> XML, but it is curious that an XPL based application (that also > >> processes db/XML) seems lighter on the memory footprint. It does feel > >> rather a disproportionate overhead just to generate a few K's worth of > >> HTML and I can't help but feel something is not quite right. > >> > >> Regards, > >> Neil > > > > -- > > You receive this message as a subscriber of the [hidden email] mailing > > list. To unsubscribe: mailto:[hidden email] > > For general help: mailto:[hidden email]?subject=help > > OW2 mailing lists service home page: http://www.ow2.org/wws -- You receive this message as a subscriber of the [hidden email] mailing list. To unsubscribe: mailto:[hidden email] For general help: mailto:[hidden email]?subject=help OW2 mailing lists service home page: http://www.ow2.org/wws |
Administrator
|
Ralf,
Mmh, I fear that oxf:xforms-submission does not deal very well with very large files. It uses the XForms engine in the background, which stores a DOM in RAM, in particular. Can you use oxf:url-generator instead here? -Erik On Thu, Dec 16, 2010 at 7:11 AM, Ralf Jung <[hidden email]> wrote: > Hi Erik, > > thanks, that was helpful! > However, the data is sent to the DB using the xforms-submission processor. I > assume the reason for the large RAM usage is that the stream it creates is > cached, too, but how can I disable this? I could not find an option in the > processor nor in the submission doc. > > Kind regards, > Ralf Jung > > Am Donnerstag 16 Dezember 2010, um 06:45:42 schrieb Erik Bruchez: >> I would expect a 45 MB file to take at least 10x in RAM, so 450 MB. If >> that is just copied once or twice, that would explain why you run out >> of memory. >> >> Now that doesn't mean things can't be improved. >> >> Both the URL generator and the HTTP serializer have configurations to >> disable caching: >> >> * >> http://wiki.orbeon.com/forms/doc/developer-guide/processors-url-generator# >> TOC-Cache-control * >> http://www.orbeon.com/orbeon/doc/processors-serializers-http#d6e26 >> >> I would try to load and serialize large documents by setting >> use-local-cache to false in both cases. >> >> -Erik >> >> On Tue, Dec 14, 2010 at 1:45 AM, Ralf Jung <[hidden email]> wrote: >> > Hi, >> > >> > not sure if this is related, but I am also under the impression that >> > there is some memory issue somewhere: I am using an XPL pipe to load a >> > document, using XSLT to split the nodes from the root into segments of >> > max. 500 nodes, and then sending each of those segments to the DB. The >> > XML file in 45MiB in size (>9k children of the root, i.e. 19 segments), >> > and if I give less than 1200MiB RAM to Orbeon, it crashes with "GC >> > overhead limit exceeded" around the 10th segment. It feels like it's >> > keeping much more data during that loop than it needs. >> > (I previously had it send the whole file to the DB in a batch, which >> > would behave even worse). >> > This is with Orbeon 3.8 CE. >> > >> > Kind regards, >> > Ralf Jung >> > >> > Am Dienstag 14 Dezember 2010, um 10:03:25 schrieb ncrofts: >> >> Hi Erik, >> >> >> >> The tests were done with the official 3.8 CE release, >> >> orbeon-3.8.0.rc1.201005041622-CE. >> >> >> >> I've not tried with a recent nightly build, but with respect all >> >> previous builds I've tried have been similarly memory intensive. I'm >> >> not sure yet what the specific bottleneck(s) are and there is obviously >> >> quite a lot of processing going on just to read from a db and process >> >> XML, but it is curious that an XPL based application (that also >> >> processes db/XML) seems lighter on the memory footprint. It does feel >> >> rather a disproportionate overhead just to generate a few K's worth of >> >> HTML and I can't help but feel something is not quite right. >> >> >> >> Regards, >> >> Neil >> > >> > -- >> > You receive this message as a subscriber of the [hidden email] mailing >> > list. To unsubscribe: mailto:[hidden email] >> > For general help: mailto:[hidden email]?subject=help >> > OW2 mailing lists service home page: http://www.ow2.org/wws > > > -- > You receive this message as a subscriber of the [hidden email] mailing list. > To unsubscribe: mailto:[hidden email] > For general help: mailto:[hidden email]?subject=help > OW2 mailing lists service home page: http://www.ow2.org/wws > > -- You receive this message as a subscriber of the [hidden email] mailing list. To unsubscribe: mailto:[hidden email] For general help: mailto:[hidden email]?subject=help OW2 mailing lists service home page: http://www.ow2.org/wws |
Hi Erik,
> It uses the XForms engine in the background, which > stores a DOM in RAM, in particular. Good to know, that explains a lot. I think it would be a good idea to document such details in the processor documentation - i.e. whether the processor can work on a SAX-like "stream" of XML events or whether the whole file is loaded at once into a DOM. I need to send the data to a site using HTTP POST (the receiver is an eXist XQuery script). From what I gather, URL generator does the opposite. URL serializer says it can work with the http: protocol, but I can find no documentation about how the data is submitted (i.e. PUT, POST, whatever). Kind regards, Ralf Am Samstag 18 Dezember 2010, um 02:46:28 schrieb Erik Bruchez: > Ralf, > > Mmh, I fear that oxf:xforms-submission does not deal very well with > very large files. > > Can you use oxf:url-generator instead here? > > -Erik > > On Thu, Dec 16, 2010 at 7:11 AM, Ralf Jung <[hidden email]> wrote: > > Hi Erik, > > > > thanks, that was helpful! > > However, the data is sent to the DB using the xforms-submission > > processor. I assume the reason for the large RAM usage is that the > > stream it creates is cached, too, but how can I disable this? I could > > not find an option in the processor nor in the submission doc. > > > > Kind regards, > > Ralf Jung > > > > Am Donnerstag 16 Dezember 2010, um 06:45:42 schrieb Erik Bruchez: > >> I would expect a 45 MB file to take at least 10x in RAM, so 450 MB. If > >> that is just copied once or twice, that would explain why you run out > >> of memory. > >> > >> Now that doesn't mean things can't be improved. > >> > >> Both the URL generator and the HTTP serializer have configurations to > >> disable caching: > >> > >> * > >> http://wiki.orbeon.com/forms/doc/developer-guide/processors-url-generato > >> r# TOC-Cache-control * > >> http://www.orbeon.com/orbeon/doc/processors-serializers-http#d6e26 > >> > >> I would try to load and serialize large documents by setting > >> use-local-cache to false in both cases. > >> > >> -Erik > >> > >> On Tue, Dec 14, 2010 at 1:45 AM, Ralf Jung <[hidden email]> wrote: > >> > Hi, > >> > > >> > not sure if this is related, but I am also under the impression that > >> > there is some memory issue somewhere: I am using an XPL pipe to load a > >> > document, using XSLT to split the nodes from the root into segments of > >> > max. 500 nodes, and then sending each of those segments to the DB. The > >> > XML file in 45MiB in size (>9k children of the root, i.e. 19 > >> > segments), and if I give less than 1200MiB RAM to Orbeon, it crashes > >> > with "GC overhead limit exceeded" around the 10th segment. It feels > >> > like it's keeping much more data during that loop than it needs. > >> > (I previously had it send the whole file to the DB in a batch, which > >> > would behave even worse). > >> > This is with Orbeon 3.8 CE. > >> > > >> > Kind regards, > >> > Ralf Jung > >> > > >> > Am Dienstag 14 Dezember 2010, um 10:03:25 schrieb ncrofts: > >> >> Hi Erik, > >> >> > >> >> The tests were done with the official 3.8 CE release, > >> >> orbeon-3.8.0.rc1.201005041622-CE. > >> >> > >> >> I've not tried with a recent nightly build, but with respect all > >> >> previous builds I've tried have been similarly memory intensive. I'm > >> >> not sure yet what the specific bottleneck(s) are and there is > >> >> obviously quite a lot of processing going on just to read from a db > >> >> and process XML, but it is curious that an XPL based application > >> >> (that also processes db/XML) seems lighter on the memory footprint. > >> >> It does feel rather a disproportionate overhead just to generate a > >> >> few K's worth of HTML and I can't help but feel something is not > >> >> quite right. > >> >> > >> >> Regards, > >> >> Neil > >> > > >> > -- > >> > You receive this message as a subscriber of the [hidden email] > >> > mailing list. To unsubscribe: mailto:[hidden email] > >> > For general help: mailto:[hidden email]?subject=help > >> > OW2 mailing lists service home page: http://www.ow2.org/wws > > > > -- > > You receive this message as a subscriber of the [hidden email] mailing > > list. To unsubscribe: mailto:[hidden email] > > For general help: mailto:[hidden email]?subject=help > > OW2 mailing lists service home page: http://www.ow2.org/wws -- You receive this message as a subscriber of the [hidden email] mailing list. To unsubscribe: mailto:[hidden email] For general help: mailto:[hidden email]?subject=help OW2 mailing lists service home page: http://www.ow2.org/wws |
Administrator
|
>> It uses the XForms engine in the background, which
>> stores a DOM in RAM, in particular. > Good to know, that explains a lot. > I think it would be a good idea to document such details in the processor > documentation - i.e. whether the processor can work on a SAX-like "stream" of > XML events or whether the whole file is loaded at once into a DOM. Good point. I have used the opportunity to move the XForms submission processor doc to the wiki: http://wiki.orbeon.com/forms/doc/developer-guide/processors-xforms-submission and added a note about (non-)streaming here. Haven't updated other processors' documentation on streaming at this point, but we should do that in the future. > I need to send the data to a site using HTTP POST (the receiver is an eXist > XQuery script). From what I gather, URL generator does the opposite. URL > serializer says it can work with the http: protocol, but I can find no > documentation about how the data is submitted (i.e. PUT, POST, whatever). A bit of history: oxf:xforms-submission was created from the requirement of doing PUT/POST of documents. We contemplated the idea of upgrading the URL serializer (which is very limited now), and then decided instead to leverage xforms:submission by simply wrapping the XForms engine into a processor. This had benefits, but as you have seen one drawback is that at this point, this is not optimal especially for large documents. -Erik > > Kind regards, > Ralf > > Am Samstag 18 Dezember 2010, um 02:46:28 schrieb Erik Bruchez: >> Ralf, >> >> Mmh, I fear that oxf:xforms-submission does not deal very well with >> very large files. >> >> Can you use oxf:url-generator instead here? >> >> -Erik >> >> On Thu, Dec 16, 2010 at 7:11 AM, Ralf Jung <[hidden email]> wrote: >> > Hi Erik, >> > >> > thanks, that was helpful! >> > However, the data is sent to the DB using the xforms-submission >> > processor. I assume the reason for the large RAM usage is that the >> > stream it creates is cached, too, but how can I disable this? I could >> > not find an option in the processor nor in the submission doc. >> > >> > Kind regards, >> > Ralf Jung >> > >> > Am Donnerstag 16 Dezember 2010, um 06:45:42 schrieb Erik Bruchez: >> >> I would expect a 45 MB file to take at least 10x in RAM, so 450 MB. If >> >> that is just copied once or twice, that would explain why you run out >> >> of memory. >> >> >> >> Now that doesn't mean things can't be improved. >> >> >> >> Both the URL generator and the HTTP serializer have configurations to >> >> disable caching: >> >> >> >> * >> >> http://wiki.orbeon.com/forms/doc/developer-guide/processors-url-generato >> >> r# TOC-Cache-control * >> >> http://www.orbeon.com/orbeon/doc/processors-serializers-http#d6e26 >> >> >> >> I would try to load and serialize large documents by setting >> >> use-local-cache to false in both cases. >> >> >> >> -Erik >> >> >> >> On Tue, Dec 14, 2010 at 1:45 AM, Ralf Jung <[hidden email]> wrote: >> >> > Hi, >> >> > >> >> > not sure if this is related, but I am also under the impression that >> >> > there is some memory issue somewhere: I am using an XPL pipe to load a >> >> > document, using XSLT to split the nodes from the root into segments of >> >> > max. 500 nodes, and then sending each of those segments to the DB. The >> >> > XML file in 45MiB in size (>9k children of the root, i.e. 19 >> >> > segments), and if I give less than 1200MiB RAM to Orbeon, it crashes >> >> > with "GC overhead limit exceeded" around the 10th segment. It feels >> >> > like it's keeping much more data during that loop than it needs. >> >> > (I previously had it send the whole file to the DB in a batch, which >> >> > would behave even worse). >> >> > This is with Orbeon 3.8 CE. >> >> > >> >> > Kind regards, >> >> > Ralf Jung >> >> > >> >> > Am Dienstag 14 Dezember 2010, um 10:03:25 schrieb ncrofts: >> >> >> Hi Erik, >> >> >> >> >> >> The tests were done with the official 3.8 CE release, >> >> >> orbeon-3.8.0.rc1.201005041622-CE. >> >> >> >> >> >> I've not tried with a recent nightly build, but with respect all >> >> >> previous builds I've tried have been similarly memory intensive. I'm >> >> >> not sure yet what the specific bottleneck(s) are and there is >> >> >> obviously quite a lot of processing going on just to read from a db >> >> >> and process XML, but it is curious that an XPL based application >> >> >> (that also processes db/XML) seems lighter on the memory footprint. >> >> >> It does feel rather a disproportionate overhead just to generate a >> >> >> few K's worth of HTML and I can't help but feel something is not >> >> >> quite right. >> >> >> >> >> >> Regards, >> >> >> Neil >> >> > >> >> > -- >> >> > You receive this message as a subscriber of the [hidden email] >> >> > mailing list. To unsubscribe: mailto:[hidden email] >> >> > For general help: mailto:[hidden email]?subject=help >> >> > OW2 mailing lists service home page: http://www.ow2.org/wws >> > >> > -- >> > You receive this message as a subscriber of the [hidden email] mailing >> > list. To unsubscribe: mailto:[hidden email] >> > For general help: mailto:[hidden email]?subject=help >> > OW2 mailing lists service home page: http://www.ow2.org/wws > > > -- > You receive this message as a subscriber of the [hidden email] mailing list. > To unsubscribe: mailto:[hidden email] > For general help: mailto:[hidden email]?subject=help > OW2 mailing lists service home page: http://www.ow2.org/wws > > -- You receive this message as a subscriber of the [hidden email] mailing list. To unsubscribe: mailto:[hidden email] For general help: mailto:[hidden email]?subject=help OW2 mailing lists service home page: http://www.ow2.org/wws |
Hi Erik,
I just wanted to find out what the URL serializer actually does when it is given a http-URL (after all, the documentation says "The oxf:, file: and http: protocols allow writing."). However, it does not look like http is actually supported: 2010-12-21 12:10:35,426 ERROR org.orbeon.oxf.main.OPS - Exception at line 48, column 42 of oxf:/var/lib/tomcat6/webapps/orbeon/WEB- INF/resources/import/import.xpl (executing processor: name='{http://www.orbeon.com/oxf/processors}url-serializer') java.net.UnknownServiceException: protocol doesn't support output at java.net.URLConnection.getOutputStream(URLConnection.java:809) at org.orbeon.oxf.processor.serializer.URLSerializer.start(URLSerializer.java:80) at org.orbeon.oxf.processor.pipeline.PipelineProcessor$5.run(PipelineProcessor.java:647) at org.orbeon.oxf.processor.ProcessorImpl.executeChildren(ProcessorImpl.java:620) at org.orbeon.oxf.processor.pipeline.PipelineProcessor.start(PipelineProcessor.java:644) at org.orbeon.oxf.processor.pipeline.foreach.ConcreteForEachProcessor.start(ConcreteForEachProcessor.java:273) at org.orbeon.oxf.processor.pipeline.PipelineProcessor$5.run(PipelineProcessor.java:647) at org.orbeon.oxf.processor.ProcessorImpl.executeChildren(ProcessorImpl.java:620) at org.orbeon.oxf.processor.pipeline.PipelineProcessor.start(PipelineProcessor.java:644) at org.orbeon.oxf.pipeline.InitUtils.runProcessor(InitUtils.java:90) at org.orbeon.oxf.pipeline.PipelineEngineImpl.executePipeline(PipelineEngineImpl.java:31) at org.orbeon.oxf.main.OPS.start(OPS.java:191) at org.orbeon.oxf.main.OPS.main(OPS.java:207) Kind regards, Ralf Jung -- You receive this message as a subscriber of the [hidden email] mailing list. To unsubscribe: mailto:[hidden email] For general help: mailto:[hidden email]?subject=help OW2 mailing lists service home page: http://www.ow2.org/wws |
Administrator
|
Ralf,
Puzzling! That's probably explained by the fact that we almost never use oxf:url-serializer, and it has been a bit out of date for a while. The issue is probably that our implementation of HTTPURLConnection doesn't override getOutpuStream(). The default implementation in the JDK simply throws an exception. https://github.com/orbeon/orbeon-forms/blob/master/src/java/org/orbeon/oxf/resources/handler/HTTPURLConnection.java That shouldn't be too hard to fix if you want to give it a try. -Erik On Tue, Dec 21, 2010 at 3:16 AM, Ralf Jung <[hidden email]> wrote: > Hi Erik, > > I just wanted to find out what the URL serializer actually does when it is > given a http-URL (after all, the documentation says "The oxf:, file: and http: > protocols allow writing."). However, it does not look like http is actually > supported: > > 2010-12-21 12:10:35,426 ERROR org.orbeon.oxf.main.OPS - Exception at line 48, > column 42 of oxf:/var/lib/tomcat6/webapps/orbeon/WEB- > INF/resources/import/import.xpl (executing processor: > name='{http://www.orbeon.com/oxf/processors}url-serializer') > java.net.UnknownServiceException: protocol doesn't support output > at java.net.URLConnection.getOutputStream(URLConnection.java:809) > at > org.orbeon.oxf.processor.serializer.URLSerializer.start(URLSerializer.java:80) > at > org.orbeon.oxf.processor.pipeline.PipelineProcessor$5.run(PipelineProcessor.java:647) > at > org.orbeon.oxf.processor.ProcessorImpl.executeChildren(ProcessorImpl.java:620) > at > org.orbeon.oxf.processor.pipeline.PipelineProcessor.start(PipelineProcessor.java:644) > at > org.orbeon.oxf.processor.pipeline.foreach.ConcreteForEachProcessor.start(ConcreteForEachProcessor.java:273) > at > org.orbeon.oxf.processor.pipeline.PipelineProcessor$5.run(PipelineProcessor.java:647) > at > org.orbeon.oxf.processor.ProcessorImpl.executeChildren(ProcessorImpl.java:620) > at > org.orbeon.oxf.processor.pipeline.PipelineProcessor.start(PipelineProcessor.java:644) > at org.orbeon.oxf.pipeline.InitUtils.runProcessor(InitUtils.java:90) > at > org.orbeon.oxf.pipeline.PipelineEngineImpl.executePipeline(PipelineEngineImpl.java:31) > at org.orbeon.oxf.main.OPS.start(OPS.java:191) > at org.orbeon.oxf.main.OPS.main(OPS.java:207) > > Kind regards, > Ralf Jung > > > -- > You receive this message as a subscriber of the [hidden email] mailing list. > To unsubscribe: mailto:[hidden email] > For general help: mailto:[hidden email]?subject=help > OW2 mailing lists service home page: http://www.ow2.org/wws > > -- You receive this message as a subscriber of the [hidden email] mailing list. To unsubscribe: mailto:[hidden email] For general help: mailto:[hidden email]?subject=help OW2 mailing lists service home page: http://www.ow2.org/wws |
Thank you for your Email. Your message is important to me.
I am currently out of the office and will return on Tuesday, November 16.
During my absence, I will have intermittent access to Email. Please be assured that I will respond to your message as soon as I can.
If this is an emergency, please contact Michael Gitelman at [hidden email] for Architecture issues, Mahesh Manickam at [hidden email]
for Web Deployment issues, or Florin Bibire at [hidden email] for PHS issues. CONFIDENTIALITY NOTICE This e-mail message from Delphi Technology, Inc. is intended only for the individual or entity to which it is addressed. This e-mail may contain information that is privileged, confidential and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you received this e-mail by accident, please notify the sender immediately and destroy this e-mail and all copies of it. |
In reply to this post by Erik Bruchez
Hi Erik,
honestly, it's not worth it. This import is done once, maybe twice a month, and while it is certainly not perfect, I can live with it taking 1.5GiB of RAM and 30 minutes. I did not expect to meet so many issues on that way, but hey, that's life ;-) First thing I had to do to get it faster is to implement the splitting of the original document manually in some Java processor, as I doubt XSLT can work without a full DOM tree. And even then, url-serializer would not give me the server's reply to my POST request, which I need to see whether all went well. Thanks again for all your help, and Merry Christmas! Ralf Am Mittwoch 22 Dezember 2010, um 03:02:20 schrieb Erik Bruchez: > Ralf, > > Puzzling! That's probably explained by the fact that we almost never > use oxf:url-serializer, and it has been a bit out of date for a while. > > The issue is probably that our implementation of HTTPURLConnection > doesn't override getOutpuStream(). The default implementation in the > JDK simply throws an exception. > > https://github.com/orbeon/orbeon-forms/blob/master/src/java/org/orbeon/oxf/ > resources/handler/HTTPURLConnection.java > > That shouldn't be too hard to fix if you want to give it a try. > > -Erik -- You receive this message as a subscriber of the [hidden email] mailing list. To unsubscribe: mailto:[hidden email] For general help: mailto:[hidden email]?subject=help OW2 mailing lists service home page: http://www.ow2.org/wws |
In reply to this post by Ralf Jung
Thank you for your Email. Your message is important to me.
I am currently out of the office and will return on Tuesday, November 16.
During my absence, I will have intermittent access to Email. Please be assured that I will respond to your message as soon as I can.
If this is an emergency, please contact Michael Gitelman at [hidden email] for Architecture issues, Mahesh Manickam at [hidden email]
for Web Deployment issues, or Florin Bibire at [hidden email] for PHS issues. CONFIDENTIALITY NOTICE This e-mail message from Delphi Technology, Inc. is intended only for the individual or entity to which it is addressed. This e-mail may contain information that is privileged, confidential and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you received this e-mail by accident, please notify the sender immediately and destroy this e-mail and all copies of it. |
Free forum by Nabble | Edit this page |