High memory usage

Hello,
We are currently deploying a large form and we have a memory problem under heavy loads.
The form.xhtml contains about 15000 lines of code, mostly generated with Form Builder, and weights 1Mb.
This form loads a simple XML file at the beginning (xforms-ready event) for the pre-filling of the data instance. No complex Xpath operations are performed on this XML file.
The data instance is saved to an external eXist database.
 
We simulated 1 user who fills this form back and forth and saves it occasionally.
For this single user in this case the memory of the heap of Orbeon increases up to 950Mb, which seems huge for a single user and 1 form.
This memory problem increases when we stimulate more users. The application stops to respond at 450 users connected (error 500)
 
We followed the tips given here :
http://wiki.orbeon.com/forms/doc/developer-guide/admin/performance-tuning
but it had little effects on the memory footprint.
Some tips given in this mailing list about the JVM helped too but we still have a high memory usage.
 
We did a Java profiling with Yourkit with 10 users filling the form.
We observed the following which may needlessly consume the memory :
- 217 000 duplicated strings, most of them empty
- 54 000 duplicated arrays, most of them empty
- 280 000 arrays of objects with big number of null elements
We can provide the screenshots of Yourkit if needed.
 
Do you have any idea on where the memory over consumption comes from and what we can do to address it ?
The numerous bind, alert, message and label automatically generated by the Form Builder but not used (the tags are left empty) can be the cause ?
 
Here is our configuration :
JVM : Oracle Java 6
Tomcat : 6.0.37
Orbeon : 4.2 PE
Java options for Tomcat :
-xms : 1500m
-xmx : 1500m
-XX:NewSize=750m,
-XX:MaxNewSize=750m,
-XX:SurvivorRatio=6,
-XX:+UseConcMarkSweepGC ;
-XX:+CMSParallelRemarkEnabled
 
The tuning of properties-local.xml :
<property as="xs:boolean" name="oxf.xforms.minimal-resources" value="true"/>
<property as="xs:boolean" name="oxf.resources.versioned" value="true"/>
<property as="xs:integer" name="oxf.cache.size" value="800"/> 
<property as="xs:string" name="oxf.xforms.client.events.filter" value="DOMFocusIn DOMFocusOut"/> 
 
For the http-serializer processor we set <use-local-cache> at false.

Thanks for your help.

mn.

Hi,

Thanks, we are aware of this, and we do want to address this sooner rather than later. We have an umbrella bug here which we created recently:

https://github.com/orbeon/orbeon-forms/issues/1606

We improved whitespace usage a bit for 4.4, but that is only a small improvement:

http://blog.orbeon.com/2013/11/saving-memory-by-removing-unneeded.html
https://github.com/orbeon/orbeon-forms/issues/1252

One issue which is very clear is that very large form definitions take way too much memory, and because they are cached aggressively you can easily run out of memory just with large form definitions. Actual form sessions also do take memory, and that is also something to improve on.

-Erik

Thanks for your answer.
In there any chance that by removing the the empty tags generated by the Form Builder (alert, hint, help ..) we can reduce the memory consumption ?

A tiny chance that it will help a tiny bit, but most likely nothing big enough to make a real difference.

-Erik