Extremely slow xml big data import [load and save]

Hi all,

I'm creating a form with an xml file import functionality. I want to iterate
the xml file imported to obtain the corresponding rows of a grid table.
Each row is made up by a set of different input text (so for example, each
row has ten input text components) that are created dinamically with the
corresponding value from the xml file imported.

The structure of the xml file is the following:

/<nodeList>
        <node>
                <value1>test</value1>
                <value2>test</value2>
                <value3>test</value3>
                <value4>test</value4>
                <value5>test</value5>
                <value6>test</value6>
                <value7>test</value7>
                <value8>test</value8>
                <value9>test</value9>
                <value10>test</value10>
        </node>
        ...
        (with N nodes)
</nodeList>/

---------------------------------

So my task is to create N rows by these N nodes, dinamically.

Firstly I achieve this task with the xforms upload component and an action
with iterate attribute to loop through the xml file nodes loaded in a
separate instance: each iteration inserted a row in a repeated-grid (with a
template) and then did a set of setvalue to load the values into each input
text component of each row. Here the logic:

/<xf:instance id="upload">
        <serialized mediatype="application/xml" filename=""/>
</xf:instance>
<xf:bind ref="instance('upload')" type="xs:base64Binary"/>
<xf:submission id="upload-submission" ref="instance('upload')"
validate="false"
                           relevant="false"
                           method="post"
                           replace="none"
                           resource="echo:"/>
<xf:action id="upload-binding">
       
       
        <xf:action event="xforms-submit" ev:observer="upload-submission">
               
                <xf:insert ref="instance('fr-service-response-instance')"
                                 
origin="saxon:parse(saxon:base64Binary-to-string(xs:base64Binary(instance('upload')),
'UTF-8'))"/>
        </xf:action>
        <xf:action event="xforms-submit-done" ev:observer="upload-submission">
                <xf:delete ref="instance('fr-form-instance')//nodes-iteration"/>
                <xf:action iterate="instance('fr-service-response-instance')//*:node">
                        <xf:insert context="instance('fr-form-instance')//nodes" ref="*"
                                           origin="instance('nodes-template')"/>
                        <xf:setvalue
ref="instance('fr-form-instance')//nodes-iteration[last()]/value1"
                                                 value="context()//*:value1"/>
                        <xf:setvalue
ref="instance('fr-form-instance')//nodes-iteration[last()]/value2"
                                                 value="context()//*:value2"/>
                        <xf:setvalue
ref="instance('fr-form-instance')//nodes-iteration[last()]/value3"
                                                 value="context()//*:value3"/>
                        ...
                        (with N xf:setvalue)
                </xf:action>
        </xf:action>
</xf:action>

...

<xf:upload id="upload-div" ref="instance('upload')">
    <xf:filename ref="@filename"/>
    <xf:mediatype ref="@mediatype"/>
    <xf:send event="xxforms-upload-done" submission="upload-submission"/>
</xf:upload>
/
---------------------------------

This solution was very slow, so I tried with the xforms repeat component and
inserting the whole xml file nodes directly into the instance associated
with the repeat component.
This solution instead was faster, but with 5000 nodes our form loading
components was very slow too (1 minute and 42 seconds).
With 5000 nodes, only the save action spends 7 minutes and 45 seconds to
finish.
The loading page of a form with these 5000 nodes saved yet spends 2 minutes
and 26 seconds to finish.
I need to reduce these times.
I also try with different environments (with better hardware) or
configurations, but the times are almost the same.

If you need I can attach my form code and xml file.

Thanks

Roberto

--
Sent from: http://discuss.orbeon.com/

--
You received this message because you are subscribed to the Google Groups "Orbeon Forms" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].

Hi Roberto,

Since you've also posted this question on Stack Overflow, I've answered it
there. Feel free to post a comment on my answer if you have any follow-up
question or feedback on the technique I'm suggesting there.

https://stackoverflow.com/questions/47866814/orbeon-extremely-slow-xml-big-data-import-load-and-save

Alex

-----
--
Follow Orbeon on Twitter: @orbeon
Follow me on Twitter: @avernet
--
Sent from: http://discuss.orbeon.com/

--
You received this message because you are subscribed to the Google Groups "Orbeon Forms" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].