Large XML Doc - Paginate ?
Locked
 
|
I have a large XML doc that I am attempting to edit with an Orbeon XML form.
The doc is too large to load all at once. Is there a way to edit a portion of the XML doc without loading the whole doc. The document is store in an exist db. |
|
|
I am out of the office until 26th July *********************************************************************************************** This email, including any attachment, is confidential and may be legally privileged. If you are not the intended recipient or if you have received this email in error, please inform the sender immediately by reply and delete all copies from your system. Do not retain, copy, disclose, distribute or otherwise use any of its contents.
Whilst we have taken reasonable precautions to ensure that this email has been swept for computer viruses, we cannot guarantee that this email does not contain such material and we therefore advise you to carry out your own virus checks. We do not accept liability for any damage or losses sustained as a result of such material.
Please note that incoming and outgoing email communications passing through our IT systems may be monitored and/or intercepted by us solely to determine whether the content is business related and compliant with company standards. *********************************************************************************************** The Stationery Office Limited is registered in England No. 3049649 at 10 Eastbourne Terrace, London, W2 6LG
|
|
|
In reply to this post by lkbrown
Yes, I've done this with numerous documents that range from several to about 10 MB. I have used xforms:group to match on a particular XPath expression so that only nodes matching a particular @id are rendered into a form. In order to specify the @id, I have passed a value as a URL parameter. I think it is also possible to use the XPL framework to extract out the node you want to render in a form with the eXist query language, and pass only that node into an instance in your form. This is probably the most memory-efficient way to render smaller forms derived from large XML files, but I haven't gone that far because I find the xforms:group method within the XForms app to be acceptably fast.
Ethan On Mon, Jul 19, 2010 at 2:32 PM, lkbrown <[hidden email]> wrote:
-- You receive this message as a subscriber of the [hidden email] mailing list. To unsubscribe: mailto:[hidden email] For general help: mailto:[hidden email]?subject=help OW2 mailing lists service home page: http://www.ow2.org/wws |
|
|
I am out of the office until 26th July *********************************************************************************************** This email, including any attachment, is confidential and may be legally privileged. If you are not the intended recipient or if you have received this email in error, please inform the sender immediately by reply and delete all copies from your system. Do not retain, copy, disclose, distribute or otherwise use any of its contents.
Whilst we have taken reasonable precautions to ensure that this email has been swept for computer viruses, we cannot guarantee that this email does not contain such material and we therefore advise you to carry out your own virus checks. We do not accept liability for any damage or losses sustained as a result of such material.
Please note that incoming and outgoing email communications passing through our IT systems may be monitored and/or intercepted by us solely to determine whether the content is business related and compliant with company standards. *********************************************************************************************** The Stationery Office Limited is registered in England No. 3049649 at 10 Eastbourne Terrace, London, W2 6LG
|
|
|
In reply to this post by Ethan Gruber
I gave xforms:group a try … but no luck. My page still has to
load the 10 MB file in my browser; Even if I am rendering a small number of
fields. I may have to try the eXist approach. But no clue where to
start. From: Ethan Gruber [via Orbeon Forms (ops-users)]
[mailto:[hidden email]] Yes, I've done this with numerous documents that range
from several to about 10 MB. I have used xforms:group to match on a
particular XPath expression so that only nodes matching a particular @id are
rendered into a form. In order to specify the @id, I have passed a value
as a URL parameter. I think it is also possible to use the XPL framework
to extract out the node you want to render in a form with the eXist query
language, and pass only that node into an instance in your form. This is
probably the most memory-efficient way to render smaller forms derived from
large XML files, but I haven't gone that far because I find the xforms:group
method within the XForms app to be acceptably fast. On Mon, Jul 19, 2010 at 2:32 PM,
lkbrown <[hidden email]> wrote:
View message @ http://orbeon-forms-ops-users.24843.n4.nabble.com/Large-XML-Doc-Paginate-tp2294425p2294435.html
|
|
|
I am out of the office until 26th July *********************************************************************************************** This email, including any attachment, is confidential and may be legally privileged. If you are not the intended recipient or if you have received this email in error, please inform the sender immediately by reply and delete all copies from your system. Do not retain, copy, disclose, distribute or otherwise use any of its contents.
Whilst we have taken reasonable precautions to ensure that this email has been swept for computer viruses, we cannot guarantee that this email does not contain such material and we therefore advise you to carry out your own virus checks. We do not accept liability for any damage or losses sustained as a result of such material.
Please note that incoming and outgoing email communications passing through our IT systems may be monitored and/or intercepted by us solely to determine whether the content is business related and compliant with company standards. *********************************************************************************************** The Stationery Office Limited is registered in England No. 3049649 at 10 Eastbourne Terrace, London, W2 6LG
|
Re: RE: Large XML Doc - Paginate ?
|
|
In reply to this post by lkbrown
By chance, do you have the instance inspector widget active? That can dramatically increase the load time for a form, even if you are rendering only a small portion of the file.
On Mon, Jul 19, 2010 at 6:02 PM, lkbrown <[hidden email]> wrote:
-- You receive this message as a subscriber of the [hidden email] mailing list. To unsubscribe: mailto:[hidden email] For general help: mailto:[hidden email]?subject=help OW2 mailing lists service home page: http://www.ow2.org/wws |
|
|
I am out of the office until 26th July *********************************************************************************************** This email, including any attachment, is confidential and may be legally privileged. If you are not the intended recipient or if you have received this email in error, please inform the sender immediately by reply and delete all copies from your system. Do not retain, copy, disclose, distribute or otherwise use any of its contents.
Whilst we have taken reasonable precautions to ensure that this email has been swept for computer viruses, we cannot guarantee that this email does not contain such material and we therefore advise you to carry out your own virus checks. We do not accept liability for any damage or losses sustained as a result of such material.
Please note that incoming and outgoing email communications passing through our IT systems may be monitored and/or intercepted by us solely to determine whether the content is business related and compliant with company standards. *********************************************************************************************** The Stationery Office Limited is registered in England No. 3049649 at 10 Eastbourne Terrace, London, W2 6LG
|
Re: Large XML Doc - Paginate ?
Administrator
|
In reply to this post by lkbrown
Lyndon,
As a rule of thumb, count that keeping in memory a live DOM for the XForms instance take about 10x the amount of memory used by the serialized XML. So if the XML document takes 10 MB on disk when serialized, it might take in the order of 100 MB of memory. If you notice that memory consumption goes up by more than that, it means that there is another problem. Using 100 MB of heap for a single instance of your form might work if you have a large enough heap, and very few users on your system. But most likely (also for performance reasons), you'll want to extract from eXist a smaller portion of the document you want to edit, and update just that portion when users save their changes. Alex On Monday, July 19, 2010, lkbrown <[hidden email]> wrote: > > > > > > > > > > > > > > > > > > > > I gave xforms:group a try … but no luck. My page still has to > load the 10 MB file in my browser; Even if I am rendering a small number of > fields. > > > > I may have to try the eXist approach. But no clue where to > start. > > > > > > > > From: Ethan Gruber [via Orbeon Forms (ops-users)] > [mailto:[hidden email] <http://user/SendEmail.jtp?type=node&node=2294681&i=0>] > Sent: Monday, July 19, 2010 2:45 PM > To: Brown, Lyndon K. > Subject: Re: Large XML Doc - Paginate ? > > > > > > Yes, I've done this with numerous documents that range > from several to about 10 MB. I have used xforms:group to match on a > particular XPath expression so that only nodes matching a particular @id are > rendered into a form. In order to specify the @id, I have passed a value > as a URL parameter. I think it is also possible to use the XPL framework > to extract out the node you want to render in a form with the eXist query > language, and pass only that node into an instance in your form. This is > probably the most memory-efficient way to render smaller forms derived from > large XML files, but I haven't gone that far because I find the xforms:group > method within the XForms app to be acceptably fast. > > Ethan > > > > On Mon, Jul 19, 2010 at 2:32 PM, > lkbrown <[hidden email] <http://user/SendEmail.jtp?type=node&node=2294435&i=0&by-user=t>> wrote: > > > I have a large XML doc that I am attempting to edit with an Orbeon XML form. > > The doc is too large to load all at once. Is there a way to edit a portion > of the XML doc without loading the whole doc. > > The document is store in an exist db. > -- > View this message in context: http://orbeon-forms-ops-users.24843.n4.nabble.com/Large-XML-Doc-Paginate-tp2294425p2294425.html > Sent from the Orbeon Forms (ops-users) mailing list archive at Nabble.com. > > > -- > You receive this message as a subscriber of the [hidden > email] <http://user/SendEmail.jtp?type=node&node=2294435&i=1&by-user=t> mailing list. > To unsubscribe: mailto:[hidden email] <http://user/SendEmail.jtp?type=node&node=2294435&i=2&by-user=t> > For general help: mailto:[hidden > email] <http://user/SendEmail.jtp?type=node&node=2294435&i=3&by-user=t>?subject=help > OW2 mailing lists service home page: http://www.ow2.org/wws > > > > > > > -- > You receive this message as a subscriber of the [hidden > email] <http://user/SendEmail.jtp?type=node&node=2294435&i=4&by-user=t> mailing list. > To unsubscribe: mailto:[hidden email] <http://user/SendEmail.jtp?type=node&node=2294435&i=5&by-user=t> > For general help: mailto:[hidden > email] <http://user/SendEmail.jtp?type=node&node=2294435&i=6&by-user=t>?subject=help > OW2 mailing lists service home page: http://www.ow2.org/wws > > > > > > > > > > > > View message @ http://orbeon-forms-ops-users.24843.n4.nabble.com/Large-XML-Doc-Paginate-tp2294425p2294435.html > > To unsubscribe from Large XML Doc - Paginate ?, click > here. > > > > > > > > > > > > > > View this message in context: RE: Large XML Doc - Paginate ? <http://orbeon-forms-ops-users.24843.n4.nabble.com/Large-XML-Doc-Paginate-tp2294425p2294681.html> > Sent from the Orbeon Forms (ops-users) mailing list archive <http://orbeon-forms-ops-users.24843.n4.nabble.com/> at Nabble.com. > Orbeon Forms - Web forms, open-source, for the Enterprise - http://www.orbeon.com/ My Twitter: http://twitter.com/avernet -- You receive this message as a subscriber of the [hidden email] mailing list. To unsubscribe: mailto:[hidden email] For general help: mailto:[hidden email]?subject=help OW2 mailing lists service home page: http://www.ow2.org/wws
--
Follow Orbeon on Twitter: @orbeon Follow me on Twitter: @avernet |
|
|
I am out of the office until 26th July *********************************************************************************************** This email, including any attachment, is confidential and may be legally privileged. If you are not the intended recipient or if you have received this email in error, please inform the sender immediately by reply and delete all copies from your system. Do not retain, copy, disclose, distribute or otherwise use any of its contents.
Whilst we have taken reasonable precautions to ensure that this email has been swept for computer viruses, we cannot guarantee that this email does not contain such material and we therefore advise you to carry out your own virus checks. We do not accept liability for any damage or losses sustained as a result of such material.
Please note that incoming and outgoing email communications passing through our IT systems may be monitored and/or intercepted by us solely to determine whether the content is business related and compliant with company standards. *********************************************************************************************** The Stationery Office Limited is registered in England No. 3049649 at 10 Eastbourne Terrace, London, W2 6LG
|
|
|
In reply to this post by Alessandro Vernet
Alex,
You recommend that "I ... extract from eXist a smaller portion of the document you want to edit, and update just that portion when users save their changes." Do you have any sample code for retrieving, editing, and saving a portion of a document? Thanks, L |
|
|
In reply to this post by Ethan Gruber
Ethan,
Removing the instance inspector widget did speed things up as well. Thanks for that comment. |
Re: Large XML Doc - Paginate ?
|
Administrator
|
In reply to this post by Alessandro Vernet
Lyndon,
You would be using the eXist REST API to get the document by sending an XQuery to eXist to get the subset you are interested in. You can either POST the query if it is more complicated, or for simple queries send it in the _query parameter. See: http://exist.sourceforge.net/devguide_rest.html#rest Then you do the same to update the document, except this time you'll always do a POST, and use the XQuery update extensions: http://exist.sourceforge.net/update_ext.html Alex On Mon, Jul 19, 2010 at 7:15 PM, Alessandro Vernet <[hidden email]> wrote: > Lyndon, > > As a rule of thumb, count that keeping in memory a live DOM for the > XForms instance take about 10x the amount of memory used by the > serialized XML. So if the XML document takes 10 MB on disk when > serialized, it might take in the order of 100 MB of memory. If you > notice that memory consumption goes up by more than that, it means > that there is another problem. > > Using 100 MB of heap for a single instance of your form might work if > you have a large enough heap, and very few users on your system. But > most likely (also for performance reasons), you'll want to extract > from eXist a smaller portion of the document you want to edit, and > update just that portion when users save their changes. > > Alex > > On Monday, July 19, 2010, lkbrown <[hidden email]> wrote: >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> I gave xforms:group a try … but no luck. My page still has to >> load the 10 MB file in my browser; Even if I am rendering a small number of >> fields. >> >> >> >> I may have to try the eXist approach. But no clue where to >> start. >> >> >> >> >> >> >> >> From: Ethan Gruber [via Orbeon Forms (ops-users)] >> [mailto:[hidden email] <http://user/SendEmail.jtp?type=node&node=2294681&i=0>] >> Sent: Monday, July 19, 2010 2:45 PM >> To: Brown, Lyndon K. >> Subject: Re: Large XML Doc - Paginate ? >> >> >> >> >> >> Yes, I've done this with numerous documents that range >> from several to about 10 MB. I have used xforms:group to match on a >> particular XPath expression so that only nodes matching a particular @id are >> rendered into a form. In order to specify the @id, I have passed a value >> as a URL parameter. I think it is also possible to use the XPL framework >> to extract out the node you want to render in a form with the eXist query >> language, and pass only that node into an instance in your form. This is >> probably the most memory-efficient way to render smaller forms derived from >> large XML files, but I haven't gone that far because I find the xforms:group >> method within the XForms app to be acceptably fast. >> >> Ethan >> >> >> >> On Mon, Jul 19, 2010 at 2:32 PM, >> lkbrown <[hidden email] <http://user/SendEmail.jtp?type=node&node=2294435&i=0&by-user=t>> wrote: >> >> >> I have a large XML doc that I am attempting to edit with an Orbeon XML form. >> >> The doc is too large to load all at once. Is there a way to edit a portion >> of the XML doc without loading the whole doc. >> >> The document is store in an exist db. >> -- >> View this message in context: http://orbeon-forms-ops-users.24843.n4.nabble.com/Large-XML-Doc-Paginate-tp2294425p2294425.html >> Sent from the Orbeon Forms (ops-users) mailing list archive at Nabble.com. >> >> >> -- >> You receive this message as a subscriber of the [hidden >> email] <http://user/SendEmail.jtp?type=node&node=2294435&i=1&by-user=t> mailing list. >> To unsubscribe: mailto:[hidden email] <http://user/SendEmail.jtp?type=node&node=2294435&i=2&by-user=t> >> For general help: mailto:[hidden >> email] <http://user/SendEmail.jtp?type=node&node=2294435&i=3&by-user=t>?subject=help >> OW2 mailing lists service home page: http://www.ow2.org/wws >> >> >> >> >> >> >> -- >> You receive this message as a subscriber of the [hidden >> email] <http://user/SendEmail.jtp?type=node&node=2294435&i=4&by-user=t> mailing list. >> To unsubscribe: mailto:[hidden email] <http://user/SendEmail.jtp?type=node&node=2294435&i=5&by-user=t> >> For general help: mailto:[hidden >> email] <http://user/SendEmail.jtp?type=node&node=2294435&i=6&by-user=t>?subject=help >> OW2 mailing lists service home page: http://www.ow2.org/wws >> >> >> >> >> >> >> >> >> >> >> >> View message @ http://orbeon-forms-ops-users.24843.n4.nabble.com/Large-XML-Doc-Paginate-tp2294425p2294435.html >> >> To unsubscribe from Large XML Doc - Paginate ?, click >> here. >> >> >> >> >> >> >> >> >> >> >> >> >> >> View this message in context: RE: Large XML Doc - Paginate ? <http://orbeon-forms-ops-users.24843.n4.nabble.com/Large-XML-Doc-Paginate-tp2294425p2294681.html> >> Sent from the Orbeon Forms (ops-users) mailing list archive <http://orbeon-forms-ops-users.24843.n4.nabble.com/> at Nabble.com. >> > > -- > Orbeon Forms - Web forms, open-source, for the Enterprise - > http://www.orbeon.com/ > My Twitter: http://twitter.com/avernet > -- Orbeon Forms - Web forms, open-source, for the Enterprise - http://www.orbeon.com/ My Twitter: http://twitter.com/avernet -- You receive this message as a subscriber of the [hidden email] mailing list. To unsubscribe: mailto:[hidden email] For general help: mailto:[hidden email]?subject=help OW2 mailing lists service home page: http://www.ow2.org/wws
--
Follow Orbeon on Twitter: @orbeon Follow me on Twitter: @avernet |
|
|
Alex,
Seems like a good approach. I plan on giving it a try over the next few days. Let's say my XML document consist of a listing of 100 books and I want to edit 10 books at a time. Reviewing the Rest API documentation, I see how I can retrieve these 10 books relatively easily. I think my challenge will be persisting the changes. I want to make various changes to each of the books in my Xform and then persist it back to the db. 1) I'll need to implement some kind of for loop for the 10 books so I can persist the changes to the db. 2) I'll need a way to match the edit entry against the original entry. Since any of the fields of the record can change, I'll need to match it against the original db entry based on something. I'm thinking position index....but this might not work because the xml doc in my orbeon form will not have the original position index. For instance, if i retrieve and edit records 90 to 100. The index of those items in my xform will be 1 to 10. When I try and update the db, I will replace the wrong records..1 to 10...not 90 to 100. any thoughts? |
Re: Large XML Doc - Paginate ?
|
Administrator
|
Lyndon,
Regarding the update, yes, you will need to implement some kind of loop, but make sure to only issue on query to eXist to update all the records. The eXist update extension allow you to do all this in one query, and you will need the "some kind of loop" to build that query. To know what to update, you'll need to have an id for each "record" (book in your exemple) you want to update. Alex On Friday, August 6, 2010, lkbrown <[hidden email]> wrote: > > Alex, > > Seems like a good approach. I plan on giving it a try over the next few > days. > > Let's say my XML document consist of a listing of 100 books and I want to > edit 10 books at a time. > Reviewing the Rest API documentation, I see how I can retrieve these 10 > books relatively easily. > > I think my challenge will be persisting the changes. I want to make various > changes to each of the books in my Xform and then persist it back to the db. > > 1) I'll need to implement some kind of for loop for the 10 books so I can > persist the changes to the db. > 2) I'll need a way to match the edit entry against the original entry. > > Since any of the fields of the record can change, I'll need to match it > against the original db entry based on something. I'm thinking position > index....but this might not work because the xml doc in my orbeon form will > not have the original position index. For instance, if i retrieve and edit > records 90 to 100. The index of those items in my xform will be 1 to 10. > When I try and update the db, I will replace the wrong records..1 to > 10...not 90 to 100. > > any thoughts? > -- > View this message in context: http://orbeon-forms-ops-users.24843.n4.nabble.com/Large-XML-Doc-Paginate-tp2294425p2316955.html > Sent from the Orbeon Forms (ops-users) mailing list archive at Nabble.com. > Orbeon Forms - Web forms, open-source, for the Enterprise - http://www.orbeon.com/ My Twitter: http://twitter.com/avernet -- You receive this message as a subscriber of the [hidden email] mailing list. To unsubscribe: mailto:[hidden email] For general help: mailto:[hidden email]?subject=help OW2 mailing lists service home page: http://www.ow2.org/wws
--
Follow Orbeon on Twitter: @orbeon Follow me on Twitter: @avernet |
|
|
Still not quite sure:
how i can pass all those id's in a single query. Or how to loop through all the change rows. Can you help me out with a sample query? |
Re: Re: Large XML Doc - Paginate ?
|
Administrator
|
Lyndon,
I imagine you will be building a query that looks like: update replace /books/book[@isbn = '123'] with <book>...</book>, update replace /books/book[@isbn = '456'] with <book>...</book>, ... Alex On Mon, Aug 9, 2010 at 9:10 AM, lkbrown <[hidden email]> wrote: > > Still not quite sure: > > how i can pass all those id's in a single query. > Or how to loop through all the change rows. > > Can you help me out with a sample query? > -- > View this message in context: http://orbeon-forms-ops-users.24843.n4.nabble.com/Large-XML-Doc-Paginate-tp2294425p2318724.html > Sent from the Orbeon Forms (ops-users) mailing list archive at Nabble.com. > > > -- > You receive this message as a subscriber of the [hidden email] mailing list. > To unsubscribe: mailto:[hidden email] > For general help: mailto:[hidden email]?subject=help > OW2 mailing lists service home page: http://www.ow2.org/wws > > -- Orbeon Forms - Web forms, open-source, for the Enterprise - http://www.orbeon.com/ My Twitter: http://twitter.com/avernet -- You receive this message as a subscriber of the [hidden email] mailing list. To unsubscribe: mailto:[hidden email] For general help: mailto:[hidden email]?subject=help OW2 mailing lists service home page: http://www.ow2.org/wws
--
Follow Orbeon on Twitter: @orbeon Follow me on Twitter: @avernet |
|
|
aargg...
seems to be a bug with the Exist XUpdate extensions. The functionality is inconsistent. Update aren't being performed. |
|
|
In reply to this post by Alessandro Vernet
Alex,
You've been very helpful. Quick Question: I am querying the exist database using the following xquery. for $found in doc("xmldb:exist:///db/ARADS/data/scenario/albums.xml")//albums return <result>{$found}</result> The result set comesback empty; even though I know there are nodes that match that criteria. In fact, any xpath expression that I append to the end of the doc function fails to return results. With the exception of /* . Do you have any clue why I can't retrieve any nodes using this notation? The following fine, but I need to query a document: let $document := <store><albums><album></album></albums></store> return <result>{$document/albums}</result> |
Re: Re: Re: Large XML Doc - Paginate ?
|
Administrator
|
Lyndon,
Not sure what's the problem. Maybe someone can help you on the eXist list? -Erik On Mon, Aug 16, 2010 at 2:35 PM, lkbrown <[hidden email]> wrote: > > Alex, > > You've been very helpful. > > Quick Question: > > I am querying the exist database using the following xquery. > > for $found in > doc("xmldb:exist:///db/ARADS/data/scenario/albums.xml")//albums > return <result>{$found}</result> > > The result set comesback empty; even though I know there are nodes that > match that criteria. > In fact, any xpath expression that I append to the end of the doc function > fails to return results. With the exception of /* . > > Do you have any clue why I can't retrieve any nodes using this notation? > > The following fine, but I need to query a document: > > let $document := <store><albums><album></album></albums></store> > return <result>{$document/albums}</result> > > -- > View this message in context: http://orbeon-forms-ops-users.24843.n4.nabble.com/Large-XML-Doc-Paginate-tp2294425p2327514.html > Sent from the Orbeon Forms (ops-users) mailing list archive at Nabble.com. > > > -- > You receive this message as a subscriber of the [hidden email] mailing list. > To unsubscribe: mailto:[hidden email] > For general help: mailto:[hidden email]?subject=help > OW2 mailing lists service home page: http://www.ow2.org/wws > > -- You receive this message as a subscriber of the [hidden email] mailing list. To unsubscribe: mailto:[hidden email] For general help: mailto:[hidden email]?subject=help OW2 mailing lists service home page: http://www.ow2.org/wws |
| Free forum by Nabble | Edit this page |