OPS and memory management
Locked
 
|
I have developed a web app in OPS that started to run out of memory recently after I added certain functionality that involves manipulating large XML files using XSLT processor. I think I can deal with that issue by switching to SAX processing (I hope), but I performed some testing which brought up some additional issues I would like to discuss. I have specified 1 GB memory to the JVM Tomcat is running in and started to invoke some functions in the application while watching the Tomcat process memory allocation (Using this on Windows 2003 server). The table below shows the actions I took and the chunk of memory used by Tomcat after that action. It looks to me like there are some problems with re-claiming memory by either the JVM, Tomcat or OPS (My code is perfect so that is out of the picture :-) ).
The actions described will not tell you much, but if you look - for example - at the line: log off and back on 769,216 I would expect the memory allocation to drop to much lower level after I logged out off the app. Also, the line: downloaded all Vct data (CSV) 1,055,000 (29,049 records) ...shows that after I performed that action (retrieving 7.2 MB of data from DB, manipulating it as XML and finally serializing into a text file) the memory occupied jumped to the maximum I set for the JVM to use and did not drop after the operation was over. I do not know if this is JVM (using version 1.5.0.10), Tomcat (using version 5.5.4) or OPS (v. 3.0.beta4.200512072017) issue or the methodology I chose for my test? Thanks A. after login 156,720 after loading POS interface 204,000 after switching to "Summaries" 216,128 after loading POS interface 264,676 after switching to "Summaries and back 339,560 after switching to "Summaries and back 420,000 after switching to "Summaries and back 428,088 after switching to "Summaries and back 521,708 after switching to "Summaries and back 577,964 after switching to "Summaries and back 654,668 after switching to "Summaries and back 768,816 log off and back on 769,216 after switching to "Summaries and back 767,716 after switching to "Summaries and back 804,152 after switching to "Summaries and back 884,068 after switching to "Summaries and back 732,476 after switching to "Summaries and back 732,492 logged-in as another user 732,488 after loading POS interface 732,580 after switching to "Summaries and back 731,940 selected subset of records in Generic 640,000 kkm session downloaded report in CSV 643,296 downloaded all Vct data (CSV) 1,055,000 (29,049 records) loaded all records 1,082,580 waited longer than usual for refresh after switching to "Summaries and back 1,083,656 kkm session downloaded all KKM data (CSV) JVM crash ______________ SEVERE: Exception invoking periodic operation: java.lang.OutOfMemoryError: Java heap space Jul 13, 2006 9:03:54 PM org.apache.catalina.core.StandardWrapperValve invoke SEVERE: Servlet.service() for servlet ops-main-servlet threw exception java.lang.OutOfMemoryError: Java heap space -- You receive this message as a subscriber of the [hidden email] mailing list. To unsubscribe: mailto:[hidden email] For general help: mailto:[hidden email]?subject=help ObjectWeb mailing lists service home page: http://www.objectweb.org/wws |
processing large XML files ( was Re: OPS and memory management)
|
|
Hello list members, To deal with the memory and processing time issues I decided to take a look at the oxf:stx processor. Now I have re-coded my XPL and did some tests. I found out that with the XML transformation implemented in STX I get the download dialog box in about 10 seconds (a huge improvement over using XSLT) and the download speed also starts at a decent speed, but over time the number of bytes received per second drops to "modem speeds". Not sure if this has anything to do with the size of my XML (~ 7MB), the way I coded the style sheet or the oxf:stx processor. The result average download speed was 6 kb/sec (the slowest link between my web browser and the server has bandwidth of cca 500 kbps). In this particular case I need a quite simple transformation - am converting something like this: <a> <b1> <c11/> <c21/> <c31/> <c41/> </b1> <b2> <c12/> <c22/> <c32/> <c42/> </b2> .... .... <bn> <c1n/> <c2n/> <c3n/> <c4n/> </bn> </a> ...to a CSV file where every bn row corresponds to a single row: "c11","c21","c31","c41" c12","c22","c32","c42" .... .... c1n","c2n","c3n","c4n" I wonder what other members of this list use to process large XML files? A On Jul 13, 2006, at 11:27 PM, Alexander Zatko wrote: I have developed a web app in OPS that started to run out of memory recently after I added certain functionality that involves manipulating large XML files using XSLT processor. I think I can deal with that issue by switching to SAX processing (I hope), but I performed some testing which brought up some additional issues I would like to discuss. I have specified 1 GB memory to the JVM Tomcat is running in and started to invoke some functions in the application while watching the Tomcat process memory allocation (Using this on Windows 2003 server). The table below shows the actions I took and the chunk of memory used by Tomcat after that action. It looks to me like there are some problems with re-claiming memory by either the JVM, Tomcat or OPS (My code is perfect so that is out of the picture :-) ). -- You receive this message as a subscriber of the [hidden email] mailing list. To unsubscribe: mailto:[hidden email] For general help: mailto:[hidden email]?subject=help ObjectWeb mailing lists service home page: http://www.objectweb.org/wws |
Re: OPS and memory management
Administrator
|
In reply to this post by Alexander Žaťko
Hi Alexander,
The VM does not necessarily return to the OS memory that has been allocated by the OS to the VM, but that is not used anymore by the VM to store a live Java object. Instead of looking at how much memory is used by the VM at the OS-level, it is better to look at how much memory on the heap is used. You can do this simply with the -verbosegc VM option. When you use that option, you will see that the size of the heap goes up to its max size, and then comes back down when a full garbage collection is done. The full GC are particularly interesting: you want to make sure that a good amount of memory is freed when they happen and that you don't have full GC happening all the time. If they do, it can be a sign that the heap size is too small or that there is a leak in the application. I hope this helps, Alex On 7/13/06, Alexander Zatko <[hidden email]> wrote: > > I have developed a web app in OPS that started to run out of memory recently > after I added certain functionality that involves manipulating large XML > files using XSLT processor. I think I can deal with that issue by switching > to SAX processing (I hope), but I performed some testing which brought up > some additional issues I would like to discuss. I have specified 1 GB memory > to the JVM Tomcat is running in and started to invoke some functions in the > application while watching the Tomcat process memory allocation (Using this > on Windows 2003 server). The table below shows the actions I took and the > chunk of memory used by Tomcat after that action. It looks to me like there > are some problems with re-claiming memory by either the JVM, Tomcat or OPS > (My code is perfect so that is out of the picture :-) ). > > The actions described will not tell you much, but if you look - for example > - at the line: > > log off and back on 769,216 > > I would expect the memory allocation to drop to much lower level after I > logged out off the app. > > Also, the line: > > downloaded all Vct data (CSV) 1,055,000 (29,049 records) > > ...shows that after I performed that action (retrieving 7.2 MB of data from > DB, manipulating it as XML and finally serializing into a text file) the > memory occupied jumped to the maximum I set for the JVM to use and did not > drop after the operation was over. > > I do not know if this is JVM (using version 1.5.0.10), Tomcat (using version > 5.5.4) or OPS (v. 3.0.beta4.200512072017) issue or the methodology I chose > for my test? > > Thanks > > A. > > after login 156,720 > after loading POS interface 204,000 > after switching to "Summaries" 216,128 > after loading POS interface 264,676 > after switching to "Summaries and back 339,560 > after switching to "Summaries and back 420,000 > after switching to "Summaries and back 428,088 > after switching to "Summaries and back 521,708 > after switching to "Summaries and back 577,964 > after switching to "Summaries and back 654,668 > after switching to "Summaries and back 768,816 > log off and back on 769,216 > after switching to "Summaries and back 767,716 > after switching to "Summaries and back 804,152 > after switching to "Summaries and back 884,068 > after switching to "Summaries and back 732,476 > after switching to "Summaries and back 732,492 > logged-in as another user 732,488 > after loading POS interface 732,580 > after switching to "Summaries and back 731,940 > selected subset of records in Generic 640,000 kkm session > downloaded report in CSV 643,296 > downloaded all Vct data (CSV) 1,055,000 (29,049 records) > loaded all records 1,082,580 waited longer than usual for refresh > after switching to "Summaries and back 1,083,656 kkm session > downloaded all KKM data (CSV) JVM crash > ______________ > SEVERE: Exception invoking periodic operation: > java.lang.OutOfMemoryError: Java heap space > Jul 13, 2006 9:03:54 PM > org.apache.catalina.core.StandardWrapperValve invoke > SEVERE: Servlet.service() for servlet ops-main-servlet threw exception > java.lang.OutOfMemoryError: Java heap space > > > > -- > You receive this message as a subscriber of the [hidden email] > mailing list. > To unsubscribe: mailto:[hidden email] > For general help: mailto:[hidden email]?subject=help > ObjectWeb mailing lists service home page: http://www.objectweb.org/wws > > > -- Blog (XML, Web apps, Open Source): http://www.orbeon.com/blog/ -- You receive this message as a subscriber of the [hidden email] mailing list. To unsubscribe: mailto:[hidden email] For general help: mailto:[hidden email]?subject=help ObjectWeb mailing lists service home page: http://www.objectweb.org/wws
--
Follow Orbeon on Twitter: @orbeon Follow me on Twitter: @avernet |
Re: OPS and memory management
|
|
Hi Alex,
I thought something like that was happening, but that does not explain why is the app crashing. I do have to re-code this part of the app anyways, as converting large XML files is problematic with XSLT processor (and joost -that the XST processor is based on - has also problems with performance). thanks A. On Jul 20, 2006, at 3:41 AM, Alessandro Vernet wrote: > Hi Alexander, > > The VM does not necessarily return to the OS memory that has been > allocated by the OS to the VM, but that is not used anymore by the VM > to store a live Java object. > > Instead of looking at how much memory is used by the VM at the > OS-level, it is better to look at how much memory on the heap is used. > You can do this simply with the -verbosegc VM option. > > When you use that option, you will see that the size of the heap goes > up to its max size, and then comes back down when a full garbage > collection is done. The full GC are particularly interesting: you want > to make sure that a good amount of memory is freed when they happen > and that you don't have full GC happening all the time. If they do, it > can be a sign that the heap size is too small or that there is a leak > in the application. > > I hope this helps, > > Alex > > On 7/13/06, Alexander Zatko <[hidden email]> wrote: >> >> I have developed a web app in OPS that started to run out of >> memory recently >> after I added certain functionality that involves manipulating >> large XML >> files using XSLT processor. I think I can deal with that issue by >> switching >> to SAX processing (I hope), but I performed some testing which >> brought up >> some additional issues I would like to discuss. I have specified 1 >> GB memory >> to the JVM Tomcat is running in and started to invoke some >> functions in the >> application while watching the Tomcat process memory allocation >> (Using this >> on Windows 2003 server). The table below shows the actions I took >> and the >> chunk of memory used by Tomcat after that action. It looks to me >> like there >> are some problems with re-claiming memory by either the JVM, >> Tomcat or OPS >> (My code is perfect so that is out of the picture :-) ). >> >> The actions described will not tell you much, but if you look - >> for example >> - at the line: >> >> log off and back on 769,216 >> >> I would expect the memory allocation to drop to much lower level >> after I >> logged out off the app. >> >> Also, the line: >> >> downloaded all Vct data (CSV) 1,055,000 (29,049 records) >> >> ...shows that after I performed that action (retrieving 7.2 MB of >> data from >> DB, manipulating it as XML and finally serializing into a text >> file) the >> memory occupied jumped to the maximum I set for the JVM to use and >> did not >> drop after the operation was over. >> >> I do not know if this is JVM (using version 1.5.0.10), Tomcat >> (using version >> 5.5.4) or OPS (v. 3.0.beta4.200512072017) issue or the methodology >> I chose >> for my test? >> >> Thanks >> >> A. >> >> after login 156,720 >> after loading POS interface 204,000 >> after switching to "Summaries" 216,128 >> after loading POS interface 264,676 >> after switching to "Summaries and back 339,560 >> after switching to "Summaries and back 420,000 >> after switching to "Summaries and back 428,088 >> after switching to "Summaries and back 521,708 >> after switching to "Summaries and back 577,964 >> after switching to "Summaries and back 654,668 >> after switching to "Summaries and back 768,816 >> log off and back on 769,216 >> after switching to "Summaries and back 767,716 >> after switching to "Summaries and back 804,152 >> after switching to "Summaries and back 884,068 >> after switching to "Summaries and back 732,476 >> after switching to "Summaries and back 732,492 >> logged-in as another user 732,488 >> after loading POS interface 732,580 >> after switching to "Summaries and back 731,940 >> selected subset of records in Generic 640,000 kkm session >> downloaded report in CSV 643,296 >> downloaded all Vct data (CSV) 1,055,000 (29,049 records) >> loaded all records 1,082,580 waited longer than usual for refresh >> after switching to "Summaries and back 1,083,656 kkm session >> downloaded all KKM data (CSV) JVM crash >> ______________ >> SEVERE: Exception invoking periodic operation: >> java.lang.OutOfMemoryError: Java heap space >> Jul 13, 2006 9:03:54 PM >> org.apache.catalina.core.StandardWrapperValve invoke >> SEVERE: Servlet.service() for servlet ops-main-servlet threw >> exception >> java.lang.OutOfMemoryError: Java heap space >> >> >> >> -- >> You receive this message as a subscriber of the ops- >> [hidden email] >> mailing list. >> To unsubscribe: mailto:[hidden email] >> For general help: mailto:[hidden email]?subject=help >> ObjectWeb mailing lists service home page: http:// >> www.objectweb.org/wws >> >> >> > > > -- > Blog (XML, Web apps, Open Source): > http://www.orbeon.com/blog/ > > > -- > You receive this message as a subscriber of the ops- > [hidden email] mailing list. > To unsubscribe: mailto:[hidden email] > For general help: mailto:[hidden email]?subject=help > ObjectWeb mailing lists service home page: http://www.objectweb.org/ > wws -- You receive this message as a subscriber of the [hidden email] mailing list. To unsubscribe: mailto:[hidden email] For general help: mailto:[hidden email]?subject=help ObjectWeb mailing lists service home page: http://www.objectweb.org/wws |
Re: processing large XML files ( was Re: OPS and memory management)
|
Administrator
|
In reply to this post by Alexander Žaťko
On 7/18/06, Alexander Zatko <[hidden email]> wrote:
> [...] > download dialog box in about 10 seconds (a huge improvement over > using XSLT) and the download speed also starts at a decent speed, > but over time the number of bytes received per second drops to > "modem speeds". Not sure if this has anything to do with the size > of my XML (~ 7MB), the way I coded the style sheet or the oxf:stx > processor. The result average download speed was 6 kb/sec (the > slowest link between my web browser and the server has bandwidth of > cca 500 kbps). Hi Alexander, I haven't used STX in the last few years, so I am afraid I won't have a good answer to your question. When you say that the speed drops over this, this is for a give request, not for across request, right? So the timing of 10 seconds is to process 7 MB of data? That doesn't seem unreasonable. It looks like you would need to run a profiler on the code to see what is really happening, and if there is anything you can do to speed this up. Alex -- Blog (XML, Web apps, Open Source): http://www.orbeon.com/blog/ -- You receive this message as a subscriber of the [hidden email] mailing list. To unsubscribe: mailto:[hidden email] For general help: mailto:[hidden email]?subject=help ObjectWeb mailing lists service home page: http://www.objectweb.org/wws
--
Follow Orbeon on Twitter: @orbeon Follow me on Twitter: @avernet |
«
Return to Orbeon Forms community mailing list
|
1 view|%1 views
| Free forum by Nabble | Edit this page |